maestro-flow 0.3.37 → 0.3.39
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/workflow-analyzer.md +2 -0
- package/.claude/agents/workflow-debugger.md +2 -0
- package/.claude/agents/workflow-executor.md +2 -0
- package/.claude/agents/workflow-integration-checker.md +1 -0
- package/.claude/agents/workflow-nyquist-auditor.md +1 -0
- package/.claude/agents/workflow-planner.md +2 -0
- package/.claude/agents/workflow-reviewer.md +2 -0
- package/.claude/agents/workflow-verifier.md +2 -0
- package/.claude/commands/learn-decompose.md +176 -176
- package/.claude/commands/learn-follow.md +167 -167
- package/.claude/commands/learn-investigate.md +221 -221
- package/.claude/commands/learn-retro.md +303 -303
- package/.claude/commands/learn-second-opinion.md +167 -167
- package/.claude/commands/maestro-amend.md +300 -300
- package/.claude/commands/maestro-analyze.md +126 -126
- package/.claude/commands/maestro-brainstorm.md +100 -100
- package/.claude/commands/maestro-composer.md +354 -354
- package/.claude/commands/maestro-execute.md +120 -114
- package/.claude/commands/maestro-fork.md +86 -86
- package/.claude/commands/maestro-init.md +78 -78
- package/.claude/commands/maestro-learn.md +140 -140
- package/.claude/commands/maestro-link-coordinate.md +1 -1
- package/.claude/commands/maestro-merge.md +61 -61
- package/.claude/commands/maestro-milestone-release.md +96 -96
- package/.claude/commands/maestro-overlay.md +178 -178
- package/.claude/commands/maestro-plan.md +150 -138
- package/.claude/commands/maestro-player.md +404 -404
- package/.claude/commands/maestro-quick.md +56 -56
- package/.claude/commands/maestro-ralph-execute.md +7 -18
- package/.claude/commands/maestro-ralph.md +9 -3
- package/.claude/commands/maestro-roadmap.md +1 -1
- package/.claude/commands/maestro-ui-design.md +93 -93
- package/.claude/commands/maestro-update.md +176 -176
- package/.claude/commands/maestro-verify.md +96 -90
- package/.claude/commands/maestro.md +121 -121
- package/.claude/commands/manage-codebase-rebuild.md +75 -75
- package/.claude/commands/manage-codebase-refresh.md +57 -57
- package/.claude/commands/manage-harvest.md +94 -94
- package/.claude/commands/manage-issue-discover.md +77 -77
- package/.claude/commands/manage-issue.md +73 -73
- package/.claude/commands/manage-knowhow-capture.md +193 -193
- package/.claude/commands/manage-knowhow.md +77 -77
- package/.claude/commands/manage-learn.md +67 -67
- package/.claude/commands/manage-status.md +51 -51
- package/.claude/commands/manage-wiki.md +62 -62
- package/.claude/commands/quality-auto-test.md +1 -1
- package/.claude/commands/quality-debug.md +121 -115
- package/.claude/commands/quality-refactor.md +55 -55
- package/.claude/commands/quality-retrospective.md +78 -78
- package/.claude/commands/quality-review.md +114 -108
- package/.claude/commands/quality-sync.md +51 -51
- package/.claude/commands/quality-test.md +103 -103
- package/.claude/commands/spec-add.md +49 -49
- package/.claude/commands/spec-load.md +51 -51
- package/.claude/commands/spec-remove.md +51 -51
- package/.claude/commands/spec-setup.md +51 -51
- package/.claude/commands/wiki-connect.md +62 -62
- package/.claude/commands/wiki-digest.md +69 -69
- package/.codex/skills/learn-decompose/SKILL.md +113 -113
- package/.codex/skills/learn-follow/SKILL.md +1 -1
- package/.codex/skills/learn-investigate/SKILL.md +83 -83
- package/.codex/skills/learn-retro/SKILL.md +83 -83
- package/.codex/skills/learn-second-opinion/SKILL.md +86 -86
- package/.codex/skills/maestro/SKILL.md +304 -304
- package/.codex/skills/maestro-analyze/SKILL.md +9 -8
- package/.codex/skills/maestro-brainstorm/SKILL.md +442 -397
- package/.codex/skills/maestro-composer/SKILL.md +213 -213
- package/.codex/skills/maestro-execute/SKILL.md +346 -318
- package/.codex/skills/maestro-fork/SKILL.md +56 -2
- package/.codex/skills/maestro-init/SKILL.md +40 -16
- package/.codex/skills/maestro-learn/SKILL.md +80 -80
- package/.codex/skills/maestro-link-coordinate/SKILL.md +257 -257
- package/.codex/skills/maestro-merge/SKILL.md +1 -1
- package/.codex/skills/maestro-milestone-audit/SKILL.md +1 -1
- package/.codex/skills/maestro-milestone-complete/SKILL.md +40 -9
- package/.codex/skills/maestro-milestone-release/SKILL.md +70 -70
- package/.codex/skills/maestro-overlay/SKILL.md +1 -1
- package/.codex/skills/maestro-plan/SKILL.md +19 -4
- package/.codex/skills/maestro-player/SKILL.md +323 -323
- package/.codex/skills/maestro-quick/SKILL.md +1 -1
- package/.codex/skills/maestro-ralph/SKILL.md +681 -578
- package/.codex/skills/maestro-roadmap/SKILL.md +518 -468
- package/.codex/skills/maestro-ui-design/SKILL.md +109 -12
- package/.codex/skills/maestro-verify/SKILL.md +27 -9
- package/.codex/skills/manage-codebase-rebuild/SKILL.md +3 -2
- package/.codex/skills/manage-codebase-refresh/SKILL.md +1 -1
- package/.codex/skills/manage-harvest/SKILL.md +91 -91
- package/.codex/skills/manage-issue/SKILL.md +19 -6
- package/.codex/skills/manage-issue-discover/SKILL.md +1 -1
- package/.codex/skills/manage-knowhow/SKILL.md +95 -95
- package/.codex/skills/manage-knowhow-capture/SKILL.md +110 -110
- package/.codex/skills/manage-learn/SKILL.md +1 -1
- package/.codex/skills/manage-status/SKILL.md +1 -1
- package/.codex/skills/manage-wiki/SKILL.md +55 -55
- package/.codex/skills/quality-auto-test/SKILL.md +547 -547
- package/.codex/skills/quality-debug/SKILL.md +339 -334
- package/.codex/skills/quality-refactor/SKILL.md +1 -1
- package/.codex/skills/quality-retrospective/SKILL.md +292 -292
- package/.codex/skills/quality-review/SKILL.md +365 -364
- package/.codex/skills/quality-sync/SKILL.md +1 -1
- package/.codex/skills/quality-test/SKILL.md +498 -498
- package/.codex/skills/spec-add/SKILL.md +101 -101
- package/.codex/skills/spec-load/SKILL.md +77 -77
- package/.codex/skills/spec-map/SKILL.md +1 -1
- package/.codex/skills/spec-remove/SKILL.md +69 -69
- package/.codex/skills/spec-setup/SKILL.md +1 -1
- package/.codex/skills/team-coordinate/SKILL.md +2 -1
- package/.codex/skills/team-executor/SKILL.md +116 -115
- package/.codex/skills/team-lifecycle-v4/SKILL.md +2 -1
- package/.codex/skills/team-lifecycle-v4/instructions/agent-instruction.md +14 -6
- package/.codex/skills/team-lifecycle-v4/roles/analyst/role.md +16 -4
- package/.codex/skills/team-lifecycle-v4/roles/executor/commands/implement.md +7 -1
- package/.codex/skills/team-lifecycle-v4/roles/planner/role.md +16 -4
- package/.codex/skills/team-lifecycle-v4/roles/writer/role.md +8 -2
- package/.codex/skills/team-quality-assurance/SKILL.md +2 -1
- package/.codex/skills/team-quality-assurance/roles/scout/role.md +9 -2
- package/.codex/skills/team-review/SKILL.md +2 -1
- package/.codex/skills/team-review/roles/reviewer/role.md +10 -1
- package/.codex/skills/team-review/roles/scanner/role.md +10 -1
- package/.codex/skills/team-tech-debt/SKILL.md +144 -143
- package/.codex/skills/team-tech-debt/roles/executor/role.md +9 -5
- package/.codex/skills/team-tech-debt/roles/scanner/role.md +10 -0
- package/.codex/skills/team-tech-debt/roles/validator/role.md +8 -2
- package/.codex/skills/team-testing/SKILL.md +2 -1
- package/.codex/skills/team-testing/roles/executor/role.md +8 -2
- package/.codex/skills/team-testing/roles/generator/role.md +8 -2
- package/.codex/skills/wiki-connect/SKILL.md +73 -73
- package/.codex/skills/wiki-digest/SKILL.md +87 -87
- package/README.md +6 -0
- package/README.zh-CN.md +6 -0
- package/dashboard/dist-server/dashboard/src/server/agents/claude-code-adapter.js +4 -0
- package/dashboard/dist-server/dashboard/src/server/agents/claude-code-adapter.js.map +1 -1
- package/dashboard/dist-server/dashboard/src/server/agents/codex-cli-adapter.js +118 -7
- package/dashboard/dist-server/dashboard/src/server/agents/codex-cli-adapter.js.map +1 -1
- package/dashboard/dist-server/shared/agent-types.d.ts +2 -0
- package/dashboard/dist-server/src/agents/cli-agent-runner.d.ts +2 -0
- package/dashboard/dist-server/src/agents/cli-agent-runner.js +4 -0
- package/dashboard/dist-server/src/agents/cli-agent-runner.js.map +1 -1
- package/dashboard/dist-server/src/commands/delegate.d.ts +2 -0
- package/dashboard/dist-server/src/commands/delegate.js +18 -0
- package/dashboard/dist-server/src/commands/delegate.js.map +1 -1
- package/dashboard/dist-server/src/config/cli-tools-config.d.ts +6 -0
- package/dashboard/dist-server/src/config/cli-tools-config.js +2 -0
- package/dashboard/dist-server/src/config/cli-tools-config.js.map +1 -1
- package/dist/shared/agent-types.d.ts +2 -0
- package/dist/shared/agent-types.d.ts.map +1 -1
- package/dist/src/agents/cli-agent-runner.d.ts +2 -0
- package/dist/src/agents/cli-agent-runner.d.ts.map +1 -1
- package/dist/src/agents/cli-agent-runner.js +4 -0
- package/dist/src/agents/cli-agent-runner.js.map +1 -1
- package/dist/src/commands/config.d.ts.map +1 -1
- package/dist/src/commands/config.js +29 -1
- package/dist/src/commands/config.js.map +1 -1
- package/dist/src/commands/delegate.d.ts +2 -0
- package/dist/src/commands/delegate.d.ts.map +1 -1
- package/dist/src/commands/delegate.js +18 -0
- package/dist/src/commands/delegate.js.map +1 -1
- package/dist/src/commands/launcher.d.ts.map +1 -1
- package/dist/src/commands/launcher.js +27 -4
- package/dist/src/commands/launcher.js.map +1 -1
- package/dist/src/config/cli-tools-config.d.ts +6 -0
- package/dist/src/config/cli-tools-config.d.ts.map +1 -1
- package/dist/src/config/cli-tools-config.js +2 -0
- package/dist/src/config/cli-tools-config.js.map +1 -1
- package/dist/src/core/overlay/applier.d.ts.map +1 -1
- package/dist/src/core/overlay/applier.js +65 -5
- package/dist/src/core/overlay/applier.js.map +1 -1
- package/dist/src/core/overlay/loader.d.ts.map +1 -1
- package/dist/src/core/overlay/loader.js +9 -4
- package/dist/src/core/overlay/loader.js.map +1 -1
- package/dist/src/core/overlay/types.d.ts +2 -0
- package/dist/src/core/overlay/types.d.ts.map +1 -1
- package/dist/src/core/overlay/types.js +2 -0
- package/dist/src/core/overlay/types.js.map +1 -1
- package/dist/src/tui/tools-ui/ToolsDashboard.d.ts.map +1 -1
- package/dist/src/tui/tools-ui/ToolsDashboard.js +1 -1
- package/dist/src/tui/tools-ui/ToolsDashboard.js.map +1 -1
- package/dist/src/tui/tools-ui/ToolsOverview.d.ts.map +1 -1
- package/dist/src/tui/tools-ui/ToolsOverview.js +51 -4
- package/dist/src/tui/tools-ui/ToolsOverview.js.map +1 -1
- package/package.json +1 -1
- package/shared/agent-types.ts +2 -0
- package/workflows/delegate-protocol.codex.md +65 -0
- package/workflows/issue-analyze.md +2 -3
- package/workflows/issue-gaps-analyze.codex.md +260 -0
- package/workflows/issue-gaps-analyze.md +214 -0
|
@@ -1,547 +1,547 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: quality-auto-test
|
|
3
|
-
description:
|
|
4
|
-
argument-hint: "<phase> [-y] [-c N] [--max-iter N] [--layer L0-L3] [--dry-run] [--re-run]"
|
|
5
|
-
allowed-tools: spawn_agents_on_csv, Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion
|
|
6
|
-
---
|
|
7
|
-
|
|
8
|
-
<purpose>
|
|
9
|
-
Unified automated testing using `spawn_agents_on_csv` for parallel test writing and failure diagnosis. Reads project state to auto-select scenario source (PRD specs, coverage gaps, or code exploration). All sources converge into a shared CSV pipeline: discover infrastructure → plan → write tests via CSV parallel → execute per layer → diagnose failures via CSV parallel → iterate → report.
|
|
10
|
-
|
|
11
|
-
**Core workflow**: Route → Source Scenarios → Build CSV → Layer-by-Layer Parallel Execution → Iterate (Diagnose CSV) → Report
|
|
12
|
-
|
|
13
|
-
**Topology**: Layers as waves (L0→L1→L2→L3 sequential, scenarios within layer parallel)
|
|
14
|
-
|
|
15
|
-
```
|
|
16
|
-
+---------------------------------------------------------------------------+
|
|
17
|
-
| AUTO-TEST CSV LAYER PIPELINE |
|
|
18
|
-
+---------------------------------------------------------------------------+
|
|
19
|
-
| |
|
|
20
|
-
| Phase 1: Route & Plan -> CSV |
|
|
21
|
-
| +-- Read project state, auto-select route (spec/gap/code) |
|
|
22
|
-
| +-- Extract scenarios per route, normalize to unified format |
|
|
23
|
-
| +-- Discover test infrastructure (framework, patterns) |
|
|
24
|
-
| +-- Build scenarios.csv with one row per scenario |
|
|
25
|
-
| +-- Layers = waves (L1, L2, L3 sequential; L0 = static pre-check) |
|
|
26
|
-
| +-- User validates test plan (skip if -y) |
|
|
27
|
-
| |
|
|
28
|
-
| Phase 2: Layer Execution Engine (write + run) |
|
|
29
|
-
| +-- L0: Static analysis (tsc + eslint) — no CSV needed |
|
|
30
|
-
| +-- For each layer L1→L3 (sequential, fail-fast on critical): |
|
|
31
|
-
| | +-- Layer N: Write Tests (parallel via spawn_agents_on_csv) |
|
|
32
|
-
| | | +-- Each agent writes one test file (RED-GREEN) |
|
|
33
|
-
| | | +-- Agent reads target source + infrastructure patterns |
|
|
34
|
-
| | | +-- Agent verifies RED (run test, check it targets behavior) |
|
|
35
|
-
| | | +-- Results: test_file written, red_result, findings |
|
|
36
|
-
| | +-- Merge write-results into master scenarios.csv |
|
|
37
|
-
| | +-- Run all layer tests together (full layer execution) |
|
|
38
|
-
| | +-- Record per-scenario pass/fail results |
|
|
39
|
-
| |
|
|
40
|
-
| Phase 3: Iteration Engine (diagnose + fix) |
|
|
41
|
-
| +-- OUTER LOOP (max_iter iterations): |
|
|
42
|
-
| | +-- For each layer with failures: |
|
|
43
|
-
| | | +-- Build diagnosis.csv from failed scenarios |
|
|
44
|
-
| | | +-- Diagnose & Fix (parallel via spawn_agents_on_csv) |
|
|
45
|
-
| | | | +-- Each agent classifies one failure cluster |
|
|
46
|
-
| | | | +-- test_defect: agent provides fix diff |
|
|
47
|
-
| | | | +-- code_defect: agent documents evidence |
|
|
48
|
-
| | | +-- Apply test_defect fixes, re-run layer |
|
|
49
|
-
| | +-- Reflect: analyze trends, log strategy |
|
|
50
|
-
| | +-- Adjust: select next strategy (conservative/aggressive/...) |
|
|
51
|
-
| | +-- Convergence check: >=95% → done |
|
|
52
|
-
| +-- discoveries.ndjson shared across all iterations |
|
|
53
|
-
| |
|
|
54
|
-
| Phase 4: Results & Routing |
|
|
55
|
-
| +-- Export results.csv |
|
|
56
|
-
| +-- Write report.json, state.json, reflection-log.md |
|
|
57
|
-
| +-- Conditional: traceability.md, issue creation |
|
|
58
|
-
| +-- Route to next step based on convergence |
|
|
59
|
-
| |
|
|
60
|
-
+---------------------------------------------------------------------------+
|
|
61
|
-
```
|
|
62
|
-
</purpose>
|
|
63
|
-
|
|
64
|
-
<context>
|
|
65
|
-
```bash
|
|
66
|
-
$quality-auto-test "3" # auto-detect source, full iteration
|
|
67
|
-
$quality-auto-test -c 4 "3" # max 4 concurrent test writers per layer
|
|
68
|
-
$quality-auto-test -y "3 --max-iter 1" # single-pass generation only
|
|
69
|
-
$quality-auto-test "3 --dry-run" # plan only, no execution
|
|
70
|
-
$quality-auto-test "3 --re-run" # re-run only previously failed scenarios
|
|
71
|
-
$quality-auto-test "3 --layer L2" # restrict to L2 integration tests
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
**Flags**:
|
|
75
|
-
- `-y, --yes`: Skip all confirmations
|
|
76
|
-
- `-c, --concurrency N`: Max concurrent agents within each layer (default: 5)
|
|
77
|
-
- `--max-iter N`: Max outer iterations (default 5). **1 = single-pass** generation only
|
|
78
|
-
- `--layer L`: Start from or restrict to specific layer (L0|L1|L2|L3)
|
|
79
|
-
- `--strategy conservative|aggressive|surgical|reflective`: Override starting iteration strategy (default: auto-selected)
|
|
80
|
-
- `--dry-run`: Generate test plan only, do not execute
|
|
81
|
-
- `--re-run`: Re-run only previously failed/blocked scenarios
|
|
82
|
-
|
|
83
|
-
**Intelligent routing** (auto-detected, priority order):
|
|
84
|
-
|
|
85
|
-
| Priority | Condition | Route |
|
|
86
|
-
|----------|-----------|-------|
|
|
87
|
-
| 1 | Active session (state.json status=running) | Resume |
|
|
88
|
-
| 2 | --re-run flag + previous failures | Re-run |
|
|
89
|
-
| 3 | Spec package exists (REQ-*.md) | spec (PRD-forward) |
|
|
90
|
-
| 4 | Nyquist gaps exist (verification.json) | gap (coverage-forward) |
|
|
91
|
-
| 5 | Default | code (exploration-forward) |
|
|
92
|
-
|
|
93
|
-
**Session Directory**: `.tests/auto-test/.csv-session/`
|
|
94
|
-
**Core Output**: `scenarios.csv` (master state) + `results.csv` (final) + `discoveries.ndjson` (shared) + `report.json` + `state.json`
|
|
95
|
-
</context>
|
|
96
|
-
|
|
97
|
-
<csv_schema>
|
|
98
|
-
|
|
99
|
-
### scenarios.csv (Master State — Test Writing Phase)
|
|
100
|
-
|
|
101
|
-
```csv
|
|
102
|
-
id,name,layer,priority,category,target_file,test_file,description,test_cases,fixtures,req_ref,infrastructure_hints,prev_context,status,red_result,findings,error
|
|
103
|
-
"AT-001","Auth token validation","L1","critical","api_contract","src/auth/token.ts","src/auth/__tests__/token.test.ts","Validate JWT token verification returns correct payload","verify valid token returns payload;verify expired token throws;verify malformed token throws","valid_token fixture;expired_token fixture","REQ-001:AC-1","vitest;describe/it pattern;see src/utils/__tests__/hash.test.ts","","","","",""
|
|
104
|
-
"AT-002","Login endpoint integration","L2","high","business_rule","src/routes/login.ts","src/routes/__tests__/login.integration.test.ts","POST /api/login returns JWT on valid credentials","valid login returns 200+token;invalid password returns 401;missing email returns 400","user_fixture;credentials_fixture","REQ-002:AC-1","supertest;see src/routes/__tests__/health.test.ts","AT-001 findings: token module exports verifyToken/generateToken","","","",""
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
**Columns**:
|
|
108
|
-
|
|
109
|
-
| Column | Phase | Description |
|
|
110
|
-
|--------|-------|-------------|
|
|
111
|
-
| `id` | Input | Scenario ID (AT-NNN format) |
|
|
112
|
-
| `name` | Input | Short scenario name |
|
|
113
|
-
| `layer` | Input | L1/L2/L3 (determines wave order) |
|
|
114
|
-
| `priority` | Input | critical/high/medium |
|
|
115
|
-
| `category` | Input | api_contract/business_rule/state_transition/user_flow/... |
|
|
116
|
-
| `target_file` | Input | Source file being tested |
|
|
117
|
-
| `test_file` | Input | Target test file path to create |
|
|
118
|
-
| `description` | Input | What this scenario validates |
|
|
119
|
-
| `test_cases` | Input | Semicolon-separated test cases |
|
|
120
|
-
| `fixtures` | Input | Required fixtures/mocks (semicolon-separated) |
|
|
121
|
-
| `req_ref` | Input | Requirement reference (REQ-NNN:AC-N or gap-id or empty) |
|
|
122
|
-
| `infrastructure_hints` | Input | Framework + pattern references from Step 3 |
|
|
123
|
-
| `prev_context` | Computed | Findings from prior layer scenarios (cross-layer propagation) |
|
|
124
|
-
| `status` | Output | pending → written → passed → failed → blocked |
|
|
125
|
-
| `red_result` | Output | expected_fail / unexpected_fail / pass (RED phase result) |
|
|
126
|
-
| `findings` | Output | Implementation notes, patterns discovered (max 500 chars) |
|
|
127
|
-
| `error` | Output | Error message if failed |
|
|
128
|
-
|
|
129
|
-
### diagnosis.csv (Iteration Phase — Failure Diagnosis)
|
|
130
|
-
|
|
131
|
-
```csv
|
|
132
|
-
id,scenario_id,layer,test_file,error_detail,expected,actual,target_file,source_context,classification,fix_code,evidence,error
|
|
133
|
-
"DX-001","AT-003","L1","src/auth/__tests__/token.test.ts","TypeError: verifyToken is not a function","verifyToken returns decoded payload","Function not exported from module","src/auth/token.ts","token.ts exports: generateToken only","test_defect","import { verifyToken } from '../token' → import { verifyToken } from '../verify-token'","src/auth/verify-token.ts:15 exports verifyToken",""
|
|
134
|
-
"DX-002","AT-005","L2","src/routes/__tests__/login.test.ts","Expected 200, received 500","POST /login returns 200 with valid credentials","Internal server error: database connection refused","src/routes/login.ts","login.ts calls UserModel.findByEmail","env_issue","","Database not available in test environment",""
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
**Columns**:
|
|
138
|
-
|
|
139
|
-
| Column | Phase | Description |
|
|
140
|
-
|--------|-------|-------------|
|
|
141
|
-
| `id` | Input | Diagnosis ID (DX-NNN) |
|
|
142
|
-
| `scenario_id` | Input | Reference to AT-NNN scenario |
|
|
143
|
-
| `layer` | Input | Layer where failure occurred |
|
|
144
|
-
| `test_file` | Input | Test file that failed |
|
|
145
|
-
| `error_detail` | Input | Full error message/stack trace excerpt |
|
|
146
|
-
| `expected` | Input | Expected behavior from scenario |
|
|
147
|
-
| `actual` | Input | Actual behavior observed |
|
|
148
|
-
| `target_file` | Input | Source file being tested |
|
|
149
|
-
| `source_context` | Input | Relevant source code context (exports, imports) |
|
|
150
|
-
| `classification` | Output | test_defect / code_defect / env_issue |
|
|
151
|
-
| `fix_code` | Output | Fix diff for test_defect (old → new) |
|
|
152
|
-
| `evidence` | Output | file:line references for diagnosis |
|
|
153
|
-
| `error` | Output | Agent error if diagnosis failed |
|
|
154
|
-
|
|
155
|
-
### Session Structure
|
|
156
|
-
|
|
157
|
-
```
|
|
158
|
-
.tests/auto-test/.csv-session/
|
|
159
|
-
+-- scenarios.csv (master state)
|
|
160
|
-
+-- results.csv (final export)
|
|
161
|
-
+-- discoveries.ndjson (shared across iterations)
|
|
162
|
-
+-- layer-L{N}-write.csv (temporary, per-layer write input)
|
|
163
|
-
+-- layer-L{N}-write-results.csv
|
|
164
|
-
+-- diagnosis-iter-{N}.csv (temporary, per-iteration diagnosis)
|
|
165
|
-
+-- diagnosis-iter-{N}-results.csv
|
|
166
|
-
```
|
|
167
|
-
</csv_schema>
|
|
168
|
-
|
|
169
|
-
<invariants>
|
|
170
|
-
1. **Start Immediately**: First action is session initialization
|
|
171
|
-
2. **Layer Order is Sacred**: Never execute L(N+1) before L(N) completes (fail-fast on critical)
|
|
172
|
-
3. **CSV is Source of Truth**: Master scenarios.csv holds all test execution state
|
|
173
|
-
4. **Context Propagation**: prev_context built from prior-layer findings in CSV, not memory
|
|
174
|
-
5. **Discovery Board Append-Only**: Never clear or modify discoveries.ndjson
|
|
175
|
-
6. **Route Auto-Detected**: Read state, never ask user for mode
|
|
176
|
-
7. **RED-GREEN Methodology**: Tests target real behavior; failing test = bug discovery (never fix source)
|
|
177
|
-
8. **Max 3 Inner Fix Attempts**: Per layer, fix test_defects up to 3 times via diagnosis CSV
|
|
178
|
-
9. **Convergence Threshold**: 95% pass rate = converged
|
|
179
|
-
10. **DO NOT STOP**: Continuous execution until convergence, max_iter, or all remaining = code_defect
|
|
180
|
-
</invariants>
|
|
181
|
-
|
|
182
|
-
<execution>
|
|
183
|
-
|
|
184
|
-
### Session Initialization
|
|
185
|
-
|
|
186
|
-
```
|
|
187
|
-
Parse from $ARGUMENTS:
|
|
188
|
-
AUTO_YES ← --yes | -y
|
|
189
|
-
maxConcurrency ← --concurrency | -c N (default: 5)
|
|
190
|
-
MAX_ITER ← --max-iter N (default: 5)
|
|
191
|
-
layerFilter ← --layer L (default: null = all)
|
|
192
|
-
startStrategy ← --strategy conservative|aggressive|surgical|reflective (default: null = auto)
|
|
193
|
-
dryRun ← --dry-run
|
|
194
|
-
reRun ← --re-run
|
|
195
|
-
phaseArg ← remaining text
|
|
196
|
-
|
|
197
|
-
Derive:
|
|
198
|
-
dateStr ← UTC+8 YYYYMMDD
|
|
199
|
-
sessionFolder ← ".tests/auto-test/.csv-session"
|
|
200
|
-
|
|
201
|
-
mkdir -p {sessionFolder}
|
|
202
|
-
```
|
|
203
|
-
|
|
204
|
-
### Phase 1: Route & Plan → CSV
|
|
205
|
-
|
|
206
|
-
#### Step 0: Parse & Load
|
|
207
|
-
|
|
208
|
-
Resolve phase dir from `state.json` artifact registry (`type='execute'`, matching phase). Error E002 if not found.
|
|
209
|
-
|
|
210
|
-
```
|
|
211
|
-
specs_test = maestro spec load --category test
|
|
212
|
-
specs_arch = maestro spec load --category arch
|
|
213
|
-
```
|
|
214
|
-
|
|
215
|
-
#### Step 1: Read State & Route
|
|
216
|
-
|
|
217
|
-
```
|
|
218
|
-
Priority: Resume > Re-run > Spec > Gap > Code
|
|
219
|
-
|
|
220
|
-
1. RESUME: .csv-session/scenarios.csv exists AND state.json status == "running"
|
|
221
|
-
→ offer resume or restart (resume = reload CSV, jump to current iteration)
|
|
222
|
-
|
|
223
|
-
2. RE-RUN: --re-run flag AND report.json has failed/blocked scenarios
|
|
224
|
-
→ load failed scenarios into CSV with status reset to pending
|
|
225
|
-
|
|
226
|
-
3. SPEC: .workflow/.spec/SPEC-*/requirements/REQ-*.md exists
|
|
227
|
-
→ ROUTE = "spec", SPEC_MODE = "full" | "degraded"
|
|
228
|
-
|
|
229
|
-
4. GAP: verification.json has gaps[] (MISSING/PARTIAL)
|
|
230
|
-
→ ROUTE = "gap"
|
|
231
|
-
|
|
232
|
-
5. CODE: Default fallback → ROUTE = "code"
|
|
233
|
-
```
|
|
234
|
-
|
|
235
|
-
#### Step 2: Source Scenarios
|
|
236
|
-
|
|
237
|
-
Execute route-specific extraction, normalize to unified format.
|
|
238
|
-
|
|
239
|
-
**Route A: spec** — Parse REQ acceptance criteria, classify layers, generate fixtures.
|
|
240
|
-
**Route B: gap** — Read verification/coverage gaps, classify files by type.
|
|
241
|
-
**Route C: code** — Explore module boundaries, API endpoints, integration points.
|
|
242
|
-
|
|
243
|
-
All routes produce unified scenario objects (see csv_schema).
|
|
244
|
-
|
|
245
|
-
#### Step 3: Discover Infrastructure
|
|
246
|
-
|
|
247
|
-
Detect framework, read 2-3 existing tests for patterns. Build `infrastructure_hints` string per scenario.
|
|
248
|
-
|
|
249
|
-
#### Step 4: Build scenarios.csv & Confirm
|
|
250
|
-
|
|
251
|
-
1. Build master `scenarios.csv` — one row per scenario, grouped by layer
|
|
252
|
-
2. Set `prev_context` for L2 scenarios from L1 scenario descriptions (cross-layer dependency)
|
|
253
|
-
3. Set `prev_context` for L3 from L2 findings
|
|
254
|
-
|
|
255
|
-
Display plan summary:
|
|
256
|
-
```
|
|
257
|
-
=== AUTO-TEST PLAN ===
|
|
258
|
-
来源: {ROUTE} | 阶段: {phase_name} | Spec: {spec_ref or "N/A"}
|
|
259
|
-
|
|
260
|
-
L0 Static: {N} checks
|
|
261
|
-
L1 Unit/API: {N} scenarios ({X} critical, {Y} high)
|
|
262
|
-
L2 Integration: {N} scenarios ({X} critical, {Y} high)
|
|
263
|
-
L3 E2E: {N} scenarios ({X} critical, {Y} high)
|
|
264
|
-
|
|
265
|
-
Total: {N} scenarios | Max iterations: {MAX_ITER} | Concurrency: {maxConcurrency}
|
|
266
|
-
Proceed? (yes/edit/cancel)
|
|
267
|
-
```
|
|
268
|
-
|
|
269
|
-
- `--dry-run`: stop here. `-y`: skip confirmation.
|
|
270
|
-
|
|
271
|
-
### Phase 2: Layer Execution Engine (Write + Run)
|
|
272
|
-
|
|
273
|
-
#### L0: Static Analysis (no CSV)
|
|
274
|
-
|
|
275
|
-
```bash
|
|
276
|
-
tsc --noEmit && eslint src/
|
|
277
|
-
```
|
|
278
|
-
|
|
279
|
-
If L0 fails → stop, do not proceed to L1.
|
|
280
|
-
|
|
281
|
-
#### Per-Layer Write Loop (L1 → L2 → L3)
|
|
282
|
-
|
|
283
|
-
For each layer (sequential, respecting --layer filter):
|
|
284
|
-
|
|
285
|
-
**1. Extract layer rows** from master `scenarios.csv` (filter by `layer == L{N}`, status == pending)
|
|
286
|
-
|
|
287
|
-
**2. Populate `prev_context`** from completed prior-layer findings in master CSV
|
|
288
|
-
|
|
289
|
-
**3. Write `layer-L{N}-write.csv`** then execute parallel test writing:
|
|
290
|
-
|
|
291
|
-
```javascript
|
|
292
|
-
spawn_agents_on_csv({
|
|
293
|
-
csv_path: `${sessionFolder}/layer-L${N}-write.csv`,
|
|
294
|
-
id_column: "id",
|
|
295
|
-
instruction: buildTestWriterInstruction(infrastructure, specsContent, phaseDir),
|
|
296
|
-
max_concurrency: maxConcurrency,
|
|
297
|
-
max_runtime_seconds: 1800,
|
|
298
|
-
output_csv_path: `${sessionFolder}/layer-L${N}-write-results.csv`,
|
|
299
|
-
output_schema: { id, status: [written|failed], red_result: [expected_fail|unexpected_fail|pass], findings, error }
|
|
300
|
-
})
|
|
301
|
-
```
|
|
302
|
-
|
|
303
|
-
**Test Writer Agent Instruction** (per scenario row):
|
|
304
|
-
```
|
|
305
|
-
You are a test writer. Write ONE test file for the given scenario.
|
|
306
|
-
|
|
307
|
-
## Task
|
|
308
|
-
- Read the target_file to understand the module under test
|
|
309
|
-
- Write test file at test_file path following infrastructure_hints patterns
|
|
310
|
-
- Each test case in test_cases becomes one it() block
|
|
311
|
-
- Use fixtures from fixtures column (infer from source if empty)
|
|
312
|
-
- Include scenario id in describe: describe("AT-NNN: {name}", ...)
|
|
313
|
-
- Run the test file once after writing
|
|
314
|
-
|
|
315
|
-
## RED-GREEN Rules
|
|
316
|
-
- If test PASSES immediately: note "pass" — may need strengthening
|
|
317
|
-
- If test FAILS as expected (tests real behavior): note "expected_fail" — good
|
|
318
|
-
- If test FAILS unexpectedly (setup/import error): fix test setup, note "unexpected_fail"
|
|
319
|
-
- NEVER modify source code — only write/fix test files
|
|
320
|
-
|
|
321
|
-
## Output
|
|
322
|
-
- status: "written" if test file created successfully, "failed" if unable
|
|
323
|
-
- red_result: the RED phase outcome
|
|
324
|
-
- findings: patterns discovered, notes for dependent scenarios (max 500 chars)
|
|
325
|
-
- error: only if status == "failed"
|
|
326
|
-
|
|
327
|
-
## Context
|
|
328
|
-
- prev_context: {prev_context} (findings from prior layer)
|
|
329
|
-
- Read discoveries.ndjson for shared patterns before starting
|
|
330
|
-
- Append to discoveries.ndjson if you find reusable patterns
|
|
331
|
-
```
|
|
332
|
-
|
|
333
|
-
**4. Merge write-results** into master `scenarios.csv`, delete temp CSV
|
|
334
|
-
|
|
335
|
-
**5. Run full layer test suite:**
|
|
336
|
-
```bash
|
|
337
|
-
{run_command} --testPathPattern="{layer_pattern}"
|
|
338
|
-
```
|
|
339
|
-
|
|
340
|
-
**6. Record per-scenario results** (pass/fail/blocked with error_detail)
|
|
341
|
-
|
|
342
|
-
**7. Fail-fast check:** If ANY critical-priority scenario failed → do NOT proceed to next layer
|
|
343
|
-
|
|
344
|
-
**If `--max-iter 1`:** After all layers written and run once, jump to Phase 4 (single-pass).
|
|
345
|
-
|
|
346
|
-
### Phase 3: Iteration Engine (Diagnose + Fix)
|
|
347
|
-
|
|
348
|
-
```
|
|
349
|
-
OUTER LOOP (max_iter iterations):
|
|
350
|
-
|
|
351
|
-
FOR each layer with failures (L1 through current):
|
|
352
|
-
|
|
353
|
-
INNER LOOP (max 3 per layer):
|
|
354
|
-
|
|
355
|
-
1. Build diagnosis.csv from failed scenarios in master CSV
|
|
356
|
-
(only scenarios with status=failed AND classification != code_defect)
|
|
357
|
-
|
|
358
|
-
2. IF diagnosis rows >= 1:
|
|
359
|
-
spawn_agents_on_csv({
|
|
360
|
-
csv_path: `${sessionFolder}/diagnosis-iter-${iter}.csv`,
|
|
361
|
-
id_column: "id",
|
|
362
|
-
instruction: buildDiagnosisInstruction(infrastructure),
|
|
363
|
-
max_concurrency: maxConcurrency,
|
|
364
|
-
max_runtime_seconds: 1200,
|
|
365
|
-
output_csv_path: `${sessionFolder}/diagnosis-iter-${iter}-results.csv`,
|
|
366
|
-
output_schema: { id, classification, fix_code, evidence, error }
|
|
367
|
-
})
|
|
368
|
-
|
|
369
|
-
3. Merge diagnosis results:
|
|
370
|
-
- test_defect with fix_code → apply fix, update scenario status to "pending"
|
|
371
|
-
- code_defect → mark as confirmed failure (stop retrying)
|
|
372
|
-
- env_issue → mark as blocked
|
|
373
|
-
|
|
374
|
-
4. Re-run ALL scenarios in this layer (catch regressions)
|
|
375
|
-
5. IF no test_defects remain: break inner loop
|
|
376
|
-
|
|
377
|
-
END INNER
|
|
378
|
-
|
|
379
|
-
Record final layer results
|
|
380
|
-
IF critical code_defects: stop layer progression (fail-fast)
|
|
381
|
-
|
|
382
|
-
END FOR
|
|
383
|
-
|
|
384
|
-
REFLECT:
|
|
385
|
-
Analyze: pass rate delta, failure clusters, strategy effectiveness
|
|
386
|
-
Append to reflection-log.md
|
|
387
|
-
|
|
388
|
-
ADJUST (Adaptive Strategy):
|
|
389
|
-
IF startStrategy provided AND iteration == 1: use startStrategy as initial
|
|
390
|
-
OTHERWISE auto-select:
|
|
391
|
-
|
|
392
|
-
| Condition | Strategy |
|
|
393
|
-
|-----------|----------|
|
|
394
|
-
| Iteration 1-2 | Conservative: fix obvious test_defects only |
|
|
395
|
-
| Pass rate >80% | Aggressive: batch-fix related failures |
|
|
396
|
-
| New regressions | Surgical: revert, fix regression only |
|
|
397
|
-
| Stuck 3+ iters | Reflective: re-analyze root cause pattern |
|
|
398
|
-
|
|
399
|
-
CONVERGENCE:
|
|
400
|
-
pass_rate >= 95% → Phase 4 (converged)
|
|
401
|
-
iteration >= max_iter → Phase 4 (max_iter_reached)
|
|
402
|
-
all remaining = code_defect → Phase 4 (confirmed_defects)
|
|
403
|
-
ELSE → next iteration
|
|
404
|
-
|
|
405
|
-
END OUTER
|
|
406
|
-
```
|
|
407
|
-
|
|
408
|
-
**Diagnosis Agent Instruction** (per failure row):
|
|
409
|
-
```
|
|
410
|
-
You are a test failure diagnostician. Classify ONE test failure and provide fix if applicable.
|
|
411
|
-
|
|
412
|
-
## Task
|
|
413
|
-
- Read test_file and target_file to understand the failure context
|
|
414
|
-
- Analyze error_detail against expected vs actual
|
|
415
|
-
- Classify the failure:
|
|
416
|
-
- test_defect: Test is wrong (bad import, wrong endpoint, bad fixture, incorrect assertion)
|
|
417
|
-
- code_defect: Source code violates business rule (actual behavior != expected requirement)
|
|
418
|
-
- env_issue: Environment problem (service down, config missing, timeout)
|
|
419
|
-
|
|
420
|
-
## Output
|
|
421
|
-
- classification: one of test_defect / code_defect / env_issue
|
|
422
|
-
- fix_code: If test_defect, provide the fix (format: "old_line → new_line" or full replacement)
|
|
423
|
-
If code_defect or env_issue, leave empty
|
|
424
|
-
- evidence: file:line references supporting your classification
|
|
425
|
-
- error: only if you cannot determine classification
|
|
426
|
-
|
|
427
|
-
## Rules
|
|
428
|
-
- NEVER suggest source code changes — only test code fixes for test_defect
|
|
429
|
-
- A test that correctly catches a real bug is a code_defect, not test_defect
|
|
430
|
-
- When uncertain between test_defect and code_defect, prefer code_defect (conservative)
|
|
431
|
-
```
|
|
432
|
-
|
|
433
|
-
### Shared Discovery Board Protocol
|
|
434
|
-
|
|
435
|
-
| Type | Dedup Key | Data Schema |
|
|
436
|
-
|------|-----------|-------------|
|
|
437
|
-
| `test_pattern` | `data.name` | `{name, file, description}` |
|
|
438
|
-
| `mock_setup` | `data.target` | `{target, setup_code, file}` |
|
|
439
|
-
| `fixture` | `data.name` | `{name, schema, file}` |
|
|
440
|
-
| `convention` | singleton | `{describe_style, assertion_lib, import_pattern}` |
|
|
441
|
-
| `blocker` | `data.issue` | `{issue, severity, layer}` |
|
|
442
|
-
|
|
443
|
-
Read before writing tests. Append-only. Dedup by type+key.
|
|
444
|
-
|
|
445
|
-
### Phase 4: Results & Routing
|
|
446
|
-
|
|
447
|
-
1. Export final `scenarios.csv` as `results.csv`
|
|
448
|
-
|
|
449
|
-
2. Write `.tests/auto-test/state.json`:
|
|
450
|
-
```json
|
|
451
|
-
{
|
|
452
|
-
"session_id": "auto-test-{YYYYMMDD-HHmmss}",
|
|
453
|
-
"phase": "{phase}", "phase_dir": "{PHASE_DIR}",
|
|
454
|
-
"source_route": "spec|gap|code|re-run",
|
|
455
|
-
"status": "converged|max_iter_reached|confirmed_defects|single_pass",
|
|
456
|
-
"iteration": N, "strategy": "conservative",
|
|
457
|
-
"strategy_history": [...],
|
|
458
|
-
"threshold": 95, "current_layer": "L2",
|
|
459
|
-
"layer_state": {
|
|
460
|
-
"L0": { "inner_iter": 1, "pass_rate": 100.0, "status": "passed" },
|
|
461
|
-
"L1": { "inner_iter": 2, "pass_rate": 95.0, "status": "passed" },
|
|
462
|
-
"L2": { ... }, "L3": { ... }
|
|
463
|
-
},
|
|
464
|
-
"pass_rate_history": [...],
|
|
465
|
-
"scenario_count": 30,
|
|
466
|
-
"csv_session": ".tests/auto-test/.csv-session/"
|
|
467
|
-
}
|
|
468
|
-
```
|
|
469
|
-
|
|
470
|
-
3. Write `.tests/auto-test/report.json` (same schema as workflow reference)
|
|
471
|
-
|
|
472
|
-
4. **Conditional: Traceability** (spec source only) — build REQ→AC→scenario→result mapping
|
|
473
|
-
|
|
474
|
-
5. **Conditional: Issue Creation** (code_defect failures):
|
|
475
|
-
```
|
|
476
|
-
FOR each failure WHERE classification == "code_defect":
|
|
477
|
-
Append to .workflow/issues/issues.jsonl
|
|
478
|
-
```
|
|
479
|
-
|
|
480
|
-
6. **Register artifact** in state.json:
|
|
481
|
-
```json
|
|
482
|
-
{ "id": "TST-NNN", "type": "test", "status": "completed|failed" }
|
|
483
|
-
```
|
|
484
|
-
|
|
485
|
-
7. **Report Display:**
|
|
486
|
-
```
|
|
487
|
-
=== AUTO-TEST RESULTS ===
|
|
488
|
-
阶段: {phase_name}
|
|
489
|
-
来源: {ROUTE}
|
|
490
|
-
迭代: {N} (策略: {strategy_history})
|
|
491
|
-
收敛: {status} ({final_pass_rate}%)
|
|
492
|
-
|
|
493
|
-
层级结果:
|
|
494
|
-
L0 Static: {pass_rate}% ({passed}/{total})
|
|
495
|
-
L1 Unit/API: {pass_rate}% ({passed}/{total})
|
|
496
|
-
L2 Integration: {pass_rate}% ({passed}/{total})
|
|
497
|
-
L3 E2E: {pass_rate}% ({passed}/{total})
|
|
498
|
-
|
|
499
|
-
场景: {passed} passed, {failed} failed, {blocked} blocked
|
|
500
|
-
Bugs: {N} discovered
|
|
501
|
-
{IF spec: "需求覆盖: {pct}% | 已验证: {n}/{total}"}
|
|
502
|
-
|
|
503
|
-
CSV Session: .tests/auto-test/.csv-session/
|
|
504
|
-
```
|
|
505
|
-
|
|
506
|
-
8. **Route:**
|
|
507
|
-
|
|
508
|
-
| Result | Next Step |
|
|
509
|
-
|--------|-----------|
|
|
510
|
-
| Converged (>=95%) | `$maestro-verify "{phase}"` |
|
|
511
|
-
| All requirements verified (spec) | `$maestro-milestone-audit` |
|
|
512
|
-
| Bugs discovered | `$quality-debug "--from-auto-test {phase}"` |
|
|
513
|
-
| Max iter, >80% | `$quality-test "{phase}"` for manual UAT |
|
|
514
|
-
| Max iter, <80% | `$quality-debug "{phase}"` |
|
|
515
|
-
| Coverage still low | `$quality-auto-test "{phase} --layer {missing}"` |
|
|
516
|
-
| Re-run all pass | `$maestro-verify "{phase}"` |
|
|
517
|
-
| Single pass, all pass | `$quality-test "{phase}"` |
|
|
518
|
-
|
|
519
|
-
</execution>
|
|
520
|
-
|
|
521
|
-
<error_codes>
|
|
522
|
-
| Error | Resolution |
|
|
523
|
-
|-------|------------|
|
|
524
|
-
| Phase not found in artifact registry | Abort: "Phase {N} not found" |
|
|
525
|
-
| No test framework detected | Abort: E003. Install framework or configure runner |
|
|
526
|
-
| Agent spawn fails (write or diagnosis) | Retry once, then mark scenario as blocked |
|
|
527
|
-
| Convergence not met after max_iter | Report max_iter_reached, suggest debug |
|
|
528
|
-
| All scenarios in layer blocked | Stop layer, report env_issue |
|
|
529
|
-
| CSV parse error | Validate format, show line |
|
|
530
|
-
| discoveries.ndjson corrupt | Ignore malformed lines, continue |
|
|
531
|
-
| Resume: no session found | Start fresh |
|
|
532
|
-
</error_codes>
|
|
533
|
-
|
|
534
|
-
<success_criteria>
|
|
535
|
-
- [ ] Session folder created with valid scenarios.csv
|
|
536
|
-
- [ ] Route auto-selected from project state (spec/gap/code)
|
|
537
|
-
- [ ] All layers executed in order with fail-fast on critical
|
|
538
|
-
- [ ] Test writing parallelized via spawn_agents_on_csv per layer
|
|
539
|
-
- [ ] Failure diagnosis parallelized via spawn_agents_on_csv per iteration
|
|
540
|
-
- [ ] discoveries.ndjson append-only throughout
|
|
541
|
-
- [ ] Cross-layer context propagation via prev_context column
|
|
542
|
-
- [ ] Iteration engine ran (inner: test_defect fix, outer: strategy adjust)
|
|
543
|
-
- [ ] state.json, report.json, reflection-log.md written
|
|
544
|
-
- [ ] If spec: traceability.md produced
|
|
545
|
-
- [ ] If failures: issues auto-created in issues.jsonl
|
|
546
|
-
- [ ] Next step routed based on convergence status
|
|
547
|
-
</success_criteria>
|
|
1
|
+
---
|
|
2
|
+
name: quality-auto-test
|
|
3
|
+
description: Auto-generate and run tests from specs or coverage gaps
|
|
4
|
+
argument-hint: "<phase> [-y] [-c N] [--max-iter N] [--layer L0-L3] [--dry-run] [--re-run]"
|
|
5
|
+
allowed-tools: spawn_agents_on_csv, Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
<purpose>
|
|
9
|
+
Unified automated testing using `spawn_agents_on_csv` for parallel test writing and failure diagnosis. Reads project state to auto-select scenario source (PRD specs, coverage gaps, or code exploration). All sources converge into a shared CSV pipeline: discover infrastructure → plan → write tests via CSV parallel → execute per layer → diagnose failures via CSV parallel → iterate → report.
|
|
10
|
+
|
|
11
|
+
**Core workflow**: Route → Source Scenarios → Build CSV → Layer-by-Layer Parallel Execution → Iterate (Diagnose CSV) → Report
|
|
12
|
+
|
|
13
|
+
**Topology**: Layers as waves (L0→L1→L2→L3 sequential, scenarios within layer parallel)
|
|
14
|
+
|
|
15
|
+
```
|
|
16
|
+
+---------------------------------------------------------------------------+
|
|
17
|
+
| AUTO-TEST CSV LAYER PIPELINE |
|
|
18
|
+
+---------------------------------------------------------------------------+
|
|
19
|
+
| |
|
|
20
|
+
| Phase 1: Route & Plan -> CSV |
|
|
21
|
+
| +-- Read project state, auto-select route (spec/gap/code) |
|
|
22
|
+
| +-- Extract scenarios per route, normalize to unified format |
|
|
23
|
+
| +-- Discover test infrastructure (framework, patterns) |
|
|
24
|
+
| +-- Build scenarios.csv with one row per scenario |
|
|
25
|
+
| +-- Layers = waves (L1, L2, L3 sequential; L0 = static pre-check) |
|
|
26
|
+
| +-- User validates test plan (skip if -y) |
|
|
27
|
+
| |
|
|
28
|
+
| Phase 2: Layer Execution Engine (write + run) |
|
|
29
|
+
| +-- L0: Static analysis (tsc + eslint) — no CSV needed |
|
|
30
|
+
| +-- For each layer L1→L3 (sequential, fail-fast on critical): |
|
|
31
|
+
| | +-- Layer N: Write Tests (parallel via spawn_agents_on_csv) |
|
|
32
|
+
| | | +-- Each agent writes one test file (RED-GREEN) |
|
|
33
|
+
| | | +-- Agent reads target source + infrastructure patterns |
|
|
34
|
+
| | | +-- Agent verifies RED (run test, check it targets behavior) |
|
|
35
|
+
| | | +-- Results: test_file written, red_result, findings |
|
|
36
|
+
| | +-- Merge write-results into master scenarios.csv |
|
|
37
|
+
| | +-- Run all layer tests together (full layer execution) |
|
|
38
|
+
| | +-- Record per-scenario pass/fail results |
|
|
39
|
+
| |
|
|
40
|
+
| Phase 3: Iteration Engine (diagnose + fix) |
|
|
41
|
+
| +-- OUTER LOOP (max_iter iterations): |
|
|
42
|
+
| | +-- For each layer with failures: |
|
|
43
|
+
| | | +-- Build diagnosis.csv from failed scenarios |
|
|
44
|
+
| | | +-- Diagnose & Fix (parallel via spawn_agents_on_csv) |
|
|
45
|
+
| | | | +-- Each agent classifies one failure cluster |
|
|
46
|
+
| | | | +-- test_defect: agent provides fix diff |
|
|
47
|
+
| | | | +-- code_defect: agent documents evidence |
|
|
48
|
+
| | | +-- Apply test_defect fixes, re-run layer |
|
|
49
|
+
| | +-- Reflect: analyze trends, log strategy |
|
|
50
|
+
| | +-- Adjust: select next strategy (conservative/aggressive/...) |
|
|
51
|
+
| | +-- Convergence check: >=95% → done |
|
|
52
|
+
| +-- discoveries.ndjson shared across all iterations |
|
|
53
|
+
| |
|
|
54
|
+
| Phase 4: Results & Routing |
|
|
55
|
+
| +-- Export results.csv |
|
|
56
|
+
| +-- Write report.json, state.json, reflection-log.md |
|
|
57
|
+
| +-- Conditional: traceability.md, issue creation |
|
|
58
|
+
| +-- Route to next step based on convergence |
|
|
59
|
+
| |
|
|
60
|
+
+---------------------------------------------------------------------------+
|
|
61
|
+
```
|
|
62
|
+
</purpose>
|
|
63
|
+
|
|
64
|
+
<context>
|
|
65
|
+
```bash
|
|
66
|
+
$quality-auto-test "3" # auto-detect source, full iteration
|
|
67
|
+
$quality-auto-test -c 4 "3" # max 4 concurrent test writers per layer
|
|
68
|
+
$quality-auto-test -y "3 --max-iter 1" # single-pass generation only
|
|
69
|
+
$quality-auto-test "3 --dry-run" # plan only, no execution
|
|
70
|
+
$quality-auto-test "3 --re-run" # re-run only previously failed scenarios
|
|
71
|
+
$quality-auto-test "3 --layer L2" # restrict to L2 integration tests
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
**Flags**:
|
|
75
|
+
- `-y, --yes`: Skip all confirmations
|
|
76
|
+
- `-c, --concurrency N`: Max concurrent agents within each layer (default: 5)
|
|
77
|
+
- `--max-iter N`: Max outer iterations (default 5). **1 = single-pass** generation only
|
|
78
|
+
- `--layer L`: Start from or restrict to specific layer (L0|L1|L2|L3)
|
|
79
|
+
- `--strategy conservative|aggressive|surgical|reflective`: Override starting iteration strategy (default: auto-selected)
|
|
80
|
+
- `--dry-run`: Generate test plan only, do not execute
|
|
81
|
+
- `--re-run`: Re-run only previously failed/blocked scenarios
|
|
82
|
+
|
|
83
|
+
**Intelligent routing** (auto-detected, priority order):
|
|
84
|
+
|
|
85
|
+
| Priority | Condition | Route |
|
|
86
|
+
|----------|-----------|-------|
|
|
87
|
+
| 1 | Active session (state.json status=running) | Resume |
|
|
88
|
+
| 2 | --re-run flag + previous failures | Re-run |
|
|
89
|
+
| 3 | Spec package exists (REQ-*.md) | spec (PRD-forward) |
|
|
90
|
+
| 4 | Nyquist gaps exist (verification.json) | gap (coverage-forward) |
|
|
91
|
+
| 5 | Default | code (exploration-forward) |
|
|
92
|
+
|
|
93
|
+
**Session Directory**: `.tests/auto-test/.csv-session/`
|
|
94
|
+
**Core Output**: `scenarios.csv` (master state) + `results.csv` (final) + `discoveries.ndjson` (shared) + `report.json` + `state.json`
|
|
95
|
+
</context>
|
|
96
|
+
|
|
97
|
+
<csv_schema>
|
|
98
|
+
|
|
99
|
+
### scenarios.csv (Master State — Test Writing Phase)
|
|
100
|
+
|
|
101
|
+
```csv
|
|
102
|
+
id,name,layer,priority,category,target_file,test_file,description,test_cases,fixtures,req_ref,infrastructure_hints,prev_context,status,red_result,findings,error
|
|
103
|
+
"AT-001","Auth token validation","L1","critical","api_contract","src/auth/token.ts","src/auth/__tests__/token.test.ts","Validate JWT token verification returns correct payload","verify valid token returns payload;verify expired token throws;verify malformed token throws","valid_token fixture;expired_token fixture","REQ-001:AC-1","vitest;describe/it pattern;see src/utils/__tests__/hash.test.ts","","","","",""
|
|
104
|
+
"AT-002","Login endpoint integration","L2","high","business_rule","src/routes/login.ts","src/routes/__tests__/login.integration.test.ts","POST /api/login returns JWT on valid credentials","valid login returns 200+token;invalid password returns 401;missing email returns 400","user_fixture;credentials_fixture","REQ-002:AC-1","supertest;see src/routes/__tests__/health.test.ts","AT-001 findings: token module exports verifyToken/generateToken","","","",""
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
**Columns**:
|
|
108
|
+
|
|
109
|
+
| Column | Phase | Description |
|
|
110
|
+
|--------|-------|-------------|
|
|
111
|
+
| `id` | Input | Scenario ID (AT-NNN format) |
|
|
112
|
+
| `name` | Input | Short scenario name |
|
|
113
|
+
| `layer` | Input | L1/L2/L3 (determines wave order) |
|
|
114
|
+
| `priority` | Input | critical/high/medium |
|
|
115
|
+
| `category` | Input | api_contract/business_rule/state_transition/user_flow/... |
|
|
116
|
+
| `target_file` | Input | Source file being tested |
|
|
117
|
+
| `test_file` | Input | Target test file path to create |
|
|
118
|
+
| `description` | Input | What this scenario validates |
|
|
119
|
+
| `test_cases` | Input | Semicolon-separated test cases |
|
|
120
|
+
| `fixtures` | Input | Required fixtures/mocks (semicolon-separated) |
|
|
121
|
+
| `req_ref` | Input | Requirement reference (REQ-NNN:AC-N or gap-id or empty) |
|
|
122
|
+
| `infrastructure_hints` | Input | Framework + pattern references from Step 3 |
|
|
123
|
+
| `prev_context` | Computed | Findings from prior layer scenarios (cross-layer propagation) |
|
|
124
|
+
| `status` | Output | pending → written → passed → failed → blocked |
|
|
125
|
+
| `red_result` | Output | expected_fail / unexpected_fail / pass (RED phase result) |
|
|
126
|
+
| `findings` | Output | Implementation notes, patterns discovered (max 500 chars) |
|
|
127
|
+
| `error` | Output | Error message if failed |
|
|
128
|
+
|
|
129
|
+
### diagnosis.csv (Iteration Phase — Failure Diagnosis)
|
|
130
|
+
|
|
131
|
+
```csv
|
|
132
|
+
id,scenario_id,layer,test_file,error_detail,expected,actual,target_file,source_context,classification,fix_code,evidence,error
|
|
133
|
+
"DX-001","AT-003","L1","src/auth/__tests__/token.test.ts","TypeError: verifyToken is not a function","verifyToken returns decoded payload","Function not exported from module","src/auth/token.ts","token.ts exports: generateToken only","test_defect","import { verifyToken } from '../token' → import { verifyToken } from '../verify-token'","src/auth/verify-token.ts:15 exports verifyToken",""
|
|
134
|
+
"DX-002","AT-005","L2","src/routes/__tests__/login.test.ts","Expected 200, received 500","POST /login returns 200 with valid credentials","Internal server error: database connection refused","src/routes/login.ts","login.ts calls UserModel.findByEmail","env_issue","","Database not available in test environment",""
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
**Columns**:
|
|
138
|
+
|
|
139
|
+
| Column | Phase | Description |
|
|
140
|
+
|--------|-------|-------------|
|
|
141
|
+
| `id` | Input | Diagnosis ID (DX-NNN) |
|
|
142
|
+
| `scenario_id` | Input | Reference to AT-NNN scenario |
|
|
143
|
+
| `layer` | Input | Layer where failure occurred |
|
|
144
|
+
| `test_file` | Input | Test file that failed |
|
|
145
|
+
| `error_detail` | Input | Full error message/stack trace excerpt |
|
|
146
|
+
| `expected` | Input | Expected behavior from scenario |
|
|
147
|
+
| `actual` | Input | Actual behavior observed |
|
|
148
|
+
| `target_file` | Input | Source file being tested |
|
|
149
|
+
| `source_context` | Input | Relevant source code context (exports, imports) |
|
|
150
|
+
| `classification` | Output | test_defect / code_defect / env_issue |
|
|
151
|
+
| `fix_code` | Output | Fix diff for test_defect (old → new) |
|
|
152
|
+
| `evidence` | Output | file:line references for diagnosis |
|
|
153
|
+
| `error` | Output | Agent error if diagnosis failed |
|
|
154
|
+
|
|
155
|
+
### Session Structure
|
|
156
|
+
|
|
157
|
+
```
|
|
158
|
+
.tests/auto-test/.csv-session/
|
|
159
|
+
+-- scenarios.csv (master state)
|
|
160
|
+
+-- results.csv (final export)
|
|
161
|
+
+-- discoveries.ndjson (shared across iterations)
|
|
162
|
+
+-- layer-L{N}-write.csv (temporary, per-layer write input)
|
|
163
|
+
+-- layer-L{N}-write-results.csv
|
|
164
|
+
+-- diagnosis-iter-{N}.csv (temporary, per-iteration diagnosis)
|
|
165
|
+
+-- diagnosis-iter-{N}-results.csv
|
|
166
|
+
```
|
|
167
|
+
</csv_schema>
|
|
168
|
+
|
|
169
|
+
<invariants>
|
|
170
|
+
1. **Start Immediately**: First action is session initialization
|
|
171
|
+
2. **Layer Order is Sacred**: Never execute L(N+1) before L(N) completes (fail-fast on critical)
|
|
172
|
+
3. **CSV is Source of Truth**: Master scenarios.csv holds all test execution state
|
|
173
|
+
4. **Context Propagation**: prev_context built from prior-layer findings in CSV, not memory
|
|
174
|
+
5. **Discovery Board Append-Only**: Never clear or modify discoveries.ndjson
|
|
175
|
+
6. **Route Auto-Detected**: Read state, never ask user for mode
|
|
176
|
+
7. **RED-GREEN Methodology**: Tests target real behavior; failing test = bug discovery (never fix source)
|
|
177
|
+
8. **Max 3 Inner Fix Attempts**: Per layer, fix test_defects up to 3 times via diagnosis CSV
|
|
178
|
+
9. **Convergence Threshold**: 95% pass rate = converged
|
|
179
|
+
10. **DO NOT STOP**: Continuous execution until convergence, max_iter, or all remaining = code_defect
|
|
180
|
+
</invariants>
|
|
181
|
+
|
|
182
|
+
<execution>
|
|
183
|
+
|
|
184
|
+
### Session Initialization
|
|
185
|
+
|
|
186
|
+
```
|
|
187
|
+
Parse from $ARGUMENTS:
|
|
188
|
+
AUTO_YES ← --yes | -y
|
|
189
|
+
maxConcurrency ← --concurrency | -c N (default: 5)
|
|
190
|
+
MAX_ITER ← --max-iter N (default: 5)
|
|
191
|
+
layerFilter ← --layer L (default: null = all)
|
|
192
|
+
startStrategy ← --strategy conservative|aggressive|surgical|reflective (default: null = auto)
|
|
193
|
+
dryRun ← --dry-run
|
|
194
|
+
reRun ← --re-run
|
|
195
|
+
phaseArg ← remaining text
|
|
196
|
+
|
|
197
|
+
Derive:
|
|
198
|
+
dateStr ← UTC+8 YYYYMMDD
|
|
199
|
+
sessionFolder ← ".tests/auto-test/.csv-session"
|
|
200
|
+
|
|
201
|
+
mkdir -p {sessionFolder}
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
### Phase 1: Route & Plan → CSV
|
|
205
|
+
|
|
206
|
+
#### Step 0: Parse & Load
|
|
207
|
+
|
|
208
|
+
Resolve phase dir from `state.json` artifact registry (`type='execute'`, matching phase). Error E002 if not found.
|
|
209
|
+
|
|
210
|
+
```
|
|
211
|
+
specs_test = maestro spec load --category test
|
|
212
|
+
specs_arch = maestro spec load --category arch
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
#### Step 1: Read State & Route
|
|
216
|
+
|
|
217
|
+
```
|
|
218
|
+
Priority: Resume > Re-run > Spec > Gap > Code
|
|
219
|
+
|
|
220
|
+
1. RESUME: .csv-session/scenarios.csv exists AND state.json status == "running"
|
|
221
|
+
→ offer resume or restart (resume = reload CSV, jump to current iteration)
|
|
222
|
+
|
|
223
|
+
2. RE-RUN: --re-run flag AND report.json has failed/blocked scenarios
|
|
224
|
+
→ load failed scenarios into CSV with status reset to pending
|
|
225
|
+
|
|
226
|
+
3. SPEC: .workflow/.spec/SPEC-*/requirements/REQ-*.md exists
|
|
227
|
+
→ ROUTE = "spec", SPEC_MODE = "full" | "degraded"
|
|
228
|
+
|
|
229
|
+
4. GAP: verification.json has gaps[] (MISSING/PARTIAL)
|
|
230
|
+
→ ROUTE = "gap"
|
|
231
|
+
|
|
232
|
+
5. CODE: Default fallback → ROUTE = "code"
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
#### Step 2: Source Scenarios
|
|
236
|
+
|
|
237
|
+
Execute route-specific extraction, normalize to unified format.
|
|
238
|
+
|
|
239
|
+
**Route A: spec** — Parse REQ acceptance criteria, classify layers, generate fixtures.
|
|
240
|
+
**Route B: gap** — Read verification/coverage gaps, classify files by type.
|
|
241
|
+
**Route C: code** — Explore module boundaries, API endpoints, integration points.
|
|
242
|
+
|
|
243
|
+
All routes produce unified scenario objects (see csv_schema).
|
|
244
|
+
|
|
245
|
+
#### Step 3: Discover Infrastructure
|
|
246
|
+
|
|
247
|
+
Detect framework, read 2-3 existing tests for patterns. Build `infrastructure_hints` string per scenario.
|
|
248
|
+
|
|
249
|
+
#### Step 4: Build scenarios.csv & Confirm
|
|
250
|
+
|
|
251
|
+
1. Build master `scenarios.csv` — one row per scenario, grouped by layer
|
|
252
|
+
2. Set `prev_context` for L2 scenarios from L1 scenario descriptions (cross-layer dependency)
|
|
253
|
+
3. Set `prev_context` for L3 from L2 findings
|
|
254
|
+
|
|
255
|
+
Display plan summary:
|
|
256
|
+
```
|
|
257
|
+
=== AUTO-TEST PLAN ===
|
|
258
|
+
来源: {ROUTE} | 阶段: {phase_name} | Spec: {spec_ref or "N/A"}
|
|
259
|
+
|
|
260
|
+
L0 Static: {N} checks
|
|
261
|
+
L1 Unit/API: {N} scenarios ({X} critical, {Y} high)
|
|
262
|
+
L2 Integration: {N} scenarios ({X} critical, {Y} high)
|
|
263
|
+
L3 E2E: {N} scenarios ({X} critical, {Y} high)
|
|
264
|
+
|
|
265
|
+
Total: {N} scenarios | Max iterations: {MAX_ITER} | Concurrency: {maxConcurrency}
|
|
266
|
+
Proceed? (yes/edit/cancel)
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
- `--dry-run`: stop here. `-y`: skip confirmation.
|
|
270
|
+
|
|
271
|
+
### Phase 2: Layer Execution Engine (Write + Run)
|
|
272
|
+
|
|
273
|
+
#### L0: Static Analysis (no CSV)
|
|
274
|
+
|
|
275
|
+
```bash
|
|
276
|
+
tsc --noEmit && eslint src/
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
If L0 fails → stop, do not proceed to L1.
|
|
280
|
+
|
|
281
|
+
#### Per-Layer Write Loop (L1 → L2 → L3)
|
|
282
|
+
|
|
283
|
+
For each layer (sequential, respecting --layer filter):
|
|
284
|
+
|
|
285
|
+
**1. Extract layer rows** from master `scenarios.csv` (filter by `layer == L{N}`, status == pending)
|
|
286
|
+
|
|
287
|
+
**2. Populate `prev_context`** from completed prior-layer findings in master CSV
|
|
288
|
+
|
|
289
|
+
**3. Write `layer-L{N}-write.csv`** then execute parallel test writing:
|
|
290
|
+
|
|
291
|
+
```javascript
|
|
292
|
+
spawn_agents_on_csv({
|
|
293
|
+
csv_path: `${sessionFolder}/layer-L${N}-write.csv`,
|
|
294
|
+
id_column: "id",
|
|
295
|
+
instruction: buildTestWriterInstruction(infrastructure, specsContent, phaseDir),
|
|
296
|
+
max_concurrency: maxConcurrency,
|
|
297
|
+
max_runtime_seconds: 1800,
|
|
298
|
+
output_csv_path: `${sessionFolder}/layer-L${N}-write-results.csv`,
|
|
299
|
+
output_schema: { id, status: [written|failed], red_result: [expected_fail|unexpected_fail|pass], findings, error }
|
|
300
|
+
})
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
**Test Writer Agent Instruction** (per scenario row):
|
|
304
|
+
```
|
|
305
|
+
You are a test writer. Write ONE test file for the given scenario.
|
|
306
|
+
|
|
307
|
+
## Task
|
|
308
|
+
- Read the target_file to understand the module under test
|
|
309
|
+
- Write test file at test_file path following infrastructure_hints patterns
|
|
310
|
+
- Each test case in test_cases becomes one it() block
|
|
311
|
+
- Use fixtures from fixtures column (infer from source if empty)
|
|
312
|
+
- Include scenario id in describe: describe("AT-NNN: {name}", ...)
|
|
313
|
+
- Run the test file once after writing
|
|
314
|
+
|
|
315
|
+
## RED-GREEN Rules
|
|
316
|
+
- If test PASSES immediately: note "pass" — may need strengthening
|
|
317
|
+
- If test FAILS as expected (tests real behavior): note "expected_fail" — good
|
|
318
|
+
- If test FAILS unexpectedly (setup/import error): fix test setup, note "unexpected_fail"
|
|
319
|
+
- NEVER modify source code — only write/fix test files
|
|
320
|
+
|
|
321
|
+
## Output
|
|
322
|
+
- status: "written" if test file created successfully, "failed" if unable
|
|
323
|
+
- red_result: the RED phase outcome
|
|
324
|
+
- findings: patterns discovered, notes for dependent scenarios (max 500 chars)
|
|
325
|
+
- error: only if status == "failed"
|
|
326
|
+
|
|
327
|
+
## Context
|
|
328
|
+
- prev_context: {prev_context} (findings from prior layer)
|
|
329
|
+
- Read discoveries.ndjson for shared patterns before starting
|
|
330
|
+
- Append to discoveries.ndjson if you find reusable patterns
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
**4. Merge write-results** into master `scenarios.csv`, delete temp CSV
|
|
334
|
+
|
|
335
|
+
**5. Run full layer test suite:**
|
|
336
|
+
```bash
|
|
337
|
+
{run_command} --testPathPattern="{layer_pattern}"
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
**6. Record per-scenario results** (pass/fail/blocked with error_detail)
|
|
341
|
+
|
|
342
|
+
**7. Fail-fast check:** If ANY critical-priority scenario failed → do NOT proceed to next layer
|
|
343
|
+
|
|
344
|
+
**If `--max-iter 1`:** After all layers written and run once, jump to Phase 4 (single-pass).
|
|
345
|
+
|
|
346
|
+
### Phase 3: Iteration Engine (Diagnose + Fix)
|
|
347
|
+
|
|
348
|
+
```
|
|
349
|
+
OUTER LOOP (max_iter iterations):
|
|
350
|
+
|
|
351
|
+
FOR each layer with failures (L1 through current):
|
|
352
|
+
|
|
353
|
+
INNER LOOP (max 3 per layer):
|
|
354
|
+
|
|
355
|
+
1. Build diagnosis.csv from failed scenarios in master CSV
|
|
356
|
+
(only scenarios with status=failed AND classification != code_defect)
|
|
357
|
+
|
|
358
|
+
2. IF diagnosis rows >= 1:
|
|
359
|
+
spawn_agents_on_csv({
|
|
360
|
+
csv_path: `${sessionFolder}/diagnosis-iter-${iter}.csv`,
|
|
361
|
+
id_column: "id",
|
|
362
|
+
instruction: buildDiagnosisInstruction(infrastructure),
|
|
363
|
+
max_concurrency: maxConcurrency,
|
|
364
|
+
max_runtime_seconds: 1200,
|
|
365
|
+
output_csv_path: `${sessionFolder}/diagnosis-iter-${iter}-results.csv`,
|
|
366
|
+
output_schema: { id, classification, fix_code, evidence, error }
|
|
367
|
+
})
|
|
368
|
+
|
|
369
|
+
3. Merge diagnosis results:
|
|
370
|
+
- test_defect with fix_code → apply fix, update scenario status to "pending"
|
|
371
|
+
- code_defect → mark as confirmed failure (stop retrying)
|
|
372
|
+
- env_issue → mark as blocked
|
|
373
|
+
|
|
374
|
+
4. Re-run ALL scenarios in this layer (catch regressions)
|
|
375
|
+
5. IF no test_defects remain: break inner loop
|
|
376
|
+
|
|
377
|
+
END INNER
|
|
378
|
+
|
|
379
|
+
Record final layer results
|
|
380
|
+
IF critical code_defects: stop layer progression (fail-fast)
|
|
381
|
+
|
|
382
|
+
END FOR
|
|
383
|
+
|
|
384
|
+
REFLECT:
|
|
385
|
+
Analyze: pass rate delta, failure clusters, strategy effectiveness
|
|
386
|
+
Append to reflection-log.md
|
|
387
|
+
|
|
388
|
+
ADJUST (Adaptive Strategy):
|
|
389
|
+
IF startStrategy provided AND iteration == 1: use startStrategy as initial
|
|
390
|
+
OTHERWISE auto-select:
|
|
391
|
+
|
|
392
|
+
| Condition | Strategy |
|
|
393
|
+
|-----------|----------|
|
|
394
|
+
| Iteration 1-2 | Conservative: fix obvious test_defects only |
|
|
395
|
+
| Pass rate >80% | Aggressive: batch-fix related failures |
|
|
396
|
+
| New regressions | Surgical: revert, fix regression only |
|
|
397
|
+
| Stuck 3+ iters | Reflective: re-analyze root cause pattern |
|
|
398
|
+
|
|
399
|
+
CONVERGENCE:
|
|
400
|
+
pass_rate >= 95% → Phase 4 (converged)
|
|
401
|
+
iteration >= max_iter → Phase 4 (max_iter_reached)
|
|
402
|
+
all remaining = code_defect → Phase 4 (confirmed_defects)
|
|
403
|
+
ELSE → next iteration
|
|
404
|
+
|
|
405
|
+
END OUTER
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
**Diagnosis Agent Instruction** (per failure row):
|
|
409
|
+
```
|
|
410
|
+
You are a test failure diagnostician. Classify ONE test failure and provide fix if applicable.
|
|
411
|
+
|
|
412
|
+
## Task
|
|
413
|
+
- Read test_file and target_file to understand the failure context
|
|
414
|
+
- Analyze error_detail against expected vs actual
|
|
415
|
+
- Classify the failure:
|
|
416
|
+
- test_defect: Test is wrong (bad import, wrong endpoint, bad fixture, incorrect assertion)
|
|
417
|
+
- code_defect: Source code violates business rule (actual behavior != expected requirement)
|
|
418
|
+
- env_issue: Environment problem (service down, config missing, timeout)
|
|
419
|
+
|
|
420
|
+
## Output
|
|
421
|
+
- classification: one of test_defect / code_defect / env_issue
|
|
422
|
+
- fix_code: If test_defect, provide the fix (format: "old_line → new_line" or full replacement)
|
|
423
|
+
If code_defect or env_issue, leave empty
|
|
424
|
+
- evidence: file:line references supporting your classification
|
|
425
|
+
- error: only if you cannot determine classification
|
|
426
|
+
|
|
427
|
+
## Rules
|
|
428
|
+
- NEVER suggest source code changes — only test code fixes for test_defect
|
|
429
|
+
- A test that correctly catches a real bug is a code_defect, not test_defect
|
|
430
|
+
- When uncertain between test_defect and code_defect, prefer code_defect (conservative)
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
### Shared Discovery Board Protocol
|
|
434
|
+
|
|
435
|
+
| Type | Dedup Key | Data Schema |
|
|
436
|
+
|------|-----------|-------------|
|
|
437
|
+
| `test_pattern` | `data.name` | `{name, file, description}` |
|
|
438
|
+
| `mock_setup` | `data.target` | `{target, setup_code, file}` |
|
|
439
|
+
| `fixture` | `data.name` | `{name, schema, file}` |
|
|
440
|
+
| `convention` | singleton | `{describe_style, assertion_lib, import_pattern}` |
|
|
441
|
+
| `blocker` | `data.issue` | `{issue, severity, layer}` |
|
|
442
|
+
|
|
443
|
+
Read before writing tests. Append-only. Dedup by type+key.
|
|
444
|
+
|
|
445
|
+
### Phase 4: Results & Routing
|
|
446
|
+
|
|
447
|
+
1. Export final `scenarios.csv` as `results.csv`
|
|
448
|
+
|
|
449
|
+
2. Write `.tests/auto-test/state.json`:
|
|
450
|
+
```json
|
|
451
|
+
{
|
|
452
|
+
"session_id": "auto-test-{YYYYMMDD-HHmmss}",
|
|
453
|
+
"phase": "{phase}", "phase_dir": "{PHASE_DIR}",
|
|
454
|
+
"source_route": "spec|gap|code|re-run",
|
|
455
|
+
"status": "converged|max_iter_reached|confirmed_defects|single_pass",
|
|
456
|
+
"iteration": N, "strategy": "conservative",
|
|
457
|
+
"strategy_history": [...],
|
|
458
|
+
"threshold": 95, "current_layer": "L2",
|
|
459
|
+
"layer_state": {
|
|
460
|
+
"L0": { "inner_iter": 1, "pass_rate": 100.0, "status": "passed" },
|
|
461
|
+
"L1": { "inner_iter": 2, "pass_rate": 95.0, "status": "passed" },
|
|
462
|
+
"L2": { ... }, "L3": { ... }
|
|
463
|
+
},
|
|
464
|
+
"pass_rate_history": [...],
|
|
465
|
+
"scenario_count": 30,
|
|
466
|
+
"csv_session": ".tests/auto-test/.csv-session/"
|
|
467
|
+
}
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
3. Write `.tests/auto-test/report.json` (same schema as workflow reference)
|
|
471
|
+
|
|
472
|
+
4. **Conditional: Traceability** (spec source only) — build REQ→AC→scenario→result mapping
|
|
473
|
+
|
|
474
|
+
5. **Conditional: Issue Creation** (code_defect failures):
|
|
475
|
+
```
|
|
476
|
+
FOR each failure WHERE classification == "code_defect":
|
|
477
|
+
Append to .workflow/issues/issues.jsonl
|
|
478
|
+
```
|
|
479
|
+
|
|
480
|
+
6. **Register artifact** in state.json:
|
|
481
|
+
```json
|
|
482
|
+
{ "id": "TST-NNN", "type": "test", "status": "completed|failed" }
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
7. **Report Display:**
|
|
486
|
+
```
|
|
487
|
+
=== AUTO-TEST RESULTS ===
|
|
488
|
+
阶段: {phase_name}
|
|
489
|
+
来源: {ROUTE}
|
|
490
|
+
迭代: {N} (策略: {strategy_history})
|
|
491
|
+
收敛: {status} ({final_pass_rate}%)
|
|
492
|
+
|
|
493
|
+
层级结果:
|
|
494
|
+
L0 Static: {pass_rate}% ({passed}/{total})
|
|
495
|
+
L1 Unit/API: {pass_rate}% ({passed}/{total})
|
|
496
|
+
L2 Integration: {pass_rate}% ({passed}/{total})
|
|
497
|
+
L3 E2E: {pass_rate}% ({passed}/{total})
|
|
498
|
+
|
|
499
|
+
场景: {passed} passed, {failed} failed, {blocked} blocked
|
|
500
|
+
Bugs: {N} discovered
|
|
501
|
+
{IF spec: "需求覆盖: {pct}% | 已验证: {n}/{total}"}
|
|
502
|
+
|
|
503
|
+
CSV Session: .tests/auto-test/.csv-session/
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
8. **Route:**
|
|
507
|
+
|
|
508
|
+
| Result | Next Step |
|
|
509
|
+
|--------|-----------|
|
|
510
|
+
| Converged (>=95%) | `$maestro-verify "{phase}"` |
|
|
511
|
+
| All requirements verified (spec) | `$maestro-milestone-audit` |
|
|
512
|
+
| Bugs discovered | `$quality-debug "--from-auto-test {phase}"` |
|
|
513
|
+
| Max iter, >80% | `$quality-test "{phase}"` for manual UAT |
|
|
514
|
+
| Max iter, <80% | `$quality-debug "{phase}"` |
|
|
515
|
+
| Coverage still low | `$quality-auto-test "{phase} --layer {missing}"` |
|
|
516
|
+
| Re-run all pass | `$maestro-verify "{phase}"` |
|
|
517
|
+
| Single pass, all pass | `$quality-test "{phase}"` |
|
|
518
|
+
|
|
519
|
+
</execution>
|
|
520
|
+
|
|
521
|
+
<error_codes>
|
|
522
|
+
| Error | Resolution |
|
|
523
|
+
|-------|------------|
|
|
524
|
+
| Phase not found in artifact registry | Abort: "Phase {N} not found" |
|
|
525
|
+
| No test framework detected | Abort: E003. Install framework or configure runner |
|
|
526
|
+
| Agent spawn fails (write or diagnosis) | Retry once, then mark scenario as blocked |
|
|
527
|
+
| Convergence not met after max_iter | Report max_iter_reached, suggest debug |
|
|
528
|
+
| All scenarios in layer blocked | Stop layer, report env_issue |
|
|
529
|
+
| CSV parse error | Validate format, show line |
|
|
530
|
+
| discoveries.ndjson corrupt | Ignore malformed lines, continue |
|
|
531
|
+
| Resume: no session found | Start fresh |
|
|
532
|
+
</error_codes>
|
|
533
|
+
|
|
534
|
+
<success_criteria>
|
|
535
|
+
- [ ] Session folder created with valid scenarios.csv
|
|
536
|
+
- [ ] Route auto-selected from project state (spec/gap/code)
|
|
537
|
+
- [ ] All layers executed in order with fail-fast on critical
|
|
538
|
+
- [ ] Test writing parallelized via spawn_agents_on_csv per layer
|
|
539
|
+
- [ ] Failure diagnosis parallelized via spawn_agents_on_csv per iteration
|
|
540
|
+
- [ ] discoveries.ndjson append-only throughout
|
|
541
|
+
- [ ] Cross-layer context propagation via prev_context column
|
|
542
|
+
- [ ] Iteration engine ran (inner: test_defect fix, outer: strategy adjust)
|
|
543
|
+
- [ ] state.json, report.json, reflection-log.md written
|
|
544
|
+
- [ ] If spec: traceability.md produced
|
|
545
|
+
- [ ] If failures: issues auto-created in issues.jsonl
|
|
546
|
+
- [ ] Next step routed based on convergence status
|
|
547
|
+
</success_criteria>
|