rungate 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/aditi.md +47 -0
- package/.claude/agents/marcus.md +50 -0
- package/.claude/agents/quinn.md +58 -0
- package/.claude/agents/rook.md +47 -0
- package/.claude/agents/serena.md +46 -0
- package/.claude/rungate.json +11 -0
- package/.fallowrc.json +28 -0
- package/.github/copilot-instructions.md +20 -0
- package/.rungate/conformity-findings.json +648 -0
- package/AGENTS.md +106 -0
- package/CODE-MAP.md +50 -0
- package/HARNESS.md +88 -0
- package/bun.lock +26 -0
- package/bunfig.toml +2 -0
- package/config/brief-policies.json +28 -0
- package/config/ceremony-profiles.json +212 -0
- package/config/consumer-cascade-map.json +24 -0
- package/config/doc-cascade-map.json +24 -0
- package/config/eval.json +84 -0
- package/config/profiles/account-plan.yaml +36 -0
- package/config/profiles/campaign.yaml +94 -0
- package/config/prove-evidence.schema.json +65 -0
- package/config/spec-policies.json +36 -0
- package/config/workflow-schema.json +281 -0
- package/gates/SCHEMA-GUIDE.md +121 -0
- package/gates/ac-quality.test.ts +106 -0
- package/gates/adversarial.test.ts +274 -0
- package/gates/brief-assembler.test.ts +429 -0
- package/gates/brief-assembler.ts +334 -0
- package/gates/bunfig.toml +2 -0
- package/gates/chain.test.ts +91 -0
- package/gates/e2e-smoke.test.ts +234 -0
- package/gates/error-classifier.test.ts +249 -0
- package/gates/error-classifier.ts +89 -0
- package/gates/orchestrator.test.ts +950 -0
- package/gates/orchestrator.ts +569 -0
- package/gates/preload.ts +77 -0
- package/gates/prompts/ac-adversary.md +20 -0
- package/gates/prompts/evidence-validator.md +22 -0
- package/gates/prompts/prove-reproducer.md +53 -0
- package/gates/prove.test.ts +164 -0
- package/gates/run-gate.ts +936 -0
- package/gates/schema-parity.test.ts +197 -0
- package/gates/schema.ts +362 -0
- package/gates/self-heal.test.ts +511 -0
- package/gates/self-heal.ts +121 -0
- package/gates/ship-orchestrator.test.ts +379 -0
- package/gates/ship-orchestrator.ts +363 -0
- package/gates/test-fixtures/broken-no-environments.json +60 -0
- package/gates/test-fixtures/valid-standard.json +54 -0
- package/gates/test-utils.ts +43 -0
- package/gates/witness.ts +220 -0
- package/gates/workflow.test.ts +1053 -0
- package/hooks/AgentBriefGuard.hook.ts +586 -0
- package/hooks/AgentVerdictCapture.hook.ts +134 -0
- package/hooks/AutoVerifyGate.hook.ts +76 -0
- package/hooks/GateEnforcement.hook.ts +273 -0
- package/hooks/IssueCloseGuard.hook.ts +116 -0
- package/hooks/MergeGuard.hook.ts +55 -0
- package/hooks/StaleTTLCleanup.hook.ts +133 -0
- package/hooks/VerifyPhaseLock.hook.ts +81 -0
- package/hooks/WorkflowStateGuard.hook.ts +37 -0
- package/hooks/lib/agentDetection.ts +70 -0
- package/hooks/lib/findWorkflow.ts +101 -0
- package/hooks/lib/parseStdin.ts +37 -0
- package/hooks/lib/paths.ts +111 -0
- package/hooks/lib/utils.ts +81 -0
- package/lib/conformity.ts +959 -0
- package/lib/paths.ts +23 -0
- package/lib/rungate-schema.ts +60 -0
- package/lib/signal-phrases.ts +7 -0
- package/package.json +20 -0
- package/prompts/ac-adversary.md +20 -0
- package/prompts/ac-format.md +34 -0
- package/prompts/aditi.md +29 -0
- package/prompts/blast-radius.md +29 -0
- package/prompts/coding-principles.md +34 -0
- package/prompts/container-rebuild.md +29 -0
- package/prompts/container-verify.md +28 -0
- package/prompts/discovery.md +27 -0
- package/prompts/environment.md +28 -0
- package/prompts/escalation-decision-tree.md +38 -0
- package/prompts/evidence-hierarchy.md +31 -0
- package/prompts/evidence-validator.md +22 -0
- package/prompts/marcus.md +29 -0
- package/prompts/prevention.md +26 -0
- package/prompts/prove-reproducer.md +43 -0
- package/prompts/quinn-decision-tree.md +31 -0
- package/prompts/quinn-ui-brief.md +106 -0
- package/prompts/quinn.md +29 -0
- package/prompts/rca.md +28 -0
- package/prompts/read-before-write.md +30 -0
- package/prompts/regression.md +26 -0
- package/prompts/rook.md +29 -0
- package/prompts/serena.md +28 -0
- package/prompts/testing-strategy.md +34 -0
- package/reference/migration-manifest.json +554 -0
- package/reference/prompts/agent-aditi.md +38 -0
- package/reference/prompts/agent-marcus.md +38 -0
- package/reference/prompts/agent-quinn.md +38 -0
- package/reference/prompts/agent-rook.md +37 -0
- package/reference/prompts/agent-serena.md +37 -0
- package/reference/prompts/council-skill.md +170 -0
- package/reference/prompts/goal-skill.md +235 -0
- package/reference/prompts/prove-skill.md +283 -0
- package/reference/prompts/quinn-standard.md +246 -0
- package/reference/prompts/ship-convergence.md +65 -0
- package/reference/prompts/ship-escalation.md +43 -0
- package/reference/prompts/ship-iteration.md +382 -0
- package/reference/prompts/ship-pending-patches.md +246 -0
- package/reference/prompts/ship-reference.md +250 -0
- package/reference/prompts/ship-skill.md +347 -0
- package/reference/prompts/ship-templates.md +215 -0
- package/reference/scripts/batch-status.sh +68 -0
- package/reference/scripts/chain-compat-check.sh +67 -0
- package/reference/scripts/context-audit.sh +314 -0
- package/reference/scripts/dead-export-check.sh +61 -0
- package/reference/scripts/generate-qc-tables.sh +92 -0
- package/reference/scripts/harness-init.sh +111 -0
- package/reference/scripts/pai-054-baseline-check.sh +40 -0
- package/reference/scripts/prove-smoke-test.sh +112 -0
- package/reference/scripts/setup-keychain-salt.sh +12 -0
- package/reference/scripts/skill-optimizer-friday.sh +88 -0
- package/reference/scripts/skill-qc-test.sh +351 -0
- package/reference/scripts/skill-synthesize-tonight.sh +66 -0
- package/reference/specs/HARNESS-EXTRACTION-SPEC.md +263 -0
- package/reference/specs/harness-v3-bash-deletion-plan.md +65 -0
- package/reference/specs/harness-v3-migration-plan.md +188 -0
- package/scripts/decision-reconcile.sh +74 -0
- package/scripts/decision-reconcile.ts +152 -0
- package/scripts/extract-constraints.ts +260 -0
- package/scripts/generate-code-map.ts +339 -0
- package/scripts/git-hooks/pre-commit +76 -0
- package/scripts/git-hooks/pre-push +271 -0
- package/scripts/harness-skill-check.sh +52 -0
- package/scripts/lib/gap-emit.sh +65 -0
- package/scripts/lib/jq-update.sh +30 -0
- package/scripts/lib/resolve-slug.sh +30 -0
- package/scripts/lib/verify-spec-compliance.sh +102 -0
- package/scripts/prove-backfill.sh +87 -0
- package/scripts/scaffold-project.ts +1488 -0
- package/scripts/scaffold-rungate-config.ts +91 -0
- package/scripts/self-containment-check.sh +109 -0
- package/scripts/skill-qc-validator.sh +402 -0
- package/scripts/slug-resolver.sh +14 -0
- package/scripts/sync-spec-tests.ts +236 -0
- package/specs/BOOTSTRAP-DATA-FLOW-SPEC.md +2493 -0
- package/specs/BOOTSTRAP-TEST-PLAN.md +571 -0
- package/specs/HARNESS-GATES.md +256 -0
- package/specs/HARNESS-SKILL-CHAIN.md +286 -0
- package/specs/HARNESS-SKILL-CONTRACT.md +93 -0
- package/specs/HARNESS-STANDARD.md +488 -0
- package/specs/SPEC-TEMPLATE.md +53 -0
- package/specs/bootstrap-data-flow.png +0 -0
- package/specs/harness-automation-matrix.md +314 -0
- package/specs/harness-automation-matrix.rules.json +64 -0
- package/test/anti.test.ts +80 -0
- package/test/contract-negative.test.ts +86 -0
- package/test/contract.test.ts +158 -0
- package/test/external-deps.test.ts +48 -0
- package/test/fixtures/golden-project/.env.example +3 -0
- package/test/fixtures/golden-project/Makefile +7 -0
- package/test/fixtures/golden-project/docs/adr/ADR-001-framework.md +23 -0
- package/test/fixtures/golden-project/docs/setup-guide.md +10 -0
- package/test/fixtures/golden-project/package.json +10 -0
- package/test/fixtures/golden-project/specs/api-spec.md +20 -0
- package/test/fixtures/golden-project/src/auth.ts +8 -0
- package/test/fixtures/golden-project/src/db.ts +11 -0
- package/test/fixtures/golden-project/src/index.ts +19 -0
- package/test/fixtures/golden-project/src/utils/format.ts +3 -0
- package/test/fixtures/golden-project/test/auth.test.ts +15 -0
- package/test/fixtures/golden-project/tsconfig.json +12 -0
- package/test/meta-sc-coverage.test.ts +159 -0
- package/test/phase-0.test.ts +846 -0
- package/test/phase-1.test.ts +617 -0
- package/test/phase-2.test.ts +620 -0
- package/test/phase-3.test.ts +99 -0
- package/test/scaffold-conformity.test.ts +8 -0
- package/test/schema-canary.test.ts +51 -0
- package/test/spec-compliance-auto.test.ts +64 -0
- package/test/spec-compliance.test.ts +295 -0
- package/test/spec-discovery.test.ts +68 -0
- package/test/spec-drift.test.ts +5 -0
- package/test/structure.test.ts +165 -0
- package/test/unit/agent-brief-guard-ship.test.ts +293 -0
- package/test/unit/extract-constraints.test.ts +145 -0
- package/test/unit/issue-close-guard.test.ts +204 -0
- package/test/unit/scaffold-project.test.ts +417 -0
- package/test/unit/skill-cooccurrence.test.ts +177 -0
- package/test/unit/skill-sequence-logger.test.ts +244 -0
- package/test/unit/workflow-state-guard.test.ts +95 -0
- package/workflows/batch-ship.js +295 -0
- package/workflows/council.js +481 -0
- package/workflows/prove.js +560 -0
- package/workflows/ship.js +789 -0
- package/workflows/verify.js +236 -0
|
@@ -0,0 +1,571 @@
|
|
|
1
|
+
---
|
|
2
|
+
doc-type: spec
|
|
3
|
+
status: draft
|
|
4
|
+
owner: jason
|
|
5
|
+
created: 2026-09-18
|
|
6
|
+
updated: 2026-09-18
|
|
7
|
+
governs: Test strategy for BOOTSTRAP-DATA-FLOW-SPEC.md — verification approach, phased implementation, golden fixture, content assertions
|
|
8
|
+
testable: true
|
|
9
|
+
parent-spec: BOOTSTRAP-DATA-FLOW-SPEC.md
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# Bootstrap Data Flow Test Plan
|
|
13
|
+
|
|
14
|
+
## Problem Statement
|
|
15
|
+
|
|
16
|
+
Specs get implemented halfway. Past experience shows that "file exists" tests pass with empty files, principles get listed but not explained, brief ordering goes wrong, and "close enough" implementations pass review. 163 SCs require a mechanical verification strategy that catches incomplete delivery at content level, not just structural level.
|
|
17
|
+
|
|
18
|
+
## Document Chain
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
BOOTSTRAP-DATA-FLOW-SPEC.md (WHAT to build — 163 SCs)
|
|
22
|
+
↓ governs
|
|
23
|
+
BOOTSTRAP-TEST-PLAN.md (HOW to verify — this document, 14 TPs)
|
|
24
|
+
↓ agent generates (reads SC text → writes test assertion)
|
|
25
|
+
test/phase-0.test.ts (scaffold output content verification)
|
|
26
|
+
test/phase-1.test.ts (knowledge extraction + doc hygiene + findings pipeline)
|
|
27
|
+
test/phase-2.test.ts (gate enforcement)
|
|
28
|
+
test/phase-3.test.ts (parallel work)
|
|
29
|
+
test/meta-sc-coverage.test.ts (catches untested SCs — triggers agent to write tests)
|
|
30
|
+
↓ runs against
|
|
31
|
+
test/fixtures/golden-project/ (synthetic project with known structure)
|
|
32
|
+
↓ scaffold produces
|
|
33
|
+
/tmp/rungate-phase{N}-test/ (ephemeral output, verified per phase)
|
|
34
|
+
↓ conformity writes
|
|
35
|
+
.rungate/conformity-findings.json (unified: findings + candidates + staleness)
|
|
36
|
+
↓ gate runner reads
|
|
37
|
+
workflow-state.json (conformityFindings field — agents read fix commands)
|
|
38
|
+
↓ agent executes
|
|
39
|
+
fixCommands → re-runs bun test → PASS
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
**Automation loop:** When spec changes → drift guard FAILS → meta-test lists untested SCs → agent writes tests for them → agent updates hash. No manual steps. The meta-test is the trigger that tells the agent "these SCs need tests."
|
|
43
|
+
|
|
44
|
+
When a spec changes (fully automated — no manual steps):
|
|
45
|
+
1. BOOTSTRAP-DATA-FLOW-SPEC.md updated → spec-drift hash triggers test FAIL
|
|
46
|
+
2. Meta-test lists untested SCs by ID
|
|
47
|
+
3. Agent reads each untested SC text from the spec
|
|
48
|
+
4. Agent writes test assertion (SC text IS the test specification — "SC-40: .gitignore has all security entries" → `expect(content).toContain("*.pem")`)
|
|
49
|
+
5. Agent runs tests → RED (implementation needed) or GREEN (already implemented)
|
|
50
|
+
6. Agent updates spec hash in test files
|
|
51
|
+
7. If RED → agent implements the feature → tests GREEN
|
|
52
|
+
8. Conformity findings JSON regenerated with current state
|
|
53
|
+
|
|
54
|
+
The SC text is the test oracle. An agent that can read "SC-40: .gitignore has full security template" can write the test assertion. This was proven in the first session — 85 tests written from SC text by the agent.
|
|
55
|
+
|
|
56
|
+
Nothing is verbal — every claim in the spec has a test, every test runs against real output, every detection result persists in a machine-readable file.
|
|
57
|
+
|
|
58
|
+
## Design Decisions
|
|
59
|
+
|
|
60
|
+
| Decision | What | Rationale |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| TD-1 | Golden test fixture as primary oracle | Output-level diffing catches everything: missing content, wrong order, missing examples. A file-exists test passes with an empty file; a golden-diff test doesn't. |
|
|
63
|
+
| TD-2 | Content-assertion tests per SC (not file-existence) | Each SC gets a test that asserts CONTENT. `expect(content).toContain("DEEP MODULES")` not `expect(existsSync(file)).toBe(true)` |
|
|
64
|
+
| TD-3 | Phased implementation with gates between phases | Can't skip ahead. Phase N tests must pass before Phase N+1 starts. Prevents "close enough" on early phases. |
|
|
65
|
+
| TD-4 | Dogfood: ship the harness using the harness | Once brief assembly works, use the harness to ship remaining phases. If briefs produce correct agent behavior, they're correct. |
|
|
66
|
+
| TD-5 | Tests written BEFORE implementation (TDD) | All 149 SCs start RED. Implementation turns them GREEN. No test written after-the-fact to match existing behavior. |
|
|
67
|
+
| TD-6 | Spec-drift hash prevents future decay | Content hash of parent spec tracked in tests. Spec change = test FAIL until tests updated. Catches drift mechanically. |
|
|
68
|
+
| TD-7 | Golden fixture is the template for all future test plans | This approach becomes the standard for testing any spec in the harness. Fixture + content assertions + phased gates. |
|
|
69
|
+
|
|
70
|
+
## Golden Test Fixture
|
|
71
|
+
|
|
72
|
+
The fixture is a synthetic project with known, deterministic structure. Scaffold runs against it. Every output file is compared to expected output at content level.
|
|
73
|
+
|
|
74
|
+
### Fixture project structure
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
test/fixtures/golden-project/
|
|
78
|
+
├── src/
|
|
79
|
+
│ ├── index.ts # Hono app, 3 routes (GET /, GET /api/health, POST /api/data)
|
|
80
|
+
│ ├── auth.ts # Module with 2 exports (validateToken, refreshToken)
|
|
81
|
+
│ ├── db.ts # Module with 3 exports (query, insert, migrate)
|
|
82
|
+
│ └── utils/
|
|
83
|
+
│ └── format.ts # Pure utility, 1 export (formatDate)
|
|
84
|
+
├── test/
|
|
85
|
+
│ └── auth.test.ts # Existing test file (3 tests)
|
|
86
|
+
├── docs/
|
|
87
|
+
│ └── setup-guide.md # Doc with frontmatter
|
|
88
|
+
├── specs/
|
|
89
|
+
│ └── api-spec.md # Spec with testable: true, 2 SCs
|
|
90
|
+
├── docs/adr/
|
|
91
|
+
│ └── ADR-001-framework.md # ADR with status: accepted
|
|
92
|
+
├── .env.example # ENV_VAR_1=, ENV_VAR_2=, SECRET_KEY=
|
|
93
|
+
├── package.json # name: "golden-project", type: "module"
|
|
94
|
+
├── tsconfig.json # strict: true
|
|
95
|
+
├── Makefile # dev: bun run src/index.ts (port 3000)
|
|
96
|
+
└── .git/ # Initialized repo
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### Why this structure
|
|
100
|
+
|
|
101
|
+
- 3 routes → CODE-MAP must find them
|
|
102
|
+
- 2+ modules with known exports → consumer scan must detect them
|
|
103
|
+
- .env.example → envVars must populate
|
|
104
|
+
- Makefile with port → rungate.json must detect port 3000
|
|
105
|
+
- Existing tests → test baseline must capture pass/fail count
|
|
106
|
+
- Spec with SCs → spec-drift must hash it
|
|
107
|
+
- ADR → doc hygiene must validate frontmatter
|
|
108
|
+
- Utility module with 1 export → tests can check shallow module detection
|
|
109
|
+
|
|
110
|
+
### Expected output files
|
|
111
|
+
|
|
112
|
+
```
|
|
113
|
+
test/expected/
|
|
114
|
+
├── AGENTS.md # Exact expected AGENTS.md output
|
|
115
|
+
├── CODE-MAP.md # Exact expected CODE-MAP output
|
|
116
|
+
├── .claude/
|
|
117
|
+
│ ├── rungate.json # Exact expected config
|
|
118
|
+
│ └── agents/
|
|
119
|
+
│ ├── marcus.md # Full expected brief with principles, ordering, examples
|
|
120
|
+
│ ├── quinn.md # Full expected brief with journey format
|
|
121
|
+
│ ├── rook.md # Full expected brief with security checklist
|
|
122
|
+
│ ├── serena.md # Full expected brief with architecture review
|
|
123
|
+
│ └── aditi.md # Full expected brief with component spec
|
|
124
|
+
├── test/
|
|
125
|
+
│ └── scaffold-conformity.test.ts # Expected conformity test
|
|
126
|
+
├── .github/
|
|
127
|
+
│ ├── copilot-instructions.md # Expected tool bridge
|
|
128
|
+
│ └── workflows/
|
|
129
|
+
│ ├── ci.yml # Expected CI workflow
|
|
130
|
+
│ └── gates.yml # Expected gates workflow
|
|
131
|
+
├── .gitignore # Expected entries
|
|
132
|
+
├── CLAUDE.md # Expected bridge content
|
|
133
|
+
└── specs/
|
|
134
|
+
└── SPEC-TEMPLATE.md # Expected template
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
### Golden fixture test pattern
|
|
138
|
+
|
|
139
|
+
```typescript
|
|
140
|
+
import { describe, test, expect, beforeAll } from "bun:test";
|
|
141
|
+
import { readFileSync, existsSync } from "fs";
|
|
142
|
+
import { execSync } from "child_process";
|
|
143
|
+
import { join } from "path";
|
|
144
|
+
|
|
145
|
+
const FIXTURE = join(import.meta.dir, "fixtures/golden-project");
|
|
146
|
+
const EXPECTED = join(import.meta.dir, "expected");
|
|
147
|
+
const OUTPUT = "/tmp/golden-test-output";
|
|
148
|
+
|
|
149
|
+
describe("golden fixture: scaffold produces exact expected output", () => {
|
|
150
|
+
beforeAll(() => {
|
|
151
|
+
// Copy fixture to temp dir (don't modify fixture)
|
|
152
|
+
execSync(`rm -rf ${OUTPUT} && cp -r ${FIXTURE} ${OUTPUT}`);
|
|
153
|
+
// Run scaffold
|
|
154
|
+
execSync(`bun run scripts/scaffold-project.ts ${OUTPUT}`, {
|
|
155
|
+
cwd: join(import.meta.dir, ".."),
|
|
156
|
+
timeout: 30000,
|
|
157
|
+
});
|
|
158
|
+
});
|
|
159
|
+
|
|
160
|
+
// Auto-generate one test per expected file
|
|
161
|
+
const expectedFiles = execSync(`find ${EXPECTED} -type f`)
|
|
162
|
+
.toString().trim().split("\n").filter(Boolean);
|
|
163
|
+
|
|
164
|
+
for (const expectedFile of expectedFiles) {
|
|
165
|
+
const relativePath = expectedFile.replace(EXPECTED + "/", "");
|
|
166
|
+
test(`${relativePath} matches expected`, () => {
|
|
167
|
+
const outputFile = join(OUTPUT, relativePath);
|
|
168
|
+
expect(existsSync(outputFile)).toBe(true);
|
|
169
|
+
const expected = readFileSync(expectedFile, "utf-8");
|
|
170
|
+
const actual = readFileSync(outputFile, "utf-8");
|
|
171
|
+
expect(actual).toBe(expected);
|
|
172
|
+
});
|
|
173
|
+
}
|
|
174
|
+
});
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### Maintaining the golden fixture
|
|
178
|
+
|
|
179
|
+
When the spec changes:
|
|
180
|
+
1. Update the expected output files to match the new spec
|
|
181
|
+
2. Run tests → they FAIL (because implementation hasn't changed)
|
|
182
|
+
3. Update implementation → tests pass
|
|
183
|
+
4. Spec-drift hash update confirms alignment
|
|
184
|
+
|
|
185
|
+
The expected output files ARE the spec materialized. They are not generated — they are hand-written to match the spec exactly. This is intentional: generated expected output would be a tautological test.
|
|
186
|
+
|
|
187
|
+
## Content Assertion Patterns
|
|
188
|
+
|
|
189
|
+
Beyond the golden fixture, each SC gets a targeted content assertion. These are the patterns:
|
|
190
|
+
|
|
191
|
+
### File content assertions
|
|
192
|
+
|
|
193
|
+
```typescript
|
|
194
|
+
// SC-115: Coding principles doc contains required sections
|
|
195
|
+
test("SC-115: coding principles has all required sections", () => {
|
|
196
|
+
const content = readFileSync("prompts/coding-principles.md", "utf-8");
|
|
197
|
+
const required = [
|
|
198
|
+
"DEEP MODULES", "BOUNDARY VALIDATION", "EXHAUSTIVE MATCHING",
|
|
199
|
+
"IMMUTABILITY", "BRANDED TYPES", "ERROR HANDLING", "ONE EXPORT"
|
|
200
|
+
];
|
|
201
|
+
for (const section of required) {
|
|
202
|
+
expect(content.toUpperCase()).toContain(section);
|
|
203
|
+
}
|
|
204
|
+
// Must have at least one code example
|
|
205
|
+
expect(content).toMatch(/```typescript/);
|
|
206
|
+
});
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
### Brief structure assertions
|
|
210
|
+
|
|
211
|
+
```typescript
|
|
212
|
+
// SC-145: Brief section ordering
|
|
213
|
+
test("SC-145: marcus brief sections in correct order", () => {
|
|
214
|
+
const brief = readFileSync(".claude/agents/marcus.md", "utf-8");
|
|
215
|
+
const headings = [...brief.matchAll(/^## .+/gm)].map(m => m[0]);
|
|
216
|
+
|
|
217
|
+
// Identity before principles
|
|
218
|
+
const identity = headings.findIndex(h => /role|identity/i.test(h));
|
|
219
|
+
const principles = headings.findIndex(h => /principles/i.test(h));
|
|
220
|
+
const methodology = headings.findIndex(h => /methodology|coding|testing/i.test(h));
|
|
221
|
+
|
|
222
|
+
expect(identity).toBeLessThan(principles);
|
|
223
|
+
expect(principles).toBeLessThan(methodology);
|
|
224
|
+
});
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
### Three-tier rule authority assertions
|
|
228
|
+
|
|
229
|
+
```typescript
|
|
230
|
+
// SC-146: Brief has Always Do / Ask First / Never Do
|
|
231
|
+
test("SC-146: brief has three-tier rule authority", () => {
|
|
232
|
+
const brief = readFileSync(".claude/agents/marcus.md", "utf-8");
|
|
233
|
+
expect(brief).toContain("## Always Do");
|
|
234
|
+
expect(brief).toContain("## Ask First");
|
|
235
|
+
expect(brief).toContain("## Never Do");
|
|
236
|
+
});
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
### Gate enforcement assertions (contract tests)
|
|
240
|
+
|
|
241
|
+
```typescript
|
|
242
|
+
// SC-50: Evidence hierarchy enforced per AC type
|
|
243
|
+
test("SC-50: CODE AC with tier D only = FAIL", () => {
|
|
244
|
+
const state = makeFixture({
|
|
245
|
+
acs: [{
|
|
246
|
+
id: "AC-1", type: "CODE",
|
|
247
|
+
evidence: { type: "command-output", content: "grep result" },
|
|
248
|
+
evidenceMethod: { type: "grep" },
|
|
249
|
+
verdict: "PASS"
|
|
250
|
+
}],
|
|
251
|
+
});
|
|
252
|
+
const result = runGate("ship", state);
|
|
253
|
+
expect(result.fail).toBeGreaterThan(0);
|
|
254
|
+
expect(result.output).toContain("evidence-tier");
|
|
255
|
+
});
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
### Rule count assertions
|
|
259
|
+
|
|
260
|
+
```typescript
|
|
261
|
+
// SC-138: Layer 1 template caps at 10 rules
|
|
262
|
+
test("SC-138: coding principles has ≤10 essential rules", () => {
|
|
263
|
+
const content = readFileSync("prompts/coding-principles.md", "utf-8");
|
|
264
|
+
const rules = content.match(/^[-*] \*\*.+\*\*/gm) || [];
|
|
265
|
+
expect(rules.length).toBeLessThanOrEqual(10);
|
|
266
|
+
});
|
|
267
|
+
|
|
268
|
+
// SC-139: AGENTS.md rules under 200 words
|
|
269
|
+
test("SC-139: AGENTS.md rules content under 200 words", () => {
|
|
270
|
+
const content = readFileSync("AGENTS.md", "utf-8");
|
|
271
|
+
// Strip tables, headers, code blocks — count only prose rules
|
|
272
|
+
const proseOnly = content
|
|
273
|
+
.replace(/\|.*\|/g, "")
|
|
274
|
+
.replace(/^#+ .+/gm, "")
|
|
275
|
+
.replace(/```[\s\S]*?```/g, "");
|
|
276
|
+
const wordCount = proseOnly.trim().split(/\s+/).length;
|
|
277
|
+
expect(wordCount).toBeLessThanOrEqual(200);
|
|
278
|
+
});
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
### Activation mode assertions
|
|
282
|
+
|
|
283
|
+
```typescript
|
|
284
|
+
// SC-143: Every Layer 1 template declares activation mode
|
|
285
|
+
test("SC-143: all templates have activation mode", () => {
|
|
286
|
+
const templates = readdirSync("prompts/").filter(f => f.endsWith(".md"));
|
|
287
|
+
for (const tmpl of templates) {
|
|
288
|
+
const content = readFileSync(`prompts/${tmpl}`, "utf-8");
|
|
289
|
+
expect(content).toMatch(/activation:\s*(always-on|file-pattern|agent-decided|manual)/);
|
|
290
|
+
}
|
|
291
|
+
});
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
## Phased Implementation
|
|
295
|
+
|
|
296
|
+
### Phase 0: Scaffold — all output files with correct content
|
|
297
|
+
**SCs:** 1-6, 8, 11, 15-21, 24-25, 37, 39-40, 73-78, 80, 82-90, 93-95, 97-101, 107, 114-126, 128, 138-148, 159-160
|
|
298
|
+
**Test:** Golden fixture — scaffold runs, EVERY output file checked for correct content against spec
|
|
299
|
+
**Gate:** `bun test test/phase-0.test.ts` — 0 failures
|
|
300
|
+
**Deliverable:** Scaffold creates all directories and files. Every file has the content the spec says it should have. No "file exists but content is wrong" — content is verified for all 13+ output files.
|
|
301
|
+
|
|
302
|
+
Output files verified:
|
|
303
|
+
- .gitignore (SC-40): all security entries
|
|
304
|
+
- package.json (SC-73): required fields added
|
|
305
|
+
- tsconfig.json (SC-74): recommended fields checked
|
|
306
|
+
- CLAUDE.md (SC-19): @AGENTS.md bridge
|
|
307
|
+
- AGENTS.md (SC-17, 18, 25, 94, 95, 128, 139): <150 lines, required sections, empty omitted, rules under 200 words
|
|
308
|
+
- CODE-MAP.md (SC-1, 39, 125, 126): git SHA frontmatter, correct sections, scan-paths
|
|
309
|
+
- .claude/rungate.json (SC-2, 4-6, 15-16, 21, 101, 107, 124): all fields detected or null, ports from Makefile, envVars from .env.example
|
|
310
|
+
- .claude/agents/marcus.md (SC-37, 115, 117, 138, 140-142, 145, 146): Core Principles, methodology refs, three-tier authority, section ordering, examples
|
|
311
|
+
- .claude/agents/quinn.md (SC-37, 55, 146): Core Principles, journey format ref, three-tier authority
|
|
312
|
+
- .claude/agents/rook.md (SC-37, 146): Core Principles, three-tier authority
|
|
313
|
+
- .claude/agents/serena.md (SC-37, 146): Core Principles, three-tier authority
|
|
314
|
+
- .claude/agents/aditi.md (SC-37, 146): Core Principles, three-tier authority
|
|
315
|
+
- .github/copilot-instructions.md (SC-123): points to AGENTS.md, create-if-missing
|
|
316
|
+
- .github/workflows/ci.yml (SC-82, 86, 87): managed header, bun test + typecheck
|
|
317
|
+
- .github/workflows/gates.yml (SC-83, 85): conformity + secret scan
|
|
318
|
+
- test/scaffold-conformity.test.ts (SC-24): imports all 9 suites
|
|
319
|
+
- specs/SPEC-TEMPLATE.md (SC-77): all frontmatter fields
|
|
320
|
+
|
|
321
|
+
### Phase 1: Knowledge extraction + doc hygiene + findings pipeline
|
|
322
|
+
**SCs:** 12-14, 22-23, 30, 75-76, 106, 137, 149-160
|
|
323
|
+
**Test:** Fixture with known bad docs → detection runs → findings written to unified JSON → gate runner reads + surfaces → agents get fix commands
|
|
324
|
+
**Gate:** `bun test test/phase-1.test.ts` — 0 failures
|
|
325
|
+
**Deliverable:**
|
|
326
|
+
- extract-constraints finds signal phrases from docs, pipes candidates to unified findings JSON
|
|
327
|
+
- Doc hygiene detects misplaced files (HYGIENE-7/8/9), bad frontmatter (HYGIENE-10), staleness
|
|
328
|
+
- All findings written to `.rungate/conformity-findings.json` with fix commands
|
|
329
|
+
- Gate runner reads findings JSON, prints structured output, stores in workflow-state.json
|
|
330
|
+
- AGENTS.md Commands table references findings JSON
|
|
331
|
+
- AGENTS.md Quick Reference #6 tells agents where to find fix commands
|
|
332
|
+
- Nothing is lost to stdout — all detection results persist in one machine-readable file
|
|
333
|
+
|
|
334
|
+
### Phase 2: Gate enforcement
|
|
335
|
+
**SCs:** 28-34, 48-72, 103-104, 109, 127, 129-135
|
|
336
|
+
**Depends on:** Phase 0
|
|
337
|
+
**Test:** Contract tests — workflow-state fixtures with known pass/fail conditions
|
|
338
|
+
**Gate:** `bun test test/phase-2.test.ts` — 0 failures
|
|
339
|
+
**Deliverable:** All gate checks enforce spec mechanically. Evidence hierarchy, RCA, blast radius, read-before-write, writer/verifier separation, test deletion prevention.
|
|
340
|
+
|
|
341
|
+
### Phase 3: Parallel work + CI
|
|
342
|
+
**SCs:** 45-47, 69-72, 110, 136
|
|
343
|
+
**Depends on:** Phase 2
|
|
344
|
+
**Test:** Worktree isolation, port allocation
|
|
345
|
+
**Gate:** `bun test test/phase-3.test.ts` — 0 failures
|
|
346
|
+
**Deliverable:** Parallel work isolation, container locking
|
|
347
|
+
|
|
348
|
+
### Phase 4: Dogfood — ship the harness using the harness
|
|
349
|
+
**SCs:** (meta-validation — no new SCs)
|
|
350
|
+
**Depends on:** Phase 0 (brief assembly works)
|
|
351
|
+
**Test:** Use the harness to create issues, ship implementations, run gates. If the harness ships itself correctly, briefs are proven correct.
|
|
352
|
+
**Gate:** Harness gates pass on harness code
|
|
353
|
+
**Deliverable:** Harness is self-hosting
|
|
354
|
+
|
|
355
|
+
## Post-Phase Spec Compliance Audit
|
|
356
|
+
|
|
357
|
+
After each phase completes, a compliance audit runs that produces a human-readable report. This is NOT a pass/fail test — it's a structured comparison of every spec claim against actual output. The report shows gaps that tests might miss and content that "passes" structurally but is incomplete semantically.
|
|
358
|
+
|
|
359
|
+
### Audit script
|
|
360
|
+
|
|
361
|
+
`bunx rungate audit-compliance <project-root> --phase N`
|
|
362
|
+
|
|
363
|
+
The script reads the parent spec (BOOTSTRAP-DATA-FLOW-SPEC.md) and the test plan, then produces a report:
|
|
364
|
+
|
|
365
|
+
```
|
|
366
|
+
╔══════════════════════════════════════════════════════════╗
|
|
367
|
+
║ SPEC COMPLIANCE AUDIT — Phase 0 ║
|
|
368
|
+
╚══════════════════════════════════════════════════════════╝
|
|
369
|
+
|
|
370
|
+
SPEC: BOOTSTRAP-DATA-FLOW-SPEC.md (2,400+ lines, 149 SCs)
|
|
371
|
+
PROJECT: /path/to/project
|
|
372
|
+
PHASE: 0 (Pre-flight + static files)
|
|
373
|
+
DATE: 2026-09-18
|
|
374
|
+
|
|
375
|
+
── FILES ──────────────────────────────────────────────────
|
|
376
|
+
|
|
377
|
+
File Spec says Actual Status
|
|
378
|
+
─────────────────────────────────────────────────────────────────────
|
|
379
|
+
.gitignore Created Phase 0.11 exists, 24 ln ✓ PASS
|
|
380
|
+
package.json Phase 0.18 exists ✓ PASS
|
|
381
|
+
tsconfig.json Phase 0.19 exists ✓ PASS
|
|
382
|
+
specs/SPEC-TEMPLATE.md Phase 0.12 exists ✓ PASS
|
|
383
|
+
test/scaffold-conformity.test.ts Phase 0.13 MISSING ✗ FAIL
|
|
384
|
+
...
|
|
385
|
+
|
|
386
|
+
── CONTENT CHECKS ─────────────────────────────────────────
|
|
387
|
+
|
|
388
|
+
SC-40: .gitignore has security template
|
|
389
|
+
Expected entries: node_modules, dist, .env*, .rungate, *.pem, *.key,
|
|
390
|
+
credentials.json, service-account*
|
|
391
|
+
Found: node_modules ✓, dist ✓, .env* ✓, .rungate ✓, *.pem ✓,
|
|
392
|
+
*.key ✓, credentials.json ✓, service-account* ✓
|
|
393
|
+
Status: ✓ PASS (8/8 entries present)
|
|
394
|
+
|
|
395
|
+
SC-73: package.json required fields
|
|
396
|
+
name: "golden-project" ✓
|
|
397
|
+
type: "module" ✓
|
|
398
|
+
scripts.test: MISSING ✗ FAIL
|
|
399
|
+
devDependencies: rungate NOT found ✗ FAIL
|
|
400
|
+
Status: ✗ FAIL (2/4 fields present)
|
|
401
|
+
|
|
402
|
+
SC-74: tsconfig.json recommended fields
|
|
403
|
+
strict: true ✓
|
|
404
|
+
module: "ESNext" ✓
|
|
405
|
+
Status: ✓ PASS (advisory — WARN only)
|
|
406
|
+
|
|
407
|
+
── BRIEF CONTENT (Phase 2 — preview only) ─────────────────
|
|
408
|
+
|
|
409
|
+
File Exists Sections Principles Examples Order Authority
|
|
410
|
+
──────────────────────────────────────────────────────────────────────────────────
|
|
411
|
+
.claude/agents/marcus.md N/A — — — — —
|
|
412
|
+
.claude/agents/quinn.md N/A — — — — —
|
|
413
|
+
(Phase 2 not yet implemented — shown for completeness)
|
|
414
|
+
|
|
415
|
+
── SUMMARY ────────────────────────────────────────────────
|
|
416
|
+
|
|
417
|
+
Files: 12 expected, 10 created, 2 missing
|
|
418
|
+
Content: 8 SCs checked, 6 PASS, 2 FAIL
|
|
419
|
+
Phase 0: NOT COMPLETE — 2 failures must be resolved
|
|
420
|
+
|
|
421
|
+
Missing:
|
|
422
|
+
- test/scaffold-conformity.test.ts (SC-24)
|
|
423
|
+
- package.json scripts.test field (SC-73)
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
### What the audit checks per file
|
|
427
|
+
|
|
428
|
+
For each file the spec says should exist:
|
|
429
|
+
|
|
430
|
+
| Check | What | How |
|
|
431
|
+
|-------|------|-----|
|
|
432
|
+
| Existence | File was created | `existsSync()` |
|
|
433
|
+
| Line count | Within spec limits (e.g., AGENTS.md ≤150) | `wc -l` |
|
|
434
|
+
| Required content | Spec-mandated entries/sections present | Keyword/pattern search |
|
|
435
|
+
| Content ordering | Sections in correct order (SC-140, SC-145) | Heading index comparison |
|
|
436
|
+
| Word count | Within budget (SC-139: rules ≤200 words) | Word count on prose |
|
|
437
|
+
| Rule count | Within caps (SC-138: ≤10 per template) | Pattern count |
|
|
438
|
+
| Examples present | Code examples exist (SC-142) | ` ```typescript` count |
|
|
439
|
+
| Authority tiers | Always Do / Ask First / Never Do (SC-146) | Section heading search |
|
|
440
|
+
| Frontmatter | Required fields present (SC-75, SC-76) | YAML parse |
|
|
441
|
+
| Cross-references | Pointers resolve to real files | `existsSync()` on referenced paths |
|
|
442
|
+
|
|
443
|
+
### Audit runs automatically after each phase gate
|
|
444
|
+
|
|
445
|
+
The phase test files include the audit as the LAST test:
|
|
446
|
+
|
|
447
|
+
```typescript
|
|
448
|
+
// At end of phase-0.test.ts
|
|
449
|
+
test("post-phase audit: compliance report", () => {
|
|
450
|
+
const report = runComplianceAudit(OUTPUT, 0);
|
|
451
|
+
console.log(report.formatted); // Human-readable output
|
|
452
|
+
|
|
453
|
+
// Store for cross-phase comparison
|
|
454
|
+
writeFileSync(
|
|
455
|
+
join(OUTPUT, ".rungate/audit-phase-0.json"),
|
|
456
|
+
JSON.stringify(report.structured, null, 2)
|
|
457
|
+
);
|
|
458
|
+
|
|
459
|
+
// Phase is complete only if all checks pass
|
|
460
|
+
expect(report.failures).toEqual([]);
|
|
461
|
+
});
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
### Cross-phase comparison
|
|
465
|
+
|
|
466
|
+
After Phase 1, the audit compares against Phase 0's report:
|
|
467
|
+
|
|
468
|
+
```
|
|
469
|
+
── PROGRESS SINCE PHASE 0 ─────────────────────────────────
|
|
470
|
+
|
|
471
|
+
New files: CODE-MAP.md, .claude/rungate.json
|
|
472
|
+
Updated files: package.json (devDependencies added)
|
|
473
|
+
SCs resolved: SC-73 (was FAIL, now PASS)
|
|
474
|
+
SCs remaining: 127 of 149
|
|
475
|
+
Coverage: 15% → 22%
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
This gives a running view of implementation progress against the full spec — not just "did this phase pass" but "how much of the total spec is now fulfilled."
|
|
479
|
+
|
|
480
|
+
## Test File Organization
|
|
481
|
+
|
|
482
|
+
```
|
|
483
|
+
test/
|
|
484
|
+
├── fixtures/
|
|
485
|
+
│ └── golden-project/ # Synthetic project with known structure
|
|
486
|
+
├── expected/ # Expected scaffold output (hand-written)
|
|
487
|
+
├── phase-0.test.ts # Pre-flight + static file tests
|
|
488
|
+
├── phase-1.test.ts # CODE-MAP + rungate.json tests
|
|
489
|
+
├── phase-2.test.ts # AGENTS.md + brief assembly tests
|
|
490
|
+
├── phase-3.test.ts # Knowledge extraction + doc hygiene tests
|
|
491
|
+
├── phase-4.test.ts # Gate enforcement contract tests
|
|
492
|
+
├── phase-5.test.ts # Parallel work + CI tests
|
|
493
|
+
├── golden-fixture.test.ts # Full golden fixture diff test
|
|
494
|
+
├── contract.test.ts # (existing) LIGHT/STANDARD tier contract tests
|
|
495
|
+
└── contract-negative.test.ts # (existing) Negative case contract tests
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
## SC Coverage Matrix
|
|
499
|
+
|
|
500
|
+
Every SC must appear in exactly one phase test file. The coverage matrix prevents orphaned SCs:
|
|
501
|
+
|
|
502
|
+
| Phase | Test file | SC range | Count |
|
|
503
|
+
|-------|-----------|----------|-------|
|
|
504
|
+
| 0 | phase-0.test.ts | 40, 73-78, 80, 93, 97-100 | 12 |
|
|
505
|
+
| 1 | phase-1.test.ts | 1, 2, 15-16, 21, 39, 101, 107, 124-126 | 10 |
|
|
506
|
+
| 2 | phase-2.test.ts | 3-6, 8, 11, 17-19, 25, 37, 61-62, 88-90, 94-95, 114-122, 128, 138-146 | 37 |
|
|
507
|
+
| 3 | phase-3.test.ts | 12-14, 22-23, 30, 75-76, 106, 137 | 9 |
|
|
508
|
+
| 4 | phase-4.test.ts | 28-34, 48-72, 103-104, 109, 127, 129-135 | 39 |
|
|
509
|
+
| 5 | phase-5.test.ts | 45-47, 69-72, 82-87, 110, 136 | 13 |
|
|
510
|
+
| Anti | (spread across phases) | SC-A1, SC-A2, SC-A3 | 3 |
|
|
511
|
+
| Existing | contract*.test.ts | (subset overlap — existing gate tests) | — |
|
|
512
|
+
| Cross-cut | golden-fixture.test.ts | All output-producing SCs | — |
|
|
513
|
+
| **Total** | | | **123 + 3 anti + 23 gate/overlap** |
|
|
514
|
+
|
|
515
|
+
**Unaccounted SC check:** At test time, a meta-test reads all SC lines from the parent spec, reads all test files, and asserts every SC ID appears in at least one `test("SC-N:` string. Missing SC = FAIL.
|
|
516
|
+
|
|
517
|
+
```typescript
|
|
518
|
+
test("meta: every SC has a test", () => {
|
|
519
|
+
const spec = readFileSync("specs/BOOTSTRAP-DATA-FLOW-SPEC.md", "utf-8");
|
|
520
|
+
const scIds = [...spec.matchAll(/SC-(\d+|A\d+)/g)].map(m => m[1]);
|
|
521
|
+
const unique = [...new Set(scIds)];
|
|
522
|
+
|
|
523
|
+
const testFiles = globSync("test/phase-*.test.ts");
|
|
524
|
+
const testContent = testFiles.map(f => readFileSync(f, "utf-8")).join("\n");
|
|
525
|
+
|
|
526
|
+
const missing = unique.filter(id => !testContent.includes(`SC-${id}`));
|
|
527
|
+
expect(missing).toEqual([]);
|
|
528
|
+
});
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
## Success Criteria
|
|
532
|
+
|
|
533
|
+
- [ ] TP-1: Golden test fixture exists at test/fixtures/golden-project/ with all required files
|
|
534
|
+
- [ ] TP-2: Expected output files exist at test/expected/ — hand-written, not generated
|
|
535
|
+
- [ ] TP-3: Every SC (146 + 3 anti) appears in at least one phase test file — meta-test verifies
|
|
536
|
+
- [ ] TP-4: Phase tests are independently runnable — `bun test test/phase-N.test.ts`
|
|
537
|
+
- [ ] TP-5: All tests RED before implementation — `bun test` shows 149 failures
|
|
538
|
+
- [ ] TP-6: Phase gate enforced — phase N+1 tests import phase N's pass count and skip if phase N hasn't passed
|
|
539
|
+
- [ ] TP-7: Golden fixture diff test runs on every `bun test` — catches drift in any output file
|
|
540
|
+
- [ ] TP-8: Content assertions test CONTENT not existence — no `existsSync()` without a corresponding content check
|
|
541
|
+
- [ ] TP-9: Meta-test validates SC coverage — missing SC in test files = FAIL
|
|
542
|
+
- [ ] TP-10: This test plan spec has spec-drift hash tracked — changes to BOOTSTRAP-DATA-FLOW-SPEC.md trigger re-evaluation
|
|
543
|
+
- [ ] TP-11: Post-phase compliance audit runs after each phase gate — produces human-readable report comparing every file and its contents against the spec
|
|
544
|
+
- [ ] TP-12: Compliance audit checks 10 dimensions per file: existence, line count, required content, ordering, word count, rule count, examples, authority tiers, frontmatter, cross-references
|
|
545
|
+
- [ ] TP-13: Cross-phase comparison shows progress delta — new files, updated files, SCs resolved since prior phase, total coverage %
|
|
546
|
+
- [ ] TP-14: `bunx rungate audit-compliance <root> --phase N` runnable standalone outside of test suite for manual verification
|
|
547
|
+
|
|
548
|
+
## SC-to-Phase Routing
|
|
549
|
+
|
|
550
|
+
Routing is automatic — the meta-test reads `### Phase N` subheaders from the Success Criteria section in `BOOTSTRAP-DATA-FLOW-SPEC.md`.
|
|
551
|
+
|
|
552
|
+
To add a new SC: paste it under the correct `### Phase` header. No routing table to maintain. The meta-test parses the section headers and routes each SC to the corresponding `test/phase-N.test.ts` file.
|
|
553
|
+
|
|
554
|
+
| Phase Header | Test File | What It Covers |
|
|
555
|
+
|---|---|---|
|
|
556
|
+
| `### Phase 0 — Scaffold Output` | `test/phase-0.test.ts` | Files, directories, content created by scaffold |
|
|
557
|
+
| `### Phase 1 — Knowledge Extraction + Doc Hygiene` | `test/phase-1.test.ts` | Constraint extraction, doc hygiene, findings pipeline |
|
|
558
|
+
| `### Phase 2 — Gate Enforcement + Ship Behavior` | `test/phase-2.test.ts` | Gates, evidence, ACs, briefs, ship-cycle behavior |
|
|
559
|
+
| `### Phase 3 — Parallel Work` | `test/phase-3.test.ts` | Worktrees, port allocation, concurrent agents |
|
|
560
|
+
| `### Anti-Criteria` | `test/anti.test.ts` | Must-NOT-happen conditions |
|
|
561
|
+
|
|
562
|
+
If an SC is not under any `### Phase` header, the meta-test reports it as "UNROUTED" — move it under the correct header to fix.
|
|
563
|
+
|
|
564
|
+
## Cautions
|
|
565
|
+
|
|
566
|
+
- Golden fixture expected output must be HAND-WRITTEN to match the spec. Auto-generating expected output from the implementation is a tautological test — it tests "does the code produce what the code produces" not "does the code produce what the spec says."
|
|
567
|
+
- Phase test files should not share test infrastructure beyond fixture creation utilities. Each phase must be independently debuggable.
|
|
568
|
+
- The SC coverage matrix will drift as SCs are added to the parent spec. The meta-test catches this mechanically — no behavioral reliance.
|
|
569
|
+
- Content assertions are FRAGILE if they match exact strings. Use patterns where possible (`toContain`, `toMatch`) rather than exact equality. Reserve exact equality for the golden fixture.
|
|
570
|
+
- Phase gate enforcement adds test ordering dependency. If Phase 1 is broken, Phase 2 tests should SKIP (not FAIL) to avoid masking the root cause with downstream failures.
|
|
571
|
+
- Dogfood phase (Phase 4) requires the harness to be functional enough to create issues and run gates. If Phase 2 gates are broken, dogfooding produces false confidence. Phase 4 only starts after Phase 2 passes.
|