rungate 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/aditi.md +47 -0
- package/.claude/agents/marcus.md +50 -0
- package/.claude/agents/quinn.md +58 -0
- package/.claude/agents/rook.md +47 -0
- package/.claude/agents/serena.md +46 -0
- package/.claude/rungate.json +11 -0
- package/.fallowrc.json +28 -0
- package/.github/copilot-instructions.md +20 -0
- package/.rungate/conformity-findings.json +648 -0
- package/AGENTS.md +106 -0
- package/CODE-MAP.md +50 -0
- package/HARNESS.md +88 -0
- package/bun.lock +26 -0
- package/bunfig.toml +2 -0
- package/config/brief-policies.json +28 -0
- package/config/ceremony-profiles.json +212 -0
- package/config/consumer-cascade-map.json +24 -0
- package/config/doc-cascade-map.json +24 -0
- package/config/eval.json +84 -0
- package/config/profiles/account-plan.yaml +36 -0
- package/config/profiles/campaign.yaml +94 -0
- package/config/prove-evidence.schema.json +65 -0
- package/config/spec-policies.json +36 -0
- package/config/workflow-schema.json +281 -0
- package/gates/SCHEMA-GUIDE.md +121 -0
- package/gates/ac-quality.test.ts +106 -0
- package/gates/adversarial.test.ts +274 -0
- package/gates/brief-assembler.test.ts +429 -0
- package/gates/brief-assembler.ts +334 -0
- package/gates/bunfig.toml +2 -0
- package/gates/chain.test.ts +91 -0
- package/gates/e2e-smoke.test.ts +234 -0
- package/gates/error-classifier.test.ts +249 -0
- package/gates/error-classifier.ts +89 -0
- package/gates/orchestrator.test.ts +950 -0
- package/gates/orchestrator.ts +569 -0
- package/gates/preload.ts +77 -0
- package/gates/prompts/ac-adversary.md +20 -0
- package/gates/prompts/evidence-validator.md +22 -0
- package/gates/prompts/prove-reproducer.md +53 -0
- package/gates/prove.test.ts +164 -0
- package/gates/run-gate.ts +936 -0
- package/gates/schema-parity.test.ts +197 -0
- package/gates/schema.ts +362 -0
- package/gates/self-heal.test.ts +511 -0
- package/gates/self-heal.ts +121 -0
- package/gates/ship-orchestrator.test.ts +379 -0
- package/gates/ship-orchestrator.ts +363 -0
- package/gates/test-fixtures/broken-no-environments.json +60 -0
- package/gates/test-fixtures/valid-standard.json +54 -0
- package/gates/test-utils.ts +43 -0
- package/gates/witness.ts +220 -0
- package/gates/workflow.test.ts +1053 -0
- package/hooks/AgentBriefGuard.hook.ts +586 -0
- package/hooks/AgentVerdictCapture.hook.ts +134 -0
- package/hooks/AutoVerifyGate.hook.ts +76 -0
- package/hooks/GateEnforcement.hook.ts +273 -0
- package/hooks/IssueCloseGuard.hook.ts +116 -0
- package/hooks/MergeGuard.hook.ts +55 -0
- package/hooks/StaleTTLCleanup.hook.ts +133 -0
- package/hooks/VerifyPhaseLock.hook.ts +81 -0
- package/hooks/WorkflowStateGuard.hook.ts +37 -0
- package/hooks/lib/agentDetection.ts +70 -0
- package/hooks/lib/findWorkflow.ts +101 -0
- package/hooks/lib/parseStdin.ts +37 -0
- package/hooks/lib/paths.ts +111 -0
- package/hooks/lib/utils.ts +81 -0
- package/lib/conformity.ts +959 -0
- package/lib/paths.ts +23 -0
- package/lib/rungate-schema.ts +60 -0
- package/lib/signal-phrases.ts +7 -0
- package/package.json +20 -0
- package/prompts/ac-adversary.md +20 -0
- package/prompts/ac-format.md +34 -0
- package/prompts/aditi.md +29 -0
- package/prompts/blast-radius.md +29 -0
- package/prompts/coding-principles.md +34 -0
- package/prompts/container-rebuild.md +29 -0
- package/prompts/container-verify.md +28 -0
- package/prompts/discovery.md +27 -0
- package/prompts/environment.md +28 -0
- package/prompts/escalation-decision-tree.md +38 -0
- package/prompts/evidence-hierarchy.md +31 -0
- package/prompts/evidence-validator.md +22 -0
- package/prompts/marcus.md +29 -0
- package/prompts/prevention.md +26 -0
- package/prompts/prove-reproducer.md +43 -0
- package/prompts/quinn-decision-tree.md +31 -0
- package/prompts/quinn-ui-brief.md +106 -0
- package/prompts/quinn.md +29 -0
- package/prompts/rca.md +28 -0
- package/prompts/read-before-write.md +30 -0
- package/prompts/regression.md +26 -0
- package/prompts/rook.md +29 -0
- package/prompts/serena.md +28 -0
- package/prompts/testing-strategy.md +34 -0
- package/reference/migration-manifest.json +554 -0
- package/reference/prompts/agent-aditi.md +38 -0
- package/reference/prompts/agent-marcus.md +38 -0
- package/reference/prompts/agent-quinn.md +38 -0
- package/reference/prompts/agent-rook.md +37 -0
- package/reference/prompts/agent-serena.md +37 -0
- package/reference/prompts/council-skill.md +170 -0
- package/reference/prompts/goal-skill.md +235 -0
- package/reference/prompts/prove-skill.md +283 -0
- package/reference/prompts/quinn-standard.md +246 -0
- package/reference/prompts/ship-convergence.md +65 -0
- package/reference/prompts/ship-escalation.md +43 -0
- package/reference/prompts/ship-iteration.md +382 -0
- package/reference/prompts/ship-pending-patches.md +246 -0
- package/reference/prompts/ship-reference.md +250 -0
- package/reference/prompts/ship-skill.md +347 -0
- package/reference/prompts/ship-templates.md +215 -0
- package/reference/scripts/batch-status.sh +68 -0
- package/reference/scripts/chain-compat-check.sh +67 -0
- package/reference/scripts/context-audit.sh +314 -0
- package/reference/scripts/dead-export-check.sh +61 -0
- package/reference/scripts/generate-qc-tables.sh +92 -0
- package/reference/scripts/harness-init.sh +111 -0
- package/reference/scripts/pai-054-baseline-check.sh +40 -0
- package/reference/scripts/prove-smoke-test.sh +112 -0
- package/reference/scripts/setup-keychain-salt.sh +12 -0
- package/reference/scripts/skill-optimizer-friday.sh +88 -0
- package/reference/scripts/skill-qc-test.sh +351 -0
- package/reference/scripts/skill-synthesize-tonight.sh +66 -0
- package/reference/specs/HARNESS-EXTRACTION-SPEC.md +263 -0
- package/reference/specs/harness-v3-bash-deletion-plan.md +65 -0
- package/reference/specs/harness-v3-migration-plan.md +188 -0
- package/scripts/decision-reconcile.sh +74 -0
- package/scripts/decision-reconcile.ts +152 -0
- package/scripts/extract-constraints.ts +260 -0
- package/scripts/generate-code-map.ts +339 -0
- package/scripts/git-hooks/pre-commit +76 -0
- package/scripts/git-hooks/pre-push +271 -0
- package/scripts/harness-skill-check.sh +52 -0
- package/scripts/lib/gap-emit.sh +65 -0
- package/scripts/lib/jq-update.sh +30 -0
- package/scripts/lib/resolve-slug.sh +30 -0
- package/scripts/lib/verify-spec-compliance.sh +102 -0
- package/scripts/prove-backfill.sh +87 -0
- package/scripts/scaffold-project.ts +1488 -0
- package/scripts/scaffold-rungate-config.ts +91 -0
- package/scripts/self-containment-check.sh +109 -0
- package/scripts/skill-qc-validator.sh +402 -0
- package/scripts/slug-resolver.sh +14 -0
- package/scripts/sync-spec-tests.ts +236 -0
- package/specs/BOOTSTRAP-DATA-FLOW-SPEC.md +2493 -0
- package/specs/BOOTSTRAP-TEST-PLAN.md +571 -0
- package/specs/HARNESS-GATES.md +256 -0
- package/specs/HARNESS-SKILL-CHAIN.md +286 -0
- package/specs/HARNESS-SKILL-CONTRACT.md +93 -0
- package/specs/HARNESS-STANDARD.md +488 -0
- package/specs/SPEC-TEMPLATE.md +53 -0
- package/specs/bootstrap-data-flow.png +0 -0
- package/specs/harness-automation-matrix.md +314 -0
- package/specs/harness-automation-matrix.rules.json +64 -0
- package/test/anti.test.ts +80 -0
- package/test/contract-negative.test.ts +86 -0
- package/test/contract.test.ts +158 -0
- package/test/external-deps.test.ts +48 -0
- package/test/fixtures/golden-project/.env.example +3 -0
- package/test/fixtures/golden-project/Makefile +7 -0
- package/test/fixtures/golden-project/docs/adr/ADR-001-framework.md +23 -0
- package/test/fixtures/golden-project/docs/setup-guide.md +10 -0
- package/test/fixtures/golden-project/package.json +10 -0
- package/test/fixtures/golden-project/specs/api-spec.md +20 -0
- package/test/fixtures/golden-project/src/auth.ts +8 -0
- package/test/fixtures/golden-project/src/db.ts +11 -0
- package/test/fixtures/golden-project/src/index.ts +19 -0
- package/test/fixtures/golden-project/src/utils/format.ts +3 -0
- package/test/fixtures/golden-project/test/auth.test.ts +15 -0
- package/test/fixtures/golden-project/tsconfig.json +12 -0
- package/test/meta-sc-coverage.test.ts +159 -0
- package/test/phase-0.test.ts +846 -0
- package/test/phase-1.test.ts +617 -0
- package/test/phase-2.test.ts +620 -0
- package/test/phase-3.test.ts +99 -0
- package/test/scaffold-conformity.test.ts +8 -0
- package/test/schema-canary.test.ts +51 -0
- package/test/spec-compliance-auto.test.ts +64 -0
- package/test/spec-compliance.test.ts +295 -0
- package/test/spec-discovery.test.ts +68 -0
- package/test/spec-drift.test.ts +5 -0
- package/test/structure.test.ts +165 -0
- package/test/unit/agent-brief-guard-ship.test.ts +293 -0
- package/test/unit/extract-constraints.test.ts +145 -0
- package/test/unit/issue-close-guard.test.ts +204 -0
- package/test/unit/scaffold-project.test.ts +417 -0
- package/test/unit/skill-cooccurrence.test.ts +177 -0
- package/test/unit/skill-sequence-logger.test.ts +244 -0
- package/test/unit/workflow-state-guard.test.ts +95 -0
- package/workflows/batch-ship.js +295 -0
- package/workflows/council.js +481 -0
- package/workflows/prove.js +560 -0
- package/workflows/ship.js +789 -0
- package/workflows/verify.js +236 -0
|
@@ -0,0 +1,2493 @@
|
|
|
1
|
+
---
|
|
2
|
+
doc-type: spec
|
|
3
|
+
status: draft
|
|
4
|
+
owner: jason
|
|
5
|
+
created: 2026-09-17
|
|
6
|
+
updated: 2026-09-18
|
|
7
|
+
governs: Bootstrap data flow — scan order, data sources, consumer requirements, re-run behavior
|
|
8
|
+
testable: true
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Bootstrap Data Flow Spec
|
|
12
|
+
|
|
13
|
+
## Problem Statement
|
|
14
|
+
|
|
15
|
+
Bootstrap generates files in the wrong order — AGENTS.md before CODE-MAP.md and rungate.json exist. The ship workflow hardcodes 50 DDB-specific values instead of reading from project config. No spec defines what data each consumer needs or where it comes from. Result: scaffolding that looks right but doesn't flow right.
|
|
16
|
+
|
|
17
|
+
## Core Principles
|
|
18
|
+
|
|
19
|
+
1. **Data flows DOWN. Never sideways, never up.** Code is scanned once. Each downstream file reads from the file above it, never from raw code.
|
|
20
|
+
2. **Null means skip, not guess.** Undetected config fields are null. Null causes the dependent workflow step to SKIP — no agent spawned, no command run. Never substitute hardcoded defaults for missing data.
|
|
21
|
+
3. **AGENTS.md under 150 lines.** Research (ETH Zurich, GitHub 2,500-repo study) shows files over 150 lines increase inference costs 20-23% with no behavior improvement. Omit empty sections. Scaffold WARNS if over 150.
|
|
22
|
+
4. **Only include non-inferrable details.** Agent context files should contain what agents can't discover from reading the code: commands, constraints, boundaries, environment config.
|
|
23
|
+
5. **One command, idempotent.** `bun scripts/scaffold-project.ts /path` handles both new and existing projects. Re-running updates what changed, preserves what was manually added.
|
|
24
|
+
6. **Everything is regenerable.** All harness-managed files can be rebuilt from scratch by re-running scaffold. The only user-owned content that survives regeneration is Hard Constraints in AGENTS.md (preserved between markers). Agent briefs, CODE-MAP, AGENTS.md sections, conformity tests, templates — all ephemeral. If a conformity check fails on missing frontmatter or broken structure, the fix is always: re-run scaffold.
|
|
25
|
+
7. **Required content, not just required files.** Phase 0 doesn't just create files — it ensures required fields exist inside them (package.json fields, spec frontmatter, agent brief frontmatter). Checking file existence without validating content is theater.
|
|
26
|
+
8. **Test suite updates flow downstream automatically.** The conformity test file in the project imports suites from the harness package. When the harness adds new checks (new suites in `lib/conformity.ts`), projects get them on next `bun install` (package update). If the import line itself changes (new suite added to the generated test), re-scaffold regenerates the test file. The harness spec drives the tests, not the project.
|
|
27
|
+
|
|
28
|
+
## File Ownership Model
|
|
29
|
+
|
|
30
|
+
Every file the harness touches has exactly one ownership type. This determines what happens on re-scaffold.
|
|
31
|
+
|
|
32
|
+
| File | Ownership | Re-scaffold behavior |
|
|
33
|
+
|------|-----------|---------------------|
|
|
34
|
+
| `.claude/agents/marcus.md` | Harness-owned | Always regenerated from template |
|
|
35
|
+
| `.claude/agents/quinn.md` | Harness-owned | Always regenerated from template |
|
|
36
|
+
| `.claude/agents/rook.md` | Harness-owned | Always regenerated from template |
|
|
37
|
+
| `.claude/agents/serena.md` | Harness-owned | Always regenerated from template |
|
|
38
|
+
| `.claude/agents/aditi.md` | Harness-owned | Always regenerated from template |
|
|
39
|
+
| `CODE-MAP.md` | Harness-owned | Regenerated if src/ changed since last scan (git SHA comparison) |
|
|
40
|
+
| `test/scaffold-conformity.test.ts` | Harness-owned | Always regenerated to match harness version |
|
|
41
|
+
| `specs/SPEC-TEMPLATE.md` | Harness-owned | Updated to latest harness template |
|
|
42
|
+
| `.gitignore` | Co-owned (additive) | Append missing required entries. Never remove existing |
|
|
43
|
+
| `package.json` | Co-owned (additive) | Add missing required fields. Never overwrite existing |
|
|
44
|
+
| `CLAUDE.md` | Co-owned (additive) | Add `@AGENTS.md` if missing. Never overwrite existing |
|
|
45
|
+
| `.github/workflows/ci.yml` | Harness-owned | Always regenerated. Config (runner, bun version) from rungate.json |
|
|
46
|
+
| `.github/workflows/gates.yml` | Harness-owned | Always regenerated. Config from rungate.json |
|
|
47
|
+
| `tsconfig.json` | Co-owned (WARN) | WARN on missing recommended fields. Never modify |
|
|
48
|
+
| `AGENTS.md` | Hybrid (preserve markers) | Regenerate all sections EXCEPT Hard Constraints between markers |
|
|
49
|
+
| `.claude/rungate.json` | Hybrid (field-level merge) | Auto-detected fields regenerated from scan. Manual fields preserved. New fields added as null |
|
|
50
|
+
| `.github/copilot-instructions.md` | Tool-bridge | Create if missing. Skip if exists |
|
|
51
|
+
| `src/*.ts` | User-owned | Never touched |
|
|
52
|
+
| `docs/*.md` | User-owned | Never touched |
|
|
53
|
+
| `specs/*.md` (content) | User-owned | Never touched (frontmatter validated, not content) |
|
|
54
|
+
| `docs/adr/*.md` | User-owned | Never touched (frontmatter validated, not content) |
|
|
55
|
+
| `prompts/discovery-brief.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
56
|
+
| `prompts/marcus-implementation.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
57
|
+
| `prompts/quinn-ui-brief.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
58
|
+
| `prompts/rook-security-brief.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
59
|
+
| `prompts/serena-architecture-brief.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
60
|
+
| `prompts/aditi-design-brief.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
61
|
+
| `prompts/environment-check.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
62
|
+
| `prompts/container-rebuild.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
63
|
+
| `prompts/container-verify.md` | Harness-owned | Ships with harness package — not in project repo |
|
|
64
|
+
| `prompts/ac-adversary.md` | Harness-owned | Ships with harness package — gate prompt |
|
|
65
|
+
| `prompts/evidence-validator.md` | Harness-owned | Ships with harness package — gate prompt |
|
|
66
|
+
| `prompts/prove-reproducer.md` | Harness-owned | Ships with harness package — gate prompt |
|
|
67
|
+
| `prompts/rca-protocol.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
68
|
+
| `prompts/blast-radius-checklist.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
69
|
+
| `prompts/prevention-oriented-fix.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
70
|
+
| `prompts/regression-prevention-checklist.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
71
|
+
| `prompts/read-before-write-protocol.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
72
|
+
| `prompts/quinn-journey-decision-tree.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
73
|
+
| `prompts/evidence-hierarchy.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
74
|
+
| `prompts/ac-format-guide.md` | Harness-owned | Ships with harness package — Layer 1 methodology |
|
|
75
|
+
|
|
76
|
+
**Ownership types explained:**
|
|
77
|
+
- **Harness-owned** — fully managed, user should not edit, always regenerated
|
|
78
|
+
- **Co-owned (additive)** — harness ensures required entries, user adds their own, never removes existing
|
|
79
|
+
- **Co-owned (WARN)** — harness recommends fields, user decides, never modifies
|
|
80
|
+
- **Hybrid** — mostly regenerated, user content between preservation markers survives
|
|
81
|
+
- **Harness-audited** — harness reads and reports drift, user owns content
|
|
82
|
+
- **Tool-bridge** — one-line pointer to AGENTS.md, only created if missing
|
|
83
|
+
- **User-owned** — harness never touches content
|
|
84
|
+
|
|
85
|
+
### .gitignore template
|
|
86
|
+
|
|
87
|
+
Full template created on new projects. On existing projects, missing required entries are appended.
|
|
88
|
+
|
|
89
|
+
```gitignore
|
|
90
|
+
# === Dependencies ===
|
|
91
|
+
node_modules/
|
|
92
|
+
|
|
93
|
+
# === Build output ===
|
|
94
|
+
dist/
|
|
95
|
+
*.tsbuildinfo
|
|
96
|
+
|
|
97
|
+
# === Secrets — CRITICAL for AI repos ===
|
|
98
|
+
# Agents may create credential files without recognizing them as sensitive.
|
|
99
|
+
# This is a mechanical backstop: even if an agent writes a secret file,
|
|
100
|
+
# git won't stage it.
|
|
101
|
+
.env
|
|
102
|
+
.env.*
|
|
103
|
+
!.env.example
|
|
104
|
+
*.pem
|
|
105
|
+
*.key
|
|
106
|
+
credentials.json
|
|
107
|
+
service-account*.json
|
|
108
|
+
*-secret*
|
|
109
|
+
*_secret*
|
|
110
|
+
|
|
111
|
+
# === Harness work directories ===
|
|
112
|
+
.rungate/
|
|
113
|
+
|
|
114
|
+
# === OS artifacts ===
|
|
115
|
+
.DS_Store
|
|
116
|
+
Thumbs.db
|
|
117
|
+
|
|
118
|
+
# === Test/coverage artifacts ===
|
|
119
|
+
coverage/
|
|
120
|
+
*.lcov
|
|
121
|
+
|
|
122
|
+
# === Logs ===
|
|
123
|
+
*.log
|
|
124
|
+
npm-debug.log*
|
|
125
|
+
|
|
126
|
+
# === Editor/IDE (local settings only) ===
|
|
127
|
+
.vscode/settings.json
|
|
128
|
+
.idea/
|
|
129
|
+
|
|
130
|
+
# === Temporary files ===
|
|
131
|
+
tmp/
|
|
132
|
+
.tmp/
|
|
133
|
+
*.swp
|
|
134
|
+
*.swo
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
**Required entries** (scaffold verifies these exist — appends if missing):
|
|
138
|
+
- `node_modules/`, `dist/`, `.env`, `.env.*`, `.rungate/`, `*.pem`, `*.key`, `credentials.json`
|
|
139
|
+
|
|
140
|
+
**Recommended entries** (included in template, not enforced on re-scaffold):
|
|
141
|
+
- `coverage/`, `*.log`, `.DS_Store`, `*.tsbuildinfo`, `tmp/`
|
|
142
|
+
|
|
143
|
+
**Never auto-add:** Project-specific entries (database files, vendor dirs, generated assets). User adds these.
|
|
144
|
+
|
|
145
|
+
## Design Decisions
|
|
146
|
+
|
|
147
|
+
| Decision | What | Rationale |
|
|
148
|
+
|---|---|---|
|
|
149
|
+
| D-1 | CODE-MAP.md generates FIRST (from code scan) | All other files depend on it — routes, components, modules, consumers |
|
|
150
|
+
| D-2 | rungate.json generates SECOND (from CODE-MAP + Makefile + package.json) | Config file that every workflow reads — must have complete data |
|
|
151
|
+
| D-3 | AGENTS.md generates THIRD (from rungate + CODE-MAP ref + docs scan) | Routing table that agents read — needs environment, consumers, docs |
|
|
152
|
+
| D-4 | .claude/agents/*.md generate FOURTH (from rungate + AGENTS.md) | Agent context — needs ports, pages, consumers from upstream |
|
|
153
|
+
| D-5 | Ship workflow reads rungate.json at startup, fills prompt templates | Zero hardcoded values — all from config |
|
|
154
|
+
| D-6 | Prompt templates in prompts/*.md with ${VAR} placeholders | Methodology is harness-owned, project data is config-driven |
|
|
155
|
+
| D-7 | If a config field is null, the dependent step is SKIPPED — no agent spawned | Library projects (no container) skip container steps cleanly |
|
|
156
|
+
| D-8 | AGENTS.md is the cross-tool standard; CLAUDE.md bridges to Claude Code | 30+ tools read AGENTS.md natively. Claude Code reads CLAUDE.md. Bridge: `@AGENTS.md` import in CLAUDE.md |
|
|
157
|
+
| D-9 | Bun + TypeScript are safe defaults for testCmd and typeCheck | All projects use this runtime/language. Ports, commands, and structure are NOT safe defaults |
|
|
158
|
+
| D-10 | Empty sections omitted from AGENTS.md | No consumers = no Consumers section. Keeps file under 150 lines naturally |
|
|
159
|
+
| D-11 | New code projects get a src/index.ts stub | Gives agents a starting point. One-liner, no framework — framework choice comes from the first /goal |
|
|
160
|
+
| D-12 | ACs use Given/When/Then format with executable verification command | AC IS the oracle — not prose. Research: 50% error reduction, 16pp higher acceptance rates |
|
|
161
|
+
| D-13 | Evidence hierarchy S/A/B/C/D/F with minimum tier per AC type | Tier S (dual-arm) for bug fixes, A (execution) for CODE, B (behavioral) for UI. Self-attestation always FAIL |
|
|
162
|
+
| D-14 | RCA required before implementation on bug-fix issues | 63% of top SWE-bench scores retrieve fixes rather than derive them. RCA forces understanding before patching |
|
|
163
|
+
| D-15 | Quinn uses typed test journeys with a11y-first perception | CHI 2026: a11y tree degradation drops success 80%→42%. Typed steps prevent Explore→Explore→Explore loop |
|
|
164
|
+
| D-16 | Proof-of-fix includes negative control (revert, confirm bug returns) | 46% of validation events carry zero discriminating information (BSG-VA). Negative control is the discriminator |
|
|
165
|
+
| D-17 | Templates three-layered: harness-generic + project-config + issue-specific | Generic ships with rungate. Project from scan. Issue from brief-assembler. Any project gets best practices free |
|
|
166
|
+
| D-18 | Spec content hash enforced at gates — mismatch = FAIL not WARN | WARN is behavioral (agent ignores). FAIL is mechanical (can't proceed with stale tests) |
|
|
167
|
+
| D-19 | Blast radius analysis required before implementation | Change failure rate up ~30% since AI adoption (Cortex 2026). Agents must map dependents before changing code |
|
|
168
|
+
| D-20 | Read-before-write ratio ≥ 3:1 | SWE-Explore: context efficiency r=0.928 with solve rate. Epoch AI uses 10:1 read-to-write ratio |
|
|
169
|
+
| D-21 | Writer and verifier must be separate agents | Same-agent test authorship has 31.7% semantic drift miss rate. Marcus writes, Quinn verifies, gate enforces |
|
|
170
|
+
| D-22 | CI workflows ship with the harness — gates enforced in pipeline, not just locally | Local gates can be skipped. CI gates can't. The pipeline is the final mechanical enforcement layer |
|
|
171
|
+
| D-23 | .github/copilot-instructions.md is a tool-bridge, not CI config | Points Copilot (chat + coding agent) to AGENTS.md. CI/CD enforcement is in .github/workflows/ |
|
|
172
|
+
|
|
173
|
+
## Bootstrap Execution Order
|
|
174
|
+
|
|
175
|
+
Single entry point: `bun scripts/scaffold-project.ts /path/to/project`
|
|
176
|
+
|
|
177
|
+
```
|
|
178
|
+
Pre-flight — INIT (ensure git + bun are set up — runs BEFORE Phase 0)
|
|
179
|
+
|
|
180
|
+
Check: is this a git repo?
|
|
181
|
+
YES → skip
|
|
182
|
+
NO → git init
|
|
183
|
+
|
|
184
|
+
Check: does package.json exist?
|
|
185
|
+
YES → skip
|
|
186
|
+
NO → bun init (creates package.json + tsconfig.json)
|
|
187
|
+
|
|
188
|
+
Check: does .gitignore exist?
|
|
189
|
+
YES → verify required entries, append missing
|
|
190
|
+
NO → create from template (BEFORE any git add — secrets never touch history)
|
|
191
|
+
|
|
192
|
+
These three checks are idempotent. On existing projects, all three skip.
|
|
193
|
+
|
|
194
|
+
Phase 0 — STRUCTURE (directories + static files — no scan dependency)
|
|
195
|
+
|
|
196
|
+
Directories:
|
|
197
|
+
0.1 specs/ ← create dir (governs: spec files with testable SCs)
|
|
198
|
+
0.2 docs/ ← create dir (governs: project documentation)
|
|
199
|
+
0.3 docs/adr/ ← create dir (governs: architecture decision records)
|
|
200
|
+
0.4 reference/ ← create dir (governs: archived/historical docs)
|
|
201
|
+
0.5 test/ ← create dir (respects existing test/ vs tests/)
|
|
202
|
+
0.6 src/ ← create dir + index.ts stub (new code projects only)
|
|
203
|
+
0.7 .claude/ ← create dir
|
|
204
|
+
0.8 .claude/agents/ ← create dir (governs: agent brief files)
|
|
205
|
+
0.9 .github/ ← create dir (governs: GitHub config)
|
|
206
|
+
0.10 .github/workflows/ ← create dir (governs: CI/CD pipeline)
|
|
207
|
+
|
|
208
|
+
Static files (no scan data needed — harness-owned templates and config):
|
|
209
|
+
0.11 .gitignore ← verify required entries (created in pre-flight if missing)
|
|
210
|
+
0.12 specs/SPEC-TEMPLATE.md ← copy harness template if specs/ empty
|
|
211
|
+
0.13 test/scaffold-conformity.test.ts ← imports from harness package (always regenerated)
|
|
212
|
+
0.14 .github/copilot-instructions.md ← tool-bridge: points Copilot to AGENTS.md (skip if exists)
|
|
213
|
+
0.15 .github/workflows/ci.yml ← CI pipeline: bun test + typecheck on PR/push (see CI Workflows)
|
|
214
|
+
0.16 .github/workflows/gates.yml ← gate enforcement: conformity + secret scan on PR (see CI Workflows)
|
|
215
|
+
0.17 CLAUDE.md bridge ← add @AGENTS.md if not present, create minimal if missing
|
|
216
|
+
0.18 package.json ← ensure required fields (see Required File Content)
|
|
217
|
+
0.19 tsconfig.json ← ensure required fields (see Required File Content)
|
|
218
|
+
0.20 Frontmatter ← add to any bare spec files (see Required File Content)
|
|
219
|
+
|
|
220
|
+
Phase 1 — SCAN (produce raw data files — requires code to exist)
|
|
221
|
+
1.1 CODE-MAP.md ← fallow + route + component + module + page scans
|
|
222
|
+
1.2 rungate.json ← CODE-MAP consumers + Makefile + package.json (null for undetected)
|
|
223
|
+
|
|
224
|
+
Phase 1.5 — EXTRACT (separate command, not auto-run)
|
|
225
|
+
1.5 extract-constraints ← signal phrase scan of docs → candidates for human review
|
|
226
|
+
On re-scaffold: HYGIENE-6 warns about unreviewed candidates
|
|
227
|
+
|
|
228
|
+
Phase 2 — GENERATE (produce agent-facing files from Phase 1 outputs)
|
|
229
|
+
2.1 AGENTS.md ← rungate.json (env, consumers, pages) + CODE-MAP (ref) + docs scan
|
|
230
|
+
2.2 .claude/agents/ ← rungate.json (ports, pages) + AGENTS.md (identity, routing)
|
|
231
|
+
|
|
232
|
+
Post-scaffold — COMMIT (stage + commit harness files)
|
|
233
|
+
|
|
234
|
+
Check: are there unstaged harness changes?
|
|
235
|
+
YES (new project) → git add . && git commit -m "scaffold: initialize project with rungate"
|
|
236
|
+
YES (re-scaffold) → git add . && git commit -m "scaffold: update harness files"
|
|
237
|
+
NO → skip (nothing changed)
|
|
238
|
+
|
|
239
|
+
.gitignore is already in place (pre-flight), so git add . is safe — secrets won't be staged.
|
|
240
|
+
On re-scaffold, only changed files are staged (git tracks this naturally).
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
## Required File Content
|
|
244
|
+
|
|
245
|
+
Phase 0 doesn't just create files — it ensures required content exists inside them. The pattern is always: **read → check → add missing → never overwrite existing.** If the file doesn't exist, create it from template. If it exists, add only what's missing.
|
|
246
|
+
|
|
247
|
+
### package.json (Phase 0.15)
|
|
248
|
+
|
|
249
|
+
**If file doesn't exist:** Create from template with all required + recommended fields.
|
|
250
|
+
**If file exists:** Read it, add missing required fields, WARN on missing recommended fields.
|
|
251
|
+
|
|
252
|
+
```json
|
|
253
|
+
{
|
|
254
|
+
"name": "${PROJECT_NAME}",
|
|
255
|
+
"type": "module",
|
|
256
|
+
"scripts": {
|
|
257
|
+
"test": "bun test"
|
|
258
|
+
},
|
|
259
|
+
"devDependencies": {
|
|
260
|
+
"rungate": "latest"
|
|
261
|
+
}
|
|
262
|
+
}
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
| Field | Required | Default value | Behavior if exists |
|
|
266
|
+
|-------|----------|--------------|-------------------|
|
|
267
|
+
| `name` | Yes | Directory name | Never overwrite |
|
|
268
|
+
| `type` | Yes | `"module"` | Never overwrite — WARN if not "module" |
|
|
269
|
+
| `scripts.test` | Yes | `"bun test"` | Never overwrite — use existing value for testCmd |
|
|
270
|
+
| `devDependencies.rungate` | Yes | `"latest"` | Add if missing, don't change version if present |
|
|
271
|
+
| `scripts.dev` | Recommended | — | WARN if missing (dev server command) |
|
|
272
|
+
| `scripts.build` | Recommended | — | WARN if missing (build command) |
|
|
273
|
+
|
|
274
|
+
**Never touch:** `dependencies`, `version`, `description`, `scripts.*` (other than test/dev/build if missing), any other existing fields.
|
|
275
|
+
|
|
276
|
+
### tsconfig.json (Phase 0.16)
|
|
277
|
+
|
|
278
|
+
**If file doesn't exist:** Create from template.
|
|
279
|
+
**If file exists:** Read it, WARN on missing recommended fields. Never overwrite.
|
|
280
|
+
|
|
281
|
+
```json
|
|
282
|
+
{
|
|
283
|
+
"compilerOptions": {
|
|
284
|
+
"strict": true,
|
|
285
|
+
"module": "ESNext",
|
|
286
|
+
"moduleResolution": "bundler",
|
|
287
|
+
"target": "ESNext",
|
|
288
|
+
"esModuleInterop": true,
|
|
289
|
+
"skipLibCheck": true,
|
|
290
|
+
"outDir": "dist",
|
|
291
|
+
"declaration": true
|
|
292
|
+
},
|
|
293
|
+
"include": ["src"]
|
|
294
|
+
}
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
| Field | Required | Default value | Behavior if exists |
|
|
298
|
+
|-------|----------|--------------|-------------------|
|
|
299
|
+
| `compilerOptions.strict` | Recommended | `true` | WARN if false — type safety for agent-written code |
|
|
300
|
+
| `compilerOptions.module` | Recommended | `"ESNext"` | Never overwrite |
|
|
301
|
+
| `include` | Recommended | `["src"]` | Never overwrite |
|
|
302
|
+
|
|
303
|
+
**Never touch:** Any existing compiler options, paths, references, custom settings.
|
|
304
|
+
|
|
305
|
+
### Spec frontmatter (Phase 0.17)
|
|
306
|
+
|
|
307
|
+
**If spec file has no frontmatter (bare .md):** Prepend frontmatter block with required fields.
|
|
308
|
+
**If spec file has frontmatter:** Check for missing required fields, add them with placeholder values.
|
|
309
|
+
|
|
310
|
+
```yaml
|
|
311
|
+
---
|
|
312
|
+
doc-type: spec
|
|
313
|
+
status: draft
|
|
314
|
+
owner: (unknown)
|
|
315
|
+
created: ${TODAY}
|
|
316
|
+
testable: false
|
|
317
|
+
governs: (describe what this spec controls)
|
|
318
|
+
---
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
| Field | Required | Default value | Conformity check |
|
|
322
|
+
|-------|----------|--------------|-----------------|
|
|
323
|
+
| `doc-type` | Yes | `spec` | Spec Discovery: FAIL if missing |
|
|
324
|
+
| `status` | Yes | `draft` | Spec Discovery: FAIL if missing |
|
|
325
|
+
| `testable` | Yes | `false` | Spec Discovery: FAIL if missing |
|
|
326
|
+
| `governs` | Yes | placeholder | Doc Hygiene: FAIL if missing |
|
|
327
|
+
| `owner` | Recommended | `(unknown)` | WARN if `(unknown)` |
|
|
328
|
+
| `created` | Recommended | today's date | No check |
|
|
329
|
+
|
|
330
|
+
### ADR frontmatter
|
|
331
|
+
|
|
332
|
+
```yaml
|
|
333
|
+
---
|
|
334
|
+
doc-type: adr
|
|
335
|
+
status: proposed
|
|
336
|
+
created: ${TODAY}
|
|
337
|
+
supersedes: (none)
|
|
338
|
+
---
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
| Field | Required | Default |
|
|
342
|
+
|-------|----------|---------|
|
|
343
|
+
| `doc-type` | Yes | `adr` |
|
|
344
|
+
| `status` | Yes | `proposed` (→ accepted → superseded) |
|
|
345
|
+
| `created` | Yes | today's date |
|
|
346
|
+
| `supersedes` | Recommended | `(none)` |
|
|
347
|
+
|
|
348
|
+
### Agent brief frontmatter (.claude/agents/*.md)
|
|
349
|
+
|
|
350
|
+
```yaml
|
|
351
|
+
---
|
|
352
|
+
name: marcus
|
|
353
|
+
description: Implementation engineer — writes code, tests, evidence
|
|
354
|
+
---
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
| Field | Required | Conformity check |
|
|
358
|
+
|-------|----------|-----------------|
|
|
359
|
+
| `name` | Yes | Must match filename (agent-validation: FAIL) |
|
|
360
|
+
| `description` | Yes | Must be present (agent-validation: FAIL) |
|
|
361
|
+
|
|
362
|
+
### .gitignore required entries (Phase 0.10)
|
|
363
|
+
|
|
364
|
+
```
|
|
365
|
+
node_modules/
|
|
366
|
+
dist/
|
|
367
|
+
.env*
|
|
368
|
+
.rungate/
|
|
369
|
+
*.log
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
**If file exists:** Read it, append any missing entries. Never remove existing entries.
|
|
373
|
+
**If file doesn't exist:** Create with all entries above.
|
|
374
|
+
|
|
375
|
+
### Conformity validation
|
|
376
|
+
|
|
377
|
+
All required fields are checked by conformity tests at gate time:
|
|
378
|
+
- `runSpecDiscovery` → spec frontmatter (doc-type, status, testable)
|
|
379
|
+
- `runDocHygiene` → spec governs field
|
|
380
|
+
- `runAgentFileValidation` → agent brief frontmatter (name, description)
|
|
381
|
+
- New: `runPackageValidation` → package.json required fields
|
|
382
|
+
- New: `runTsconfigValidation` → tsconfig.json recommended fields (WARN only)
|
|
383
|
+
|
|
384
|
+
## New Project vs Existing Project
|
|
385
|
+
|
|
386
|
+
### New project (empty directory)
|
|
387
|
+
|
|
388
|
+
No preconditions — scaffold handles everything.
|
|
389
|
+
|
|
390
|
+
| Phase | What happens | Result |
|
|
391
|
+
|-------|-------------|--------|
|
|
392
|
+
| Pre-flight | `git init` + `bun init` + `.gitignore` created | Git repo with package.json, secrets excluded |
|
|
393
|
+
| 0 (dirs) | All 10 dirs created, src/index.ts stub written | Agents have a starting point |
|
|
394
|
+
| 0 (static) | Spec template, conformity test, CI workflows, copilot-instructions, CLAUDE.md bridge, devDep, git hooks | Project ready for agents before any scan |
|
|
395
|
+
| 1.1 | CODE-MAP.md generated — minimal (just src/index.ts) | Structure documented |
|
|
396
|
+
| 1.2 | rungate.json generated — testCmd="bun test", everything else null | Safe defaults only |
|
|
397
|
+
| 2.1 | AGENTS.md generated — ~40-60 lines, empty sections omitted | Concise, under 150 |
|
|
398
|
+
| 2.2 | Agent briefs generated — Bun/TS defaults, null ports | Agents know runtime, not app |
|
|
399
|
+
| Post-scaffold | `git add . && git commit -m "scaffold: initialize project with rungate"` | Clean initial commit |
|
|
400
|
+
|
|
401
|
+
After first /ship cycle: Marcus creates real code → re-run scaffold → CODE-MAP and rungate.json populate with detected ports, routes, consumers → AGENTS.md and agent briefs update with real data.
|
|
402
|
+
|
|
403
|
+
### Existing project (files already there)
|
|
404
|
+
|
|
405
|
+
| Phase | What happens | Result |
|
|
406
|
+
|-------|-------------|--------|
|
|
407
|
+
| Pre-flight | git init: SKIP (exists). bun init: SKIP (package.json exists). .gitignore: verify entries | No destructive changes |
|
|
408
|
+
| 0 (dirs) | Dirs: SKIP (exist) | No change |
|
|
409
|
+
| 0 (static) | .gitignore verified, spec template updated, conformity test updated, CI workflows regenerated, CLAUDE.md bridge checked | Static files current |
|
|
410
|
+
| 1.1 | CODE-MAP.md refreshed if src/ changed since last scan (git diff against stored SHA) | Current code state — no arbitrary time/count thresholds |
|
|
411
|
+
| 1.2 | rungate.json field-level merge — auto-detect fields updated, manual fields preserved | Config current without losing user settings |
|
|
412
|
+
| 1.5 | HYGIENE-6 warns about unreviewed constraint candidates | User runs extract-constraints manually |
|
|
413
|
+
| 2.1 | AGENTS.md refreshed — Hard Constraints PRESERVED, env/pages/consumers UPDATED from rungate.json | Always current, never loses manual data |
|
|
414
|
+
| 2.2 | Agent briefs REGENERATED — always overwritten from rungate.json | Always match current config |
|
|
415
|
+
| Post-scaffold | `git add . && git commit -m "scaffold: update harness files"` (only if changes) | Changes tracked in git |
|
|
416
|
+
|
|
417
|
+
### Re-scaffold audits
|
|
418
|
+
|
|
419
|
+
Audits run during re-scaffold fall into two categories: auto-fixable (the scan has the data to fix it) and report-only (needs human judgment).
|
|
420
|
+
|
|
421
|
+
#### Auto-fixed (the scaffold has the right data — just fix it)
|
|
422
|
+
|
|
423
|
+
| Audit | What it checks | Action |
|
|
424
|
+
|-------|---------------|--------|
|
|
425
|
+
| Broken refs | File paths in AGENTS.md that don't resolve | **AUTO-FIX**: remove broken ref from AGENTS.md. Log: "Removed broken ref: {path}" |
|
|
426
|
+
| Unlisted specs | Specs in specs/ not mentioned in AGENTS.md | **AUTO-FIX**: add to AGENTS.md specs table. AGENTS.md is regenerated anyway — this is a bug if it happens |
|
|
427
|
+
| Missing CODE-MAP ref | CODE-MAP.md exists but not in AGENTS.md routing | **AUTO-FIX**: add to doc routing. Same as above — regeneration should handle this |
|
|
428
|
+
| Pages drift | Routes in code not in rungate.json pages map | **AUTO-FIX**: update rungate.json pages (auto-detected field). Agents get correct page URLs |
|
|
429
|
+
|
|
430
|
+
These should never need manual intervention — the scaffold already has the correct data from the code scan. If AGENTS.md regeneration and rungate.json field-level merge are working correctly, these audits are catching regeneration bugs, not drift.
|
|
431
|
+
|
|
432
|
+
#### Report only (needs human judgment)
|
|
433
|
+
|
|
434
|
+
| Audit | What it checks | Action |
|
|
435
|
+
|-------|---------------|--------|
|
|
436
|
+
| Stale docs | Docs older than threshold (spec 90d, guide 180d, ADR exempt) | WARN with age — stale ≠ wrong, human decides to archive or update |
|
|
437
|
+
| AGENTS.md line count | AGENTS.md over 150 lines (ETH Zurich: >150 lines = 20-23% cost increase, no behavior improvement) | WARN with count — may be legitimate for complex projects with many consumers/pages |
|
|
438
|
+
| Stale constraints | Hard Constraints referencing files that no longer exist | WARN with list — constraint may still be valid as a principle even if the referenced file was deleted |
|
|
439
|
+
| Constraint candidates | Unreviewed candidates from extract-constraints | WARN with count — 70% false positive rate, human review mandatory |
|
|
440
|
+
|
|
441
|
+
## Data Sources (what gets scanned from code)
|
|
442
|
+
|
|
443
|
+
| Scan | Tool | Produces | Used By |
|
|
444
|
+
|------|------|----------|---------|
|
|
445
|
+
| Dead code, unused exports, circular deps | fallow | CODE-MAP.md § Code Health | Marcus (cascade impact), gates (health check) |
|
|
446
|
+
| API route definitions | regex scan of src/ (framework-agnostic: Hono, Express, Fastify, tRPC, etc.) | CODE-MAP.md § API Routes | Marcus (knows endpoints), AGENTS.md (route count) |
|
|
447
|
+
| UI components | file scan of detected components dir (from CODE-MAP, not hardcoded path) | CODE-MAP.md § Components | Quinn (what exists to test) |
|
|
448
|
+
| Page→component mappings | import scan of detected router file | CODE-MAP.md § Page→Component Map | Quinn (targeted UI testing) |
|
|
449
|
+
| Module import chains | import scan of src/ | CODE-MAP.md § Module Dependencies | Marcus (cascade impact analysis) |
|
|
450
|
+
| Consumer modules | CODE-MAP module scan (not hardcoded patterns) | rungate.json consumers field | AGENTS.md, Marcus (cascade impact) |
|
|
451
|
+
| Entry points | package.json main/exports + Makefile targets | CODE-MAP.md § Summary | Agent orientation |
|
|
452
|
+
| Package info | package.json | CODE-MAP.md § Summary + rungate.json | AGENTS.md (project name, deps), workflows (test command) |
|
|
453
|
+
| Makefile targets | Makefile (if exists) | rungate.json (dev/prod commands) | Workflows (rebuild, dev-all, test-up) |
|
|
454
|
+
| Git remote | git remote get-url origin | rungate.json repo field | PR creation, issue tracking |
|
|
455
|
+
| tsconfig.json | tsconfig.json exists + compilerOptions | rungate.json dev.typeCheck | Ship workflow (type checking) |
|
|
456
|
+
| Router file | Detected from common patterns (App.tsx, routes.tsx, app/routes/, pages/) — not hardcoded path | rungate.json pages map | Quinn (URL navigation), AGENTS.md (pages table) |
|
|
457
|
+
| Test files | file scan of test/ or tests/ | CODE-MAP.md § Test Coverage | Marcus (knows what's tested vs not), gates (baseline) |
|
|
458
|
+
| MCP servers | .claude/settings.json mcp section (if exists) | rungate.json mcp field | Agent briefs (available tools), AGENTS.md |
|
|
459
|
+
| Environment variables | .env.example (if exists) | rungate.json envVars field | Agents know what env vars the project expects |
|
|
460
|
+
| Specs | specs/*.md frontmatter | AGENTS.md specs table | Conformity tests, gates |
|
|
461
|
+
| Docs | docs/*.md | AGENTS.md doc routing table | Agent DISCOVERY, context loading |
|
|
462
|
+
|
|
463
|
+
## Consumer Requirements
|
|
464
|
+
|
|
465
|
+
### CODE-MAP.md
|
|
466
|
+
|
|
467
|
+
Produced by: `scripts/generate-code-map.ts`
|
|
468
|
+
Read by: AGENTS.md (references it), agents/marcus.md (Module Dependencies, Code Health), agents/quinn.md (Page→Component Map, API Routes)
|
|
469
|
+
|
|
470
|
+
| Section | What's In It | Who Reads It |
|
|
471
|
+
|---------|-------------|--------------|
|
|
472
|
+
| Summary | Counts: dirs, deps, routes, components, entry points, unused, circular | Quick orientation |
|
|
473
|
+
| Directory Structure | Dirs with file counts and types | Project navigation |
|
|
474
|
+
| Entry Points | package.json main/exports, Makefile entry targets | Agent orientation |
|
|
475
|
+
| API Routes | Method, path, file, line for every route (framework-agnostic detection) | Marcus (endpoint awareness) |
|
|
476
|
+
| UI Components | Component names (detected from component dirs, not hardcoded path) | Quinn (what exists to test) |
|
|
477
|
+
| Module Dependencies | Top modules by import count (highest-impact first, no arbitrary cutoff) | Marcus (cascade impact) |
|
|
478
|
+
| Page→Component Map | Which components render on each page (from detected router file) | Quinn (targeted UI testing) |
|
|
479
|
+
| Test Coverage | Test files in test/, what modules they cover (import analysis) | Marcus (knows what's tested vs gaps), gates (baseline) |
|
|
480
|
+
| Code Health | Circular deps, unused files, unused exports | Marcus (tech debt awareness) |
|
|
481
|
+
| Package Scripts | Available npm/bun scripts | Workflow commands |
|
|
482
|
+
|
|
483
|
+
**Required frontmatter:**
|
|
484
|
+
```yaml
|
|
485
|
+
---
|
|
486
|
+
scanned-at-sha: abc1234
|
|
487
|
+
scanned-at: 2026-09-18T21:00:00Z
|
|
488
|
+
scan-paths: [src/, lib/, test/]
|
|
489
|
+
---
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
**Staleness detection (git SHA, not time-based):**
|
|
493
|
+
|
|
494
|
+
CODE-MAP.md frontmatter includes `scanned-at-sha` — the git commit SHA when the scan ran. On re-scaffold:
|
|
495
|
+
|
|
496
|
+
```
|
|
497
|
+
1. Read CODE-MAP.md frontmatter → get scanned-at-sha
|
|
498
|
+
2. Run: git log --oneline {scanned-at-sha}..HEAD -- src/ lib/ gates/ hooks/
|
|
499
|
+
3. Any commits returned? → YES → regenerate CODE-MAP
|
|
500
|
+
4. No commits? → SKIP (source hasn't changed)
|
|
501
|
+
5. No scanned-at-sha? (first run or missing) → always regenerate
|
|
502
|
+
```
|
|
503
|
+
|
|
504
|
+
This replaces the previous 14-day/50-commit heuristic. One changed route file triggers a re-scan immediately. 100 doc-only commits don't trigger a pointless re-scan.
|
|
505
|
+
|
|
506
|
+
The scan paths (`src/ lib/ gates/ hooks/`) are the same directories CODE-MAP scans for routes, components, and modules. Changes outside these paths (docs, specs, reference) don't affect CODE-MAP.
|
|
507
|
+
|
|
508
|
+
### rungate.json
|
|
509
|
+
|
|
510
|
+
Produced by: `scripts/scaffold-project.ts` → `generateOrAuditProjectHarness()`
|
|
511
|
+
Read by: AGENTS.md, agents/*.md, ship.js, prove.js, verify.js, gates, CI workflows
|
|
512
|
+
|
|
513
|
+
**Ownership: Hybrid (field-level merge).** Auto-detected fields are regenerated from scan on every re-scaffold. Manual fields are preserved — never overwritten. New fields added as null.
|
|
514
|
+
|
|
515
|
+
**Re-scaffold behavior:**
|
|
516
|
+
```
|
|
517
|
+
1. Read existing rungate.json
|
|
518
|
+
2. Re-scan code (CODE-MAP, Makefile, package.json, git remote)
|
|
519
|
+
3. Auto-detected fields → OVERWRITE with fresh scan data
|
|
520
|
+
4. Manual fields → PRESERVE existing value
|
|
521
|
+
5. New fields (from harness update) → ADD with null
|
|
522
|
+
6. Unknown fields (user-added) → PRESERVE
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
| Field | Source | Type | Who Reads It |
|
|
526
|
+
|-------|--------|------|-------------|
|
|
527
|
+
| harnessVersion | Harness package.json version | Auto-detect | Conformity (version mismatch WARN), upgrade detection |
|
|
528
|
+
| scaffoldedAt | Scaffold execution timestamp | Auto-detect | Audit trail, staleness detection |
|
|
529
|
+
| project | package.json name | Auto-detect | Slug generation, issue tracking |
|
|
530
|
+
| repo | git remote | Auto-detect | git push, PR creation |
|
|
531
|
+
| issueRepo | User-configured (may differ from repo) | Manual | gh issue commands |
|
|
532
|
+
| dev.start | Makefile or package.json scripts.dev | Auto-detect | Ship workflow (start dev server) |
|
|
533
|
+
| dev.apiBase | Makefile port mappings | Auto-detect | Ship workflow (health checks), agents |
|
|
534
|
+
| dev.uiBase | Makefile port mappings | Auto-detect | Ship workflow (UI checks), Quinn |
|
|
535
|
+
| dev.testCmd | package.json scripts.test or "bun test" | Auto-detect | Ship workflow (run tests), gates |
|
|
536
|
+
| dev.typeCheck | tsconfig.json exists → "bunx tsc --noEmit" | Auto-detect | Ship workflow (type checking) |
|
|
537
|
+
| dev.preStart | User-configured (e.g., seed DB, start dependencies) | Manual | Pre-dev setup commands |
|
|
538
|
+
| prod.rebuild | Makefile target `rebuild` (if exists) | Auto-detect | Ship workflow (container deploy) |
|
|
539
|
+
| prod.apiBase | Makefile port mappings | Auto-detect | Ship workflow (smoke test URL) |
|
|
540
|
+
| prod.smokeTest | Makefile target `smoke` (if exists) | Auto-detect | Ship workflow (post-deploy health check) |
|
|
541
|
+
| test.rebuild | Makefile (make test-rebuild) | Auto-detect | Ship workflow (test container rebuild) |
|
|
542
|
+
| test.start | Makefile (make prove-up) | Auto-detect | Prove workflow (start prove container) |
|
|
543
|
+
| test.stop | Makefile (make prove-down) | Auto-detect | Prove workflow (stop prove container) |
|
|
544
|
+
| test.apiBase | Makefile port mappings | Auto-detect | Ship/prove workflows (test container URL) |
|
|
545
|
+
| pages | Router file scan (detected path) | Auto-detect | Quinn (URL navigation), AGENTS.md |
|
|
546
|
+
| consumers | CODE-MAP consumer modules | Auto-detect | AGENTS.md, Marcus (cascade impact) |
|
|
547
|
+
| contextDocs | User-configured | Manual | DISCOVERY phase (additional reading) |
|
|
548
|
+
| codeCommittedPaths | User-configured | Manual | Gates (code-committed check) |
|
|
549
|
+
| specs | Convention (specs/) | Auto-detect | Gate discovery |
|
|
550
|
+
| ci.runner | User-configured | Manual | CI workflow (GitHub Actions runner) |
|
|
551
|
+
| ci.bunVersion | User-configured | Manual | CI workflow (Bun version) |
|
|
552
|
+
| ci.branches | User-configured | Manual | CI workflow (push trigger branches) |
|
|
553
|
+
| mcp | Auto-detect from .claude/settings.json | Hybrid | Agent briefs (available tools), AGENTS.md |
|
|
554
|
+
| envVars | .env.example (if exists) | Auto-detect | Agents know what env vars the project expects |
|
|
555
|
+
|
|
556
|
+
**Null means skip, not guess:** Undetected auto-detect fields are null. Unconfigured manual fields are null. Null causes the dependent workflow step to SKIP — no agent spawned, no command run.
|
|
557
|
+
|
|
558
|
+
### AGENTS.md
|
|
559
|
+
|
|
560
|
+
Produced by: `scripts/scaffold-project.ts` → `generateAgentsMd()`
|
|
561
|
+
Read by: All agents (explicitly — "Read AGENTS.md first"), copilot-instructions.md (pointer), CLAUDE.md (via @AGENTS.md bridge)
|
|
562
|
+
|
|
563
|
+
**Sizing constraint:** MUST be under 150 lines. Scaffold WARNS if over. Empty sections are OMITTED (no consumers = no Consumers section).
|
|
564
|
+
|
|
565
|
+
| Section | Source | Purpose | Omit when |
|
|
566
|
+
|---------|--------|---------|-----------|
|
|
567
|
+
| Project Identity | README.md first paragraph (skip frontmatter + HTML) | Agent knows what the project is | Never — always present |
|
|
568
|
+
| Hard Constraints | Preserved from existing AGENTS.md + extract-constraints | Rules agents can't discover from code | Empty placeholder on new project |
|
|
569
|
+
| Commands | rungate.json testCmd + Makefile targets + CLI commands | Agent knows how to build/test/deploy/create files | Never — at minimum has "bun test" |
|
|
570
|
+
| Where to Create Things | Static table (from File Location Contract) | Agent knows correct location for every file type | Never |
|
|
571
|
+
| Harness-Managed Files | Static table (from File Ownership Model) | Agent knows what NOT to edit + how to customize | Never |
|
|
572
|
+
| Available MCP Servers | rungate.json mcp field | Agent knows what tools are available | No MCP servers configured |
|
|
573
|
+
| Project Structure | Directory scan | Agent knows where to put code | Never — at minimum has src/ |
|
|
574
|
+
| Code Style | Detected: runtime (Bun/Node), language (TS/JS), module system (ESM/CJS) | Agent follows conventions | Never |
|
|
575
|
+
| Documentation Routing | docs/ scan (top 10 by relevance priority) + CODE-MAP first | Agent finds the right doc fast | No docs/ dir |
|
|
576
|
+
| Specs | specs/ frontmatter scan | Agent knows governing specs | No specs |
|
|
577
|
+
| Environment | rungate.json dev/prod sections | Agent knows ports, commands | All values null |
|
|
578
|
+
| Pages | rungate.json pages map | Agent knows URL paths | No pages detected |
|
|
579
|
+
| Consumers | rungate.json consumers | Agent knows cascade impact | No consumers detected |
|
|
580
|
+
| Boundaries | Static (never rebuild, never commit creds) | What agents must not do | Never |
|
|
581
|
+
| Workflow | git remote + Makefile targets | Agent knows how to ship | No git remote |
|
|
582
|
+
| Reference Files | reference/ scan | Historical docs index | reference/ empty |
|
|
583
|
+
|
|
584
|
+
### .claude/agents/*.md
|
|
585
|
+
|
|
586
|
+
Produced by: `scripts/scaffold-project.ts` → `generateAgentBriefs()`
|
|
587
|
+
Read by: Claude Code (auto-loaded as system prompt when agent is spawned via subagent_type)
|
|
588
|
+
|
|
589
|
+
| Agent | Config from rungate.json | Layer 1 templates embedded | Purpose |
|
|
590
|
+
|-------|-------------------------|---------------------------|---------|
|
|
591
|
+
| marcus.md | Source dirs, consumers, test commands, fallow step, envVars | RCA protocol, blast radius checklist, regression prevention, read-before-write protocol, prevention-oriented fix, evidence hierarchy | Implementation: code, tests, evidence |
|
|
592
|
+
| quinn.md | Pages map, dev/test ports, viewport, components | Quinn journey decision tree, evidence hierarchy | QA: typed journey testing, a11y-first assertions |
|
|
593
|
+
| rook.md | Source dirs, .gitignore entries, envVars | Evidence hierarchy | Security: scan targets, secret detection, OWASP patterns |
|
|
594
|
+
| serena.md | ADR location (docs/adr/), specs location (specs/), module boundaries from CODE-MAP | Evidence hierarchy | Architecture: ADR review, cross-boundary analysis |
|
|
595
|
+
| aditi.md | Component paths from CODE-MAP, pages map | Evidence hierarchy | Design: component specs, a11y requirements |
|
|
596
|
+
|
|
597
|
+
These briefs are ALWAYS regenerated on re-scaffold — they're harness-owned templates, not user-customized. Config values come from rungate.json. Layer 1 methodology templates come from the harness `prompts/` directory. Null config values shown as "not configured — edit .claude/rungate.json."
|
|
598
|
+
|
|
599
|
+
### Cross-Tool Bridge
|
|
600
|
+
|
|
601
|
+
| File | Purpose | Generated by |
|
|
602
|
+
|------|---------|-------------|
|
|
603
|
+
| AGENTS.md | Cross-vendor standard (30+ tools) | scaffold Phase 2.1 |
|
|
604
|
+
| CLAUDE.md | Claude Code primary config | scaffold Phase 0.17 |
|
|
605
|
+
| .github/copilot-instructions.md | GitHub Copilot (chat + coding agent) — NOT CI/CD | scaffold Phase 0.14 |
|
|
606
|
+
|
|
607
|
+
Scaffold Phase 0.17: If CLAUDE.md exists and doesn't contain `@AGENTS.md`, prepend `@AGENTS.md` as first line. If CLAUDE.md doesn't exist, create minimal one with `@AGENTS.md` import.
|
|
608
|
+
|
|
609
|
+
**copilot-instructions.md content** (created at Phase 0.14):
|
|
610
|
+
```markdown
|
|
611
|
+
# Copilot Instructions
|
|
612
|
+
|
|
613
|
+
For project context, conventions, constraints, and commands, see [AGENTS.md](../AGENTS.md) in the repository root.
|
|
614
|
+
|
|
615
|
+
AGENTS.md is the single source of truth for all AI tools working on this project.
|
|
616
|
+
```
|
|
617
|
+
|
|
618
|
+
### CI/CD Workflows
|
|
619
|
+
|
|
620
|
+
Produced by: Harness-owned templates in Phase 0
|
|
621
|
+
Ownership: **Harness-owned** — always regenerated. Customizable settings come from rungate.json, not from editing the workflow file.
|
|
622
|
+
|
|
623
|
+
The harness provides two workflow files that enforce gates in the CI pipeline. Local gates can be skipped (agent ignores WARN, user bypasses). CI gates can't — they block merge mechanically.
|
|
624
|
+
|
|
625
|
+
#### rungate.json CI config
|
|
626
|
+
|
|
627
|
+
```json
|
|
628
|
+
{
|
|
629
|
+
"ci": {
|
|
630
|
+
"runner": "ubuntu-latest",
|
|
631
|
+
"bunVersion": "latest",
|
|
632
|
+
"branches": ["main"]
|
|
633
|
+
}
|
|
634
|
+
}
|
|
635
|
+
```
|
|
636
|
+
|
|
637
|
+
| Field | Default | What it controls |
|
|
638
|
+
|-------|---------|-----------------|
|
|
639
|
+
| `ci.runner` | `"ubuntu-latest"` | GitHub Actions runner (e.g., `"self-hosted"`, `"ubuntu-24.04"`) |
|
|
640
|
+
| `ci.bunVersion` | `"latest"` | Bun version in CI (pin for reproducibility) |
|
|
641
|
+
| `ci.branches` | `["main"]` | Which branches trigger push CI |
|
|
642
|
+
|
|
643
|
+
All fields optional — defaults used when null.
|
|
644
|
+
|
|
645
|
+
#### `.github/workflows/ci.yml` (Phase 0.15)
|
|
646
|
+
|
|
647
|
+
Runs on every PR and push to configured branches. Always regenerated from template + rungate.json config.
|
|
648
|
+
|
|
649
|
+
```yaml
|
|
650
|
+
# Managed by rungate — do not edit manually.
|
|
651
|
+
# Customize runner/version in .claude/rungate.json under "ci".
|
|
652
|
+
# For additional CI steps, create a separate workflow file.
|
|
653
|
+
|
|
654
|
+
name: CI
|
|
655
|
+
on:
|
|
656
|
+
pull_request:
|
|
657
|
+
push:
|
|
658
|
+
branches: [main] # from ci.branches
|
|
659
|
+
|
|
660
|
+
jobs:
|
|
661
|
+
test:
|
|
662
|
+
runs-on: ubuntu-latest # from ci.runner
|
|
663
|
+
steps:
|
|
664
|
+
- uses: actions/checkout@v4
|
|
665
|
+
- uses: oven-sh/setup-bun@v2
|
|
666
|
+
with:
|
|
667
|
+
bun-version: latest # from ci.bunVersion
|
|
668
|
+
- run: bun install
|
|
669
|
+
- run: bun test
|
|
670
|
+
- run: bunx tsc --noEmit
|
|
671
|
+
```
|
|
672
|
+
|
|
673
|
+
**What this enforces:**
|
|
674
|
+
- `bun test` runs ALL conformity suites (scaffold, spec-discovery, spec-drift, doc-hygiene, agent-validation, fallow, package-validation, tsconfig-validation) + any project tests
|
|
675
|
+
- `bunx tsc --noEmit` catches type errors
|
|
676
|
+
- PR can't merge if either fails (when branch protection requires this check)
|
|
677
|
+
|
|
678
|
+
#### `.github/workflows/gates.yml` (Phase 0.16)
|
|
679
|
+
|
|
680
|
+
Runs on PRs only. Always regenerated from template + rungate.json config.
|
|
681
|
+
|
|
682
|
+
```yaml
|
|
683
|
+
# Managed by rungate — do not edit manually.
|
|
684
|
+
# Customize runner/version in .claude/rungate.json under "ci".
|
|
685
|
+
# For additional gate checks, create a separate workflow file.
|
|
686
|
+
|
|
687
|
+
name: Harness Gates
|
|
688
|
+
on:
|
|
689
|
+
pull_request:
|
|
690
|
+
|
|
691
|
+
jobs:
|
|
692
|
+
gates:
|
|
693
|
+
runs-on: ubuntu-latest # from ci.runner
|
|
694
|
+
steps:
|
|
695
|
+
- uses: actions/checkout@v4
|
|
696
|
+
- uses: oven-sh/setup-bun@v2
|
|
697
|
+
with:
|
|
698
|
+
bun-version: latest # from ci.bunVersion
|
|
699
|
+
- run: bun install
|
|
700
|
+
- name: Secret scan
|
|
701
|
+
run: bunx rungate check-secrets --staged
|
|
702
|
+
- name: Conformity check
|
|
703
|
+
run: bun test test/scaffold-conformity.test.ts
|
|
704
|
+
- name: Spec drift check
|
|
705
|
+
run: bunx rungate check-drift
|
|
706
|
+
```
|
|
707
|
+
|
|
708
|
+
**What this enforces:**
|
|
709
|
+
- **Secret scan** — catches .env files, credentials, API keys in PR diff (HYGIENE-12 in CI)
|
|
710
|
+
- **Conformity** — all structural checks pass (misplaced files, missing frontmatter, broken refs)
|
|
711
|
+
- **Spec drift** — tests match current spec (content hash comparison)
|
|
712
|
+
|
|
713
|
+
#### Workflow rules
|
|
714
|
+
|
|
715
|
+
| Scenario | What happens |
|
|
716
|
+
|----------|-------------|
|
|
717
|
+
| File doesn't exist | Created from template |
|
|
718
|
+
| File exists | **Regenerated** — harness owns these files |
|
|
719
|
+
| User wants different runner | Set `ci.runner` in rungate.json — picked up on next scaffold |
|
|
720
|
+
| User wants custom CI steps | Create a separate workflow file (e.g., `deploy.yml`) — harness never touches it |
|
|
721
|
+
| Harness adds new gate checks | Next re-scaffold regenerates workflows with new steps |
|
|
722
|
+
| Branch protection setup | User configures in GitHub settings — harness doesn't touch |
|
|
723
|
+
|
|
724
|
+
#### What the harness does NOT create
|
|
725
|
+
|
|
726
|
+
| Workflow | Why not |
|
|
727
|
+
|----------|---------|
|
|
728
|
+
| Deploy workflows | Project-specific (containers, cloud, static hosting) |
|
|
729
|
+
| Release automation | Project-specific (versioning strategy, changelog format) |
|
|
730
|
+
| Environment-specific CI | Project-specific (staging, prod, review apps) |
|
|
731
|
+
| Custom Actions | User-owned |
|
|
732
|
+
|
|
733
|
+
These are out of scope for the harness. The harness provides the quality gates. Deploy and release are project decisions. Users create separate workflow files for these — the harness never touches them.
|
|
734
|
+
|
|
735
|
+
### Prompt Templates (`prompts/`)
|
|
736
|
+
|
|
737
|
+
All prompt templates live in the **harness repo** (`prompts/`), not in projects. The brief-assembler reads templates, fills them with project config (from rungate.json) and issue data (from workflow-state.json), and generates complete briefs at ship time.
|
|
738
|
+
|
|
739
|
+
#### Ship workflow prompts (per-issue, filled at ship time)
|
|
740
|
+
|
|
741
|
+
Every agent spawned during a ship cycle gets a filled prompt template. No agent runs without one.
|
|
742
|
+
|
|
743
|
+
| Template | Agent | Variables From Config | Status |
|
|
744
|
+
|----------|-------|---------------------|--------|
|
|
745
|
+
| discovery-brief.md | DA (DISCOVERY phase) | ${PROJECT}, ${CONTEXT_DOCS}, ${REPO}, ${SPECS} | Missing |
|
|
746
|
+
| marcus-implementation.md | Marcus | ${DEV_API}, ${DEV_UI}, ${TEST_CMD}, ${CONSUMERS}, ${SOURCE_DIRS}, ${ENV_VARS} | Missing |
|
|
747
|
+
| quinn-ui-brief.md | Quinn | ${DEV_UI}, ${PAGES}, ${TEST_API}, ${COMPONENTS}, ${VIEWPORT} | Exists |
|
|
748
|
+
| rook-security-brief.md | Rook | ${SOURCE_DIRS}, ${GITIGNORE_ENTRIES}, ${ENV_VARS}, ${CHANGED_FILES} | Missing |
|
|
749
|
+
| serena-architecture-brief.md | Serena | ${ADR_DIR}, ${SPECS_DIR}, ${MODULE_DEPS}, ${CODE_HEALTH} | Missing |
|
|
750
|
+
| aditi-design-brief.md | Aditi | ${COMPONENT_PATHS}, ${PAGES}, ${DESIGN_SYSTEM} | Missing |
|
|
751
|
+
| environment-check.md | Ship workflow | ${DEV_API}, ${DEV_UI}, ${DEV_START} | Missing |
|
|
752
|
+
| container-rebuild.md | Ship workflow | ${TEST_REBUILD} or null → skip | Missing |
|
|
753
|
+
| container-verify.md | Ship workflow | ${TEST_API}, ${SMOKE_TEST} | Missing |
|
|
754
|
+
|
|
755
|
+
#### Gate prompts (used by gate runner during scope/verify/ship/prove)
|
|
756
|
+
|
|
757
|
+
| Template | Purpose | Used by | Status |
|
|
758
|
+
|----------|---------|---------|--------|
|
|
759
|
+
| ac-adversary.md | B1 AC adversary agent | scope + verify gates | Exists |
|
|
760
|
+
| evidence-validator.md | B2 evidence validation agent | ship gate | Exists |
|
|
761
|
+
| prove-reproducer.md | B3 prove reproduction agent | prove gate | Exists |
|
|
762
|
+
|
|
763
|
+
#### Layer 1 methodology templates (harness-generic, embedded in briefs)
|
|
764
|
+
|
|
765
|
+
These are the research-derived templates that get embedded into per-agent briefs. They contain no project-specific data — they're the same for every project. The brief-assembler includes the relevant subset per agent role.
|
|
766
|
+
|
|
767
|
+
| Template | Purpose | Embedded in | Status |
|
|
768
|
+
|----------|---------|-------------|--------|
|
|
769
|
+
| rca-protocol.md | 4-phase investigation (Hermes adapted) | Marcus briefs (bug-fix issues) | Missing |
|
|
770
|
+
| blast-radius-checklist.md | Pre-implementation dependency analysis | Marcus briefs (all issues) | Missing |
|
|
771
|
+
| prevention-oriented-fix.md | Post-fix guard/type-narrow/audit | Marcus briefs (bug-fix issues) | Missing |
|
|
772
|
+
| regression-prevention-checklist.md | Test baseline, failing test first, count check | Marcus briefs (all issues) | Missing |
|
|
773
|
+
| read-before-write-protocol.md | 3:1 ratio, read callers/tests/types first | Marcus briefs (all issues) | Missing |
|
|
774
|
+
| quinn-journey-decision-tree.md | Step-by-step decision tree for UI testing | Quinn briefs (all issues with UI ACs) | Missing |
|
|
775
|
+
| evidence-hierarchy.md | S/A/B/C/D/F tier definitions + examples | All agent briefs | Missing |
|
|
776
|
+
| ac-format-guide.md | Given/When/Then + oracle format reference | DISCOVERY phase (AC authoring) | Missing |
|
|
777
|
+
| coding-principles.md | Deep modules, Zod at boundaries, assertNever, branded types, immutability, error handling, context loading architecture — EXPLAINED not just named | Marcus + Serena briefs (all issues) | Missing |
|
|
778
|
+
| testing-strategy.md | PBT, contract tests, tautological trap, boundary testing (horizontal+vertical), test evidence per AC type, mutation testing as quality signal | Marcus briefs (all issues) | Missing |
|
|
779
|
+
| escalation-decision-tree.md | When to research vs iterate, available tools, circuit breaker routing | All agent briefs | Missing |
|
|
780
|
+
|
|
781
|
+
#### Template lifecycle
|
|
782
|
+
|
|
783
|
+
1. Templates ship with the harness npm package
|
|
784
|
+
2. Projects get them via `bun install` (devDependency)
|
|
785
|
+
3. Brief-assembler reads templates + rungate.json + issue ACs → generates filled briefs
|
|
786
|
+
4. Template improvements in harness → projects get them on next `bun install`
|
|
787
|
+
5. No project-level prompts/ directory needed — everything comes from the harness
|
|
788
|
+
|
|
789
|
+
## Complete Project Inventory
|
|
790
|
+
|
|
791
|
+
Every project integrated with the harness has these files. On new projects, scaffold creates them. On existing projects, scaffold refreshes/audits them.
|
|
792
|
+
|
|
793
|
+
### Files from scaffold (harness-managed)
|
|
794
|
+
|
|
795
|
+
| File | Created by | Updated on re-run | Purpose |
|
|
796
|
+
|------|-----------|-------------------|---------|
|
|
797
|
+
| `AGENTS.md` | Phase 2.1 | Refreshed (HC preserved) | Cross-tool agent context (<150 lines) |
|
|
798
|
+
| `CODE-MAP.md` | Phase 1.1 | Refreshed if src/ changed (git SHA) | Code structure scan |
|
|
799
|
+
| `.claude/rungate.json` | Phase 1.2 | Audited (gaps reported) | Project config (ports, commands, pages) |
|
|
800
|
+
| `.claude/agents/marcus.md` | Phase 2.2 | Always regenerated | Engineer brief — source dirs, test cmds, consumers |
|
|
801
|
+
| `.claude/agents/quinn.md` | Phase 2.2 | Always regenerated | QA brief — ports, pages, viewport, methodology ref |
|
|
802
|
+
| `.claude/agents/rook.md` | Phase 2.2 | Always regenerated | Security brief — scan targets, security baseline |
|
|
803
|
+
| `.claude/agents/serena.md` | Phase 2.2 | Always regenerated | Architect brief — ADR location, module boundaries |
|
|
804
|
+
| `.claude/agents/aditi.md` | Phase 2.2 | Always regenerated | Designer brief — component paths, design system |
|
|
805
|
+
| `package.json` | Phase 0.18 | Required fields added if missing | name, type:module, scripts.test, devDep |
|
|
806
|
+
| `tsconfig.json` | Phase 0.19 | WARN on missing recommended fields | strict:true, module:ESNext |
|
|
807
|
+
| `.gitignore` | Phase 0.11 | Verified, missing entries appended | node_modules, dist, .env*, .rungate, secrets |
|
|
808
|
+
| `.github/copilot-instructions.md` | Phase 0.14 | Skip if exists | Tool-bridge: points Copilot to AGENTS.md |
|
|
809
|
+
| `.github/workflows/ci.yml` | Phase 0.15 | Always regenerated (harness-owned) | CI pipeline: bun test + typecheck on PR/push |
|
|
810
|
+
| `.github/workflows/gates.yml` | Phase 0.16 | Always regenerated (harness-owned) | Gate enforcement: secret scan + conformity + spec drift |
|
|
811
|
+
| `test/scaffold-conformity.test.ts` | Phase 0.13 | Always regenerated | Conformity + hygiene + drift tests (harness-owned) |
|
|
812
|
+
| `specs/SPEC-TEMPLATE.md` | Phase 0.12 | Updated to latest | Template for creating new specs |
|
|
813
|
+
| `CLAUDE.md` (bridge) | Phase 0.17 | Add @AGENTS.md if missing | Claude Code reads this → imports AGENTS.md |
|
|
814
|
+
|
|
815
|
+
### Files created by developer/agent
|
|
816
|
+
|
|
817
|
+
| File | Purpose | Guidance |
|
|
818
|
+
|------|---------|----------|
|
|
819
|
+
| `src/index.ts` | Entry point (stub created by scaffold) | Framework choice comes from first /goal |
|
|
820
|
+
| `src/**/*.ts` | Source code | Agent writes, harness never touches |
|
|
821
|
+
| `test/**/*.test.ts` | Project tests (not conformity) | Agent writes, harness never touches |
|
|
822
|
+
| `specs/*.md` | Governing specs | Use `bunx rungate create-spec "title"` — never create manually |
|
|
823
|
+
| `docs/*.md` | Project documentation | Scanned by AGENTS.md doc routing on re-scaffold |
|
|
824
|
+
| `docs/adr/*.md` | Architecture decision records | Use `bunx rungate create-adr "title"` — never create manually |
|
|
825
|
+
| `scripts/*.ts` | Project-specific scripts | User-created, harness never touches |
|
|
826
|
+
| `.github/workflows/*.yml` (user) | Custom CI workflows (deploy, etc.) | Harness never touches user-created workflows |
|
|
827
|
+
| `.husky/*` or `package.json` hooks | Git hooks (project-specific) | In-repo, not global |
|
|
828
|
+
| `reference/` | Archived/historical docs | Stale docs moved here by doc hygiene |
|
|
829
|
+
| `reference/rejected-constraints.md` | Rejected extract-constraints candidates | Content-hash dedup prevents resurfacing |
|
|
830
|
+
|
|
831
|
+
### Conformity Tests (auto-run via `bun test`)
|
|
832
|
+
|
|
833
|
+
The scaffold creates `test/scaffold-conformity.test.ts` which imports test suites from the harness. These run on every `bun test` in the project:
|
|
834
|
+
|
|
835
|
+
| Suite | Import | What it tests |
|
|
836
|
+
|-------|--------|---------------|
|
|
837
|
+
| **Scaffold Conformity** | `runScaffoldConformity(ROOT)` | SCs from specs with `testable: true` — auto-generates assertions from parseable SC patterns |
|
|
838
|
+
| **Spec Discovery** | `runSpecDiscovery(ROOT)` | Every .md in specs/ has frontmatter, `testable` field present, at least one testable spec exists |
|
|
839
|
+
| **Spec Drift** | `runSpecDrift(ROOT)` | Every SPEC-REF points to existing file, every testable spec referenced by at least one test |
|
|
840
|
+
| **Doc Hygiene** | `runDocHygiene(ROOT)` | Specs have `governs` field, `updated` field, no orphaned files, all specs listed in AGENTS.md |
|
|
841
|
+
| **Agent File Validation** | `runAgentFileValidation(ROOT)` | .claude/agents/ exists, all agent files have frontmatter (name + description), names lowercase, name matches filename |
|
|
842
|
+
| **Fallow** | `runFallowCheck(ROOT)` | No unused files, no unused exports, no unused deps, no circular deps |
|
|
843
|
+
| **Constraint Candidates** | HYGIENE-6 (inside Doc Hygiene) | Warns about unreviewed extract-constraints candidates |
|
|
844
|
+
| **Package Validation** | `runPackageValidation(ROOT)` | package.json has name, type:module, scripts.test, devDependencies.rungate — FAIL if required missing |
|
|
845
|
+
| **Tsconfig Validation** | `runTsconfigValidation(ROOT)` | tsconfig.json exists, strict:true recommended — WARN only, never FAIL |
|
|
846
|
+
|
|
847
|
+
**What this means for developers/agents:** Run `bun test` — if it passes, the project is conformant. If it fails, the failure message says exactly what's wrong and how to fix it. Zero manual checking needed.
|
|
848
|
+
|
|
849
|
+
### AGENTS.md Required Sections
|
|
850
|
+
|
|
851
|
+
AGENTS.md must include guidance on how to work with the harness. The Commands and Specs sections are mandatory:
|
|
852
|
+
|
|
853
|
+
```markdown
|
|
854
|
+
## Commands
|
|
855
|
+
|
|
856
|
+
| Action | Command |
|
|
857
|
+
|--------|---------|
|
|
858
|
+
| Install | `bun install` |
|
|
859
|
+
| Test | `bun test` |
|
|
860
|
+
| Type check | `bunx tsc --noEmit` |
|
|
861
|
+
| Conformity | `bun test test/scaffold-conformity.test.ts` |
|
|
862
|
+
| Sync spec tests | `bunx rungate sync-tests .` |
|
|
863
|
+
| Create spec | `bunx rungate create-spec "title"` |
|
|
864
|
+
| Create ADR | `bunx rungate create-adr "title"` |
|
|
865
|
+
| Extract constraints | `bunx rungate extract-constraints .` |
|
|
866
|
+
| Re-scaffold | `bun ~/Projects/rungate/scripts/scaffold-project.ts .` |
|
|
867
|
+
|
|
868
|
+
## Specs
|
|
869
|
+
|
|
870
|
+
Create specs with `bunx rungate create-spec "title"` — creates at `specs/` with correct frontmatter.
|
|
871
|
+
Create ADRs with `bunx rungate create-adr "title"` — creates at `docs/adr/` with auto-incremented number.
|
|
872
|
+
Set `testable: true` in frontmatter → `bun test` auto-generates assertions from SCs.
|
|
873
|
+
SCs must follow parseable patterns (see template for examples).
|
|
874
|
+
|
|
875
|
+
## Harness-Managed Files
|
|
876
|
+
|
|
877
|
+
These files are managed by rungate and regenerated on re-scaffold. **Do not edit them directly.**
|
|
878
|
+
|
|
879
|
+
| File | How to customize | What NOT to do |
|
|
880
|
+
|------|-----------------|----------------|
|
|
881
|
+
| `.github/workflows/ci.yml` | Set `ci.runner`, `ci.bunVersion`, `ci.branches` in `.claude/rungate.json` | Don't edit the YAML — it regenerates |
|
|
882
|
+
| `.github/workflows/gates.yml` | Settings from `.claude/rungate.json` `ci` section | Don't edit the YAML — it regenerates |
|
|
883
|
+
| `.claude/agents/*.md` | Settings from `.claude/rungate.json` (ports, pages, consumers) | Don't edit briefs — they regenerate |
|
|
884
|
+
| `test/scaffold-conformity.test.ts` | Runs automatically — no config needed | Don't edit — it regenerates |
|
|
885
|
+
| `CODE-MAP.md` | Auto-generated from code scan | Don't edit — it regenerates |
|
|
886
|
+
|
|
887
|
+
**To add custom CI steps:** Create a separate workflow file (e.g., `.github/workflows/deploy.yml`). The harness never touches user-created workflow files.
|
|
888
|
+
|
|
889
|
+
**To change project config:** Edit `.claude/rungate.json` then re-scaffold. Changes flow into agent briefs, AGENTS.md, and CI workflows.
|
|
890
|
+
|
|
891
|
+
## Where to Create Things
|
|
892
|
+
|
|
893
|
+
Everything must be self-contained in the repo. No global installs.
|
|
894
|
+
|
|
895
|
+
| I need to create... | Put it here | Not here |
|
|
896
|
+
|---------------------|-------------|----------|
|
|
897
|
+
| A new spec | `bunx rungate create-spec "title"` → `specs/` | Don't create manually at root |
|
|
898
|
+
| A new ADR | `bunx rungate create-adr "title"` → `docs/adr/` | Don't create in specs/ |
|
|
899
|
+
| A git pre-commit hook | `package.json` scripts or `.husky/pre-commit` | Not ~/.gitconfig or global hooks |
|
|
900
|
+
| A git pre-push hook | `package.json` scripts or `.husky/pre-push` | Not ~/.gitconfig or global hooks |
|
|
901
|
+
| A project script | `scripts/my-script.ts` | Not root, not src/ |
|
|
902
|
+
| A custom CI workflow | `.github/workflows/my-workflow.yml` | Don't edit ci.yml or gates.yml |
|
|
903
|
+
| A Claude Code hook | `.claude/settings.json` in the repo | Not global Claude settings |
|
|
904
|
+
| Documentation | `docs/my-doc.md` | Not root (HYGIENE-7 enforces) |
|
|
905
|
+
| Test files | `test/my-test.test.ts` | Not src/ |
|
|
906
|
+
```
|
|
907
|
+
|
|
908
|
+
## File Location Contract (anti-sprawl)
|
|
909
|
+
|
|
910
|
+
Every file type has ONE correct location. Agents and developers follow this contract. Conformity tests enforce it mechanically.
|
|
911
|
+
|
|
912
|
+
| What | Where | Wrong place | Enforcement |
|
|
913
|
+
|------|-------|-------------|-------------|
|
|
914
|
+
| Specs | `specs/*.md` | NOT docs/, NOT root | HYGIENE-8: FAIL |
|
|
915
|
+
| ADRs | `docs/adr/*.md` | NOT specs/, NOT root | HYGIENE-9: FAIL |
|
|
916
|
+
| Documentation | `docs/*.md` | NOT root (except AGENTS.md, CODE-MAP.md, README.md, CLAUDE.md, CONTRIBUTING.md, ARCHITECTURE.md, PRINCIPLES.md) | HYGIENE-7: FAIL |
|
|
917
|
+
| Archived docs | `reference/` | NOT docs/ (moved when stale) | Staleness audit |
|
|
918
|
+
| Agent briefs | `.claude/agents/*.md` | NOT root, NOT docs/ | Agent validation |
|
|
919
|
+
| Project config | `.claude/rungate.json` | NOT root | Schema check |
|
|
920
|
+
| Test files | `test/*.test.ts` | NOT src/ | Convention |
|
|
921
|
+
| Source code | `src/` | NOT root | Convention |
|
|
922
|
+
| Harness hooks/prompts | In harness (npm package) | NOT in project repo | Self-containment |
|
|
923
|
+
| Project git hooks | `package.json` scripts or `.husky/` | NOT global (~/.git/hooks) | Must be in-repo |
|
|
924
|
+
| Project scripts | `scripts/` | NOT root, NOT src/ | Convention |
|
|
925
|
+
| CI/CD workflows (harness) | `.github/workflows/ci.yml`, `gates.yml` | Don't edit — harness-owned | Regenerated |
|
|
926
|
+
| CI/CD workflows (project) | `.github/workflows/*.yml` (user-created) | NOT outside .github/workflows/ | Convention |
|
|
927
|
+
| Claude Code hooks | `.claude/settings.json` | NOT global settings | In-repo for portability |
|
|
928
|
+
|
|
929
|
+
### Self-containment principle
|
|
930
|
+
|
|
931
|
+
Everything the project needs must be IN the repo. When someone clones the repo and runs `bun install`, all hooks, workflows, tests, and config should work. No global installs, no machine-specific hooks, no external dependencies beyond the harness npm package.
|
|
932
|
+
|
|
933
|
+
| Need | Where it lives | Why |
|
|
934
|
+
|------|---------------|-----|
|
|
935
|
+
| Git pre-commit hook | `package.json` scripts or `.husky/` in repo | Global hooks don't follow the clone |
|
|
936
|
+
| Git pre-push hook | `package.json` scripts or `.husky/` in repo | Global hooks don't follow the clone |
|
|
937
|
+
| Claude Code hooks | `.claude/settings.json` in repo | Per-machine settings don't transfer |
|
|
938
|
+
| CI/CD workflows | `.github/workflows/` in repo | Actions read from repo, not external |
|
|
939
|
+
| Test suites | `test/` in repo (imports from harness) | Tests must run on any machine |
|
|
940
|
+
| Project-specific scripts | `scripts/` in repo | Not on the harness, not global |
|
|
941
|
+
|
|
942
|
+
**The distinction:**
|
|
943
|
+
- **Harness-provided** (ci.yml, gates.yml, conformity test, agent briefs) → comes from the harness package, regenerated on scaffold
|
|
944
|
+
- **Project-specific** (deploy.yml, custom hooks, project scripts) → created by developer/agent, lives in repo, never touched by harness
|
|
945
|
+
- **Global** → NOT ALLOWED. Nothing should require a global install or machine-specific setup beyond `bun install`
|
|
946
|
+
|
|
947
|
+
If a markdown file doesn't fit any category, the agent must ask before creating it.
|
|
948
|
+
|
|
949
|
+
### Mechanical enforcement (HYGIENE-7/8/9)
|
|
950
|
+
|
|
951
|
+
```
|
|
952
|
+
HYGIENE-7: No .md files at root except AGENTS.md, CODE-MAP.md, README.md,
|
|
953
|
+
CLAUDE.md, CONTRIBUTING.md, ARCHITECTURE.md, PRINCIPLES.md — FAIL with fix instruction
|
|
954
|
+
HYGIENE-8: No spec-like files outside specs/ (files with "testable:"
|
|
955
|
+
frontmatter in wrong location) — FAIL with move command
|
|
956
|
+
HYGIENE-9: No ADR-like files outside docs/adr/ (files matching ADR-NNN
|
|
957
|
+
pattern in wrong location) — FAIL with move command
|
|
958
|
+
```
|
|
959
|
+
|
|
960
|
+
These run on every `bun test`. Misplaced files are FAIL, not WARN. The error message includes the fix command.
|
|
961
|
+
|
|
962
|
+
## CLI Commands for File Creation
|
|
963
|
+
|
|
964
|
+
Agents create specs and ADRs via CLI — guarantees correct location, format, and frontmatter. Never copy templates manually.
|
|
965
|
+
|
|
966
|
+
| Command | What it creates | Where |
|
|
967
|
+
|---------|----------------|-------|
|
|
968
|
+
| `bunx rungate create-spec "title"` | Spec with frontmatter + section stubs | `specs/title-slug.md` |
|
|
969
|
+
| `bunx rungate create-adr "title"` | ADR with template + auto-incremented number | `docs/adr/ADR-NNN-title-slug.md` |
|
|
970
|
+
| `bunx rungate sync-tests .` | Test assertions from testable specs | `test/spec-compliance-auto.test.ts` |
|
|
971
|
+
| `bunx rungate extract-constraints .` | Constraint candidates from docs | Candidates for review |
|
|
972
|
+
|
|
973
|
+
## Gate-Time Conformity Enforcement
|
|
974
|
+
|
|
975
|
+
Conformity tests run at TWO points in the ship cycle — not just at the end.
|
|
976
|
+
|
|
977
|
+
```
|
|
978
|
+
SCOPE GATE (before work starts):
|
|
979
|
+
→ runs test/scaffold-conformity.test.ts
|
|
980
|
+
→ verifies repo is clean BEFORE agent starts coding
|
|
981
|
+
→ catches: stale state, broken refs, misplaced files from prior work
|
|
982
|
+
→ FAIL = blocked. Fix before proceeding.
|
|
983
|
+
|
|
984
|
+
VERIFY GATE (after work complete):
|
|
985
|
+
→ runs test/scaffold-conformity.test.ts AGAIN
|
|
986
|
+
→ catches anything the agent introduced:
|
|
987
|
+
- new spec in wrong dir? FAIL
|
|
988
|
+
- new .md at root? FAIL
|
|
989
|
+
- frontmatter missing? FAIL
|
|
990
|
+
- circular dep introduced? FAIL
|
|
991
|
+
→ FAIL = blocked. Fix before shipping.
|
|
992
|
+
```
|
|
993
|
+
|
|
994
|
+
Conformity suite runs in <500ms (pure file system checks, no network/LLM).
|
|
995
|
+
|
|
996
|
+
### Findings flow (detection → structured output → agent fix → re-verify)
|
|
997
|
+
|
|
998
|
+
```
|
|
999
|
+
bun test (in project)
|
|
1000
|
+
→ conformity suites run (HYGIENE-1 through HYGIENE-10)
|
|
1001
|
+
→ extract-constraints runs (signal phrases → candidates, staleness)
|
|
1002
|
+
→ HYGIENE-REPORT writes .rungate/conformity-findings.json:
|
|
1003
|
+
{ findings: [{ruleId, severity, file, message, fixCommand}],
|
|
1004
|
+
constraintCandidates: [{rule, source, hash, status}],
|
|
1005
|
+
staleness: [{file, daysSince, threshold, type}] }
|
|
1006
|
+
→ Gate runner (run-gate.ts) reads .rungate/conformity-findings.json
|
|
1007
|
+
→ Prints structured output with fix commands
|
|
1008
|
+
→ Stores conformityFindings in workflow-state.json
|
|
1009
|
+
→ Agent reads workflow-state.json → executes fixCommands → re-runs bun test
|
|
1010
|
+
```
|
|
1011
|
+
|
|
1012
|
+
Agent discovery paths:
|
|
1013
|
+
- **During ship gate:** workflow-state.json `conformityFindings` field (gate runner puts it there)
|
|
1014
|
+
- **Running bun test directly:** HYGIENE-REPORT prints path + details to terminal
|
|
1015
|
+
- **Reading project docs:** AGENTS.md Commands table → `cat .rungate/conformity-findings.json`
|
|
1016
|
+
|
|
1017
|
+
Nothing is lost to stdout. Constraint candidates persist in the JSON file until applied or rejected.
|
|
1018
|
+
|
|
1019
|
+
Structural violations are FAIL, not WARN:
|
|
1020
|
+
|
|
1021
|
+
| Check | Severity | Rationale |
|
|
1022
|
+
|-------|----------|-----------|
|
|
1023
|
+
| Misplaced spec/ADR/doc | FAIL | Wrong location = harness can't find it |
|
|
1024
|
+
| Missing frontmatter | FAIL | No frontmatter = no test generation |
|
|
1025
|
+
| Broken file references | FAIL | Broken ref = broken routing |
|
|
1026
|
+
| Missing package.json required fields | FAIL | name, type:module, scripts.test, devDep required for harness |
|
|
1027
|
+
| Secret patterns in staged files | FAIL | HYGIENE-12 — mechanical catch for 44% AI security flaw rate |
|
|
1028
|
+
| Spec drift (hash mismatch) | FAIL | Tests don't match current spec = stale tests |
|
|
1029
|
+
| AGENTS.md over 150 lines | WARN | May be legitimate for complex projects |
|
|
1030
|
+
| Stale docs | WARN | Stale ≠ wrong, needs human judgment |
|
|
1031
|
+
| Missing tsconfig.json strict:true | WARN | Recommended, not required |
|
|
1032
|
+
| Unreviewed constraint candidates | WARN | 70% false positive rate — human review needed |
|
|
1033
|
+
|
|
1034
|
+
## Agent Brief Core Principles
|
|
1035
|
+
|
|
1036
|
+
Every agent brief (`.claude/agents/*.md`) includes a shared Core Principles block. These are non-negotiable rules derived from a year of learned failures (25+ correction entries, 50 tiered learned rules).
|
|
1037
|
+
|
|
1038
|
+
### Universal block (in EVERY agent brief, ~8 lines)
|
|
1039
|
+
|
|
1040
|
+
```
|
|
1041
|
+
## Core Principles
|
|
1042
|
+
- Verify before asserting — try it, then report what happened
|
|
1043
|
+
- Never report PASS with known gaps — list every gap
|
|
1044
|
+
- Read AGENTS.md FIRST — project identity, constraints, commands
|
|
1045
|
+
- Run `bun test` after every change — conformity is mechanical
|
|
1046
|
+
- Null in config means skip — never guess values
|
|
1047
|
+
- Research before guessing — use available tools (see Research and Escalation below)
|
|
1048
|
+
```
|
|
1049
|
+
|
|
1050
|
+
### Role-specific rules (added per agent)
|
|
1051
|
+
|
|
1052
|
+
| Agent | Key rules from learned failures |
|
|
1053
|
+
|-------|-------------------------------|
|
|
1054
|
+
| Marcus | Read governing spec BEFORE coding. Modify ONLY files in brief. Commit after every green cycle. Never duplicate components — add props. Schema validation at boundaries (Zod). Every new module gets a drift test. |
|
|
1055
|
+
| Quinn | Test full workflow (trigger → result → USE result). Compare against visual spec if referenced. Test edge cases after golden path. Verify fix commit deployed before testing. Open every rendered URL. |
|
|
1056
|
+
| Rook | Check git diff for .env, credentials, tokens, API keys. Scan pattern siblings (shared-import files). |
|
|
1057
|
+
| Serena | Read existing ADRs before proposing new ones. Every structural decision gets a spec. |
|
|
1058
|
+
|
|
1059
|
+
## Coding Principles (shipped with harness as `prompts/coding-principles.md`)
|
|
1060
|
+
|
|
1061
|
+
AI agents produce 10-50x more boundary violations per session than human engineers (Ousterhout observation, confirmed by SWE-bench analysis). These principles are EXPLAINED in the template — agents need rationale, not just rules. Context7 MCP provides current library docs to supplement stale training data.
|
|
1062
|
+
|
|
1063
|
+
### Deep modules over shallow wrappers
|
|
1064
|
+
|
|
1065
|
+
Modules should provide powerful functionality behind simple interfaces. A module with a 5-method interface that handles 15 edge cases internally is better than a module with a 15-method interface that pushes edge cases to callers. AI agents default to creating pass-through wrappers — the brief must explicitly say "absorb complexity, don't redistribute it."
|
|
1066
|
+
|
|
1067
|
+
**Gate signal:** If a new module has more exported functions than internal functions, it's likely shallow. WARN if exports > 5 for a single module file.
|
|
1068
|
+
|
|
1069
|
+
### Boundary validation with Zod
|
|
1070
|
+
|
|
1071
|
+
Validate at system boundaries (user input, API responses, config files, external data) using Zod schemas. Internal function calls between trusted modules do NOT need runtime validation — TypeScript types handle that. The boundary is where untrusted data enters the system.
|
|
1072
|
+
|
|
1073
|
+
**Template explains:** "Boundary = where data crosses a trust line. HTTP request body: boundary. Function arg from your own module: not boundary. Config file read from disk: boundary. Object passed between your functions: not boundary."
|
|
1074
|
+
|
|
1075
|
+
### Exhaustive matching (assertNever)
|
|
1076
|
+
|
|
1077
|
+
Every switch/case on a union type must have a default branch that calls `assertNever(x)` — a function that accepts `never` and throws. This catches unhandled variants at compile time when new variants are added. Without it, new enum values silently fall through.
|
|
1078
|
+
|
|
1079
|
+
```typescript
|
|
1080
|
+
function assertNever(x: never): never {
|
|
1081
|
+
throw new Error(`Unexpected value: ${x}`);
|
|
1082
|
+
}
|
|
1083
|
+
```
|
|
1084
|
+
|
|
1085
|
+
### Immutability by default
|
|
1086
|
+
|
|
1087
|
+
Use `readonly` arrays and objects. Use `as const` for literal types. Mutate only when performance requires it and document why. AI agents frequently introduce mutation bugs because they copy patterns without understanding ownership semantics.
|
|
1088
|
+
|
|
1089
|
+
### Branded types for domain values
|
|
1090
|
+
|
|
1091
|
+
IDs, slugs, paths, and other stringly-typed values should use branded types to prevent accidental interchange:
|
|
1092
|
+
|
|
1093
|
+
```typescript
|
|
1094
|
+
type IssueId = number & { readonly __brand: "IssueId" };
|
|
1095
|
+
type SlugId = string & { readonly __brand: "SlugId" };
|
|
1096
|
+
```
|
|
1097
|
+
|
|
1098
|
+
### Error handling: fail loud at boundaries, propagate inside
|
|
1099
|
+
|
|
1100
|
+
At boundaries: catch, log, return structured error. Inside modules: let errors propagate — don't catch-and-rethrow with less information. Never swallow errors silently. Never use empty catch blocks.
|
|
1101
|
+
|
|
1102
|
+
### One export per concern
|
|
1103
|
+
|
|
1104
|
+
Each file exports one primary thing. Utility files with 10+ exports are a code smell. If a file has unrelated exports, split it. The test for "related": could you rename the file to describe all exports in 3 words?
|
|
1105
|
+
|
|
1106
|
+
### Context loading architecture (D-23)
|
|
1107
|
+
|
|
1108
|
+
Anthropic's Claude Code architecture confirms the routing table approach:
|
|
1109
|
+
|
|
1110
|
+
1. **CLAUDE.md** — always loaded for every non-fork subagent automatically. Project rules, conventions, bridge to AGENTS.md
|
|
1111
|
+
2. **AGENTS.md** — loaded via CLAUDE.md bridge. Routing table telling agents WHERE to find methodology docs
|
|
1112
|
+
3. **Skills** — loaded on demand. Methodology files (coding principles, testing strategy, RCA protocol). Can be preloaded per agent type via frontmatter `skills` field
|
|
1113
|
+
4. **Agent definition frontmatter** — `skills: [skill-name]` preloads specific skills for specific agent types
|
|
1114
|
+
5. **Brief/task message** — task-specific instructions + pointers to skills/docs. Stays small (~35 lines L3)
|
|
1115
|
+
|
|
1116
|
+
Templates are condensed reference cards (~20-30 lines each), not full spec sections. Agent reads the card, follows the steps. Full rationale lives in the spec. Context budget: heaviest agent (Marcus bug-fix) uses ~314 lines / 3.1% of 200k context window. Full 5-agent cycle uses 11.1%.
|
|
1117
|
+
|
|
1118
|
+
## Testing Strategy (shipped with harness as `prompts/testing-strategy.md`)
|
|
1119
|
+
|
|
1120
|
+
AI-generated tests have 91% line coverage but only 34% mutation score — they test the implementation, not the behavior. The testing pyramid INVERTS for AI code: property-based > contract > integration > unit.
|
|
1121
|
+
|
|
1122
|
+
### Testing pyramid for AI-authored code
|
|
1123
|
+
|
|
1124
|
+
Traditional pyramid (unit > integration > e2e) assumes humans write focused units. AI agents write sprawling functions that pass unit tests trivially. The effective pyramid:
|
|
1125
|
+
|
|
1126
|
+
1. **Property-based testing (PBT)** — highest value. Define invariants ("output is always sorted", "length never negative", "round-trip encode/decode is identity"). Anthropic's own research found 984 bugs across 100 packages using PBT. 56% were true bugs.
|
|
1127
|
+
2. **Contract tests** — verify module boundaries. Consumer-driven: the consumer defines what it expects, the provider proves it delivers. Catches the integration seam where AI agents create the most bugs.
|
|
1128
|
+
3. **Integration tests** — test real data flow through real dependencies. No mocks for databases, APIs, file systems unless the real thing is genuinely unavailable. Mocked tests pass; prod breaks.
|
|
1129
|
+
4. **Unit tests** — lowest priority for AI code. Still valuable for pure functions with complex logic. But an AI agent writing unit tests for its own code is the tautological testing trap — testing WHAT it built, not WHETHER what it built is correct.
|
|
1130
|
+
|
|
1131
|
+
### Tautological testing trap
|
|
1132
|
+
|
|
1133
|
+
AI writes implementation → AI writes test for that implementation → test passes → but the test only verifies the code does what the code does, not what the SPEC says. The test is a tautology.
|
|
1134
|
+
|
|
1135
|
+
**Mitigation:** Writer and verifier must be separate agents (SC-66). Marcus writes code + tests. Quinn verifies behavior. Gate checks that test assertions reference AC thresholds, not implementation details.
|
|
1136
|
+
|
|
1137
|
+
### Consumer-driven contract tests
|
|
1138
|
+
|
|
1139
|
+
When module A depends on module B: A defines a contract ("I expect B.get(id) to return `{name: string, active: boolean}`"). B's test suite includes A's contract as a test case. If B changes its return type, A's contract test fails BEFORE the integration breaks.
|
|
1140
|
+
|
|
1141
|
+
**Gate signal:** If a module has >3 importers and no contract test, WARN.
|
|
1142
|
+
|
|
1143
|
+
### Mutation testing as quality signal
|
|
1144
|
+
|
|
1145
|
+
Coverage % measures lines executed, not behavior verified. Mutation testing modifies code (change `>` to `>=`, remove a line, swap a constant) and checks if tests catch the mutation. Kill rate = test quality.
|
|
1146
|
+
|
|
1147
|
+
**Harness integration:** Future work (see Evaluated Concerns). Current minimum viable: test count comparison at verify gate. Mutation testing adds ~10min per run — evaluate when CI pipeline is mature.
|
|
1148
|
+
|
|
1149
|
+
### Boundary testing (horizontal and vertical)
|
|
1150
|
+
|
|
1151
|
+
- **Horizontal:** test at module boundaries — the interface between modules. What happens when module A passes unexpected data to module B?
|
|
1152
|
+
- **Vertical:** test at layer boundaries — HTTP → service → database. What happens when the database returns unexpected data?
|
|
1153
|
+
|
|
1154
|
+
AI agents test the happy path vertically (request → response works). They skip horizontal boundaries (what if the service returns null? what if the type is wrong?).
|
|
1155
|
+
|
|
1156
|
+
**Template instruction:** "After every golden path test, write one boundary test: null input, empty array, type mismatch, concurrent access."
|
|
1157
|
+
|
|
1158
|
+
### Test evidence requirements per AC type
|
|
1159
|
+
|
|
1160
|
+
| AC Type | Minimum test evidence | Why |
|
|
1161
|
+
|---------|----------------------|-----|
|
|
1162
|
+
| CODE | Bun test output showing pass + assertion count | Mechanical proof |
|
|
1163
|
+
| UI | Quinn journey PASS + screenshot on FAIL | Functional + visual |
|
|
1164
|
+
| BUG-FIX | Regression test (fails without fix, passes with) + negative control | Proves fix, not coincidence |
|
|
1165
|
+
| OUTCOME | Live execution output showing expected state | Can't unit-test outcomes |
|
|
1166
|
+
| ANTI | Grep/test proving the anti-pattern is absent | Absence proof |
|
|
1167
|
+
|
|
1168
|
+
## Research and Escalation
|
|
1169
|
+
|
|
1170
|
+
Agents must know they have research tools and WHEN to use them. The #1 failure pattern is an agent iterating 5 times on the wrong approach instead of stopping to research. The circuit breaker (3 iterations max) should trigger research, not just stop.
|
|
1171
|
+
|
|
1172
|
+
### Available research tools (declared in AGENTS.md and agent briefs)
|
|
1173
|
+
|
|
1174
|
+
| Tool | What it does | When to use |
|
|
1175
|
+
|------|-------------|------------|
|
|
1176
|
+
| Context7 MCP | Fetches current documentation for any library, framework, SDK, API | Unknown API behavior, version-specific syntax, framework patterns |
|
|
1177
|
+
| WebSearch | Searches the web for current information | Error messages, unfamiliar patterns, "how does X work" |
|
|
1178
|
+
| Council workflow | Multi-perspective debate with parallel agents | Architecture decisions, tradeoff analysis, design disagreements |
|
|
1179
|
+
| Research skill | Parallel multi-source investigation | Complex unknowns, multi-faceted questions, API investigation |
|
|
1180
|
+
| Explore agent | Fast read-only codebase search | "Where is X defined", "which files reference Y" |
|
|
1181
|
+
|
|
1182
|
+
### Escalation decision tree (embedded in every agent brief)
|
|
1183
|
+
|
|
1184
|
+
```
|
|
1185
|
+
STUCK? (same error on iteration 2, or no progress after iteration 1)
|
|
1186
|
+
|
|
1187
|
+
1. Is it a library/framework question?
|
|
1188
|
+
→ YES → Use Context7 MCP: query the library docs
|
|
1189
|
+
→ Got answer? → Apply it
|
|
1190
|
+
→ No answer? → WebSearch for the error/pattern
|
|
1191
|
+
|
|
1192
|
+
2. Is it a codebase question? (where is X, how does Y work here)
|
|
1193
|
+
→ YES → Spawn Explore agent with specific search query
|
|
1194
|
+
→ Found it? → Apply it
|
|
1195
|
+
→ Not found? → The code may not exist yet — check with DA
|
|
1196
|
+
|
|
1197
|
+
3. Is it a design question? (should we do A or B)
|
|
1198
|
+
→ YES → Escalate to DA — don't make architecture decisions alone
|
|
1199
|
+
→ DA may invoke Council for multi-perspective analysis
|
|
1200
|
+
|
|
1201
|
+
4. Is it an unknown API or external system?
|
|
1202
|
+
→ YES → Use Research skill for parallel investigation
|
|
1203
|
+
→ Don't guess API behavior — verify it
|
|
1204
|
+
|
|
1205
|
+
5. Still stuck after research?
|
|
1206
|
+
→ STOP. Report what you tried, what you found, and what's blocking.
|
|
1207
|
+
→ Don't iterate further — the problem needs human input.
|
|
1208
|
+
```
|
|
1209
|
+
|
|
1210
|
+
### Circuit breaker → research escalation
|
|
1211
|
+
|
|
1212
|
+
```
|
|
1213
|
+
Iteration 1: Normal implementation
|
|
1214
|
+
Iteration 2: If same error → MUST research before trying again
|
|
1215
|
+
Iteration 3: STOP — report what you tried, what you researched, what's blocking
|
|
1216
|
+
```
|
|
1217
|
+
|
|
1218
|
+
Three iterations, not five. The research is clear: failure trajectories are 12-82% longer than successful ones (SWE-bench taxonomy study). CMU found 3-7 turns is optimal — more turns often degrades quality. Five iterations down a hole is four too many.
|
|
1219
|
+
|
|
1220
|
+
The research requirement at iteration 2 is mechanical — if the same error occurs twice, the gate checks that a research tool was invoked before iteration 3. Without research evidence, the agent is repeating the same mistake and must stop.
|
|
1221
|
+
|
|
1222
|
+
**Why 3 not 5:**
|
|
1223
|
+
- If you haven't solved it in 2 attempts + research, you're missing something fundamental
|
|
1224
|
+
- Each additional iteration increases vulnerability count 37.6% (IEEE-ISTAS 2025)
|
|
1225
|
+
- Stopping early preserves context window for the DA to diagnose
|
|
1226
|
+
- The DA can invoke council, research skill, or re-scope — options the agent doesn't have
|
|
1227
|
+
|
|
1228
|
+
### What goes in rungate.json
|
|
1229
|
+
|
|
1230
|
+
```json
|
|
1231
|
+
{
|
|
1232
|
+
"research": {
|
|
1233
|
+
"context7": true,
|
|
1234
|
+
"webSearch": true,
|
|
1235
|
+
"council": true,
|
|
1236
|
+
"explore": true
|
|
1237
|
+
}
|
|
1238
|
+
}
|
|
1239
|
+
```
|
|
1240
|
+
|
|
1241
|
+
These flags tell the brief-assembler which research tools to include in agent briefs. If a tool isn't available (MCP server not configured), the flag is false and the brief omits it. Agents only see tools they can actually use.
|
|
1242
|
+
|
|
1243
|
+
### What goes in AGENTS.md
|
|
1244
|
+
|
|
1245
|
+
```markdown
|
|
1246
|
+
## Research Tools
|
|
1247
|
+
|
|
1248
|
+
When stuck (2+ iterations without progress), use these before continuing:
|
|
1249
|
+
|
|
1250
|
+
| Need | Tool | Example |
|
|
1251
|
+
|------|------|---------|
|
|
1252
|
+
| Library docs | Context7 MCP | `resolve-library-id` → `query-docs` for Hono, Zod, etc. |
|
|
1253
|
+
| Error diagnosis | WebSearch | Search exact error message |
|
|
1254
|
+
| Architecture decision | Council | Escalate to DA for council debate |
|
|
1255
|
+
| Find code | Explore agent | "Where is the auth middleware defined?" |
|
|
1256
|
+
```
|
|
1257
|
+
|
|
1258
|
+
## AC Format Standard
|
|
1259
|
+
|
|
1260
|
+
Every acceptance criterion uses a structured format with an executable oracle. Prose descriptions are not ACs — they're wishes.
|
|
1261
|
+
|
|
1262
|
+
### AC ID scheme
|
|
1263
|
+
|
|
1264
|
+
- IDs are per-issue: AC-1, AC-2, AC-3 within each issue
|
|
1265
|
+
- Anti-criteria (must NOT happen) use: AC-A1, AC-A2
|
|
1266
|
+
- IDs are stable — once assigned, never renumbered even if ACs are removed
|
|
1267
|
+
- Each test must annotate which AC it covers: `// covers: AC-1`
|
|
1268
|
+
- Gate checks: every AC has at least one test. Untested AC = WARN
|
|
1269
|
+
|
|
1270
|
+
### Required fields per AC
|
|
1271
|
+
|
|
1272
|
+
```
|
|
1273
|
+
AC-NNN: [title]
|
|
1274
|
+
Given: [precondition with specific values — not "valid input" but "email=test@x.com"]
|
|
1275
|
+
When: [exact action — HTTP method + path, CLI command, or UI action]
|
|
1276
|
+
Then: [exact expected output with threshold — status code, value, element state]
|
|
1277
|
+
Verify: [executable command that returns pass/fail — the oracle]
|
|
1278
|
+
Not: [explicit exclusions — what this AC does NOT cover]
|
|
1279
|
+
Assumption: [what would break if this AC's protection is bypassed — powers mutation testing]
|
|
1280
|
+
Evidence-tier: [minimum S/A/B/C required for this AC type]
|
|
1281
|
+
```
|
|
1282
|
+
|
|
1283
|
+
### Anti-criteria format (must NOT happen)
|
|
1284
|
+
|
|
1285
|
+
```
|
|
1286
|
+
AC-A1: [title — negative assertion]
|
|
1287
|
+
Given: [precondition]
|
|
1288
|
+
When: [action that SHOULD NOT produce a result]
|
|
1289
|
+
Then: [expected ABSENCE — no output, no file, no element]
|
|
1290
|
+
Verify: [command that returns pass if the thing does NOT exist]
|
|
1291
|
+
Not: [what this anti-criterion doesn't cover]
|
|
1292
|
+
```
|
|
1293
|
+
|
|
1294
|
+
Anti-criteria catch removal and deletion requirements. Agents have 71.7% deletion recall — they often retain code that should be removed. Explicit anti-criteria force verification of absence.
|
|
1295
|
+
|
|
1296
|
+
### Four-question checklist (every AC must answer all four)
|
|
1297
|
+
|
|
1298
|
+
1. **Trigger** — what action/input produces the behavior?
|
|
1299
|
+
2. **Output** — what observable result confirms success?
|
|
1300
|
+
3. **Verification** — what command proves it? (executable, not prose)
|
|
1301
|
+
4. **Exclusions** — what is explicitly NOT in scope?
|
|
1302
|
+
|
|
1303
|
+
### Garbage test
|
|
1304
|
+
|
|
1305
|
+
Every AC is tested: "Could garbage data satisfy this criterion?" If yes, tighten until the answer is no. Examples:
|
|
1306
|
+
- BAD: "Page loads successfully" → any 200 passes, even error pages
|
|
1307
|
+
- GOOD: "GET /dashboard → 200, body contains heading 'Dashboard' + user menu shows 'jason@...'"
|
|
1308
|
+
- BAD: "Handles errors gracefully" → unfalsifiable
|
|
1309
|
+
- GOOD: "POST /api/login with wrong password → 401, body = `{error: 'invalid_credentials'}`"
|
|
1310
|
+
|
|
1311
|
+
### AC type classification
|
|
1312
|
+
|
|
1313
|
+
| AC Type | When | Evidence-tier minimum | Verification pattern |
|
|
1314
|
+
|---------|------|----------------------|---------------------|
|
|
1315
|
+
| CODE | Logic, data, API behavior | A (execution) | bun test, curl, CLI command |
|
|
1316
|
+
| UI | Visual state, layout, interaction | B (behavioral) | browser_snapshot assertion |
|
|
1317
|
+
| OUTCOME | End-to-end user flow | A (execution) | Full flow reproduction |
|
|
1318
|
+
| BUG-FIX | Regression from reported issue | S (dual-arm) | Fails before fix, passes after |
|
|
1319
|
+
| ANTI | Must NOT exist/happen | A (execution) | grep/test confirms absence |
|
|
1320
|
+
|
|
1321
|
+
### Examples per type
|
|
1322
|
+
|
|
1323
|
+
**CODE:**
|
|
1324
|
+
```
|
|
1325
|
+
AC-1: Login returns JWT on valid credentials
|
|
1326
|
+
Given: user exists with email="test@x.com", password="correct-pw"
|
|
1327
|
+
When: POST /api/auth/login with {email, password}
|
|
1328
|
+
Then: 200, body contains JWT with exp > now + 3600
|
|
1329
|
+
Verify: curl -s -X POST localhost:${DEV_API}/api/auth/login -d '{"email":"test@x.com","password":"correct-pw"}' | jq -e '.token'
|
|
1330
|
+
Not: No OAuth, no SSO, no MFA in this AC
|
|
1331
|
+
Assumption: If JWT validation is bypassed, auth is broken
|
|
1332
|
+
Evidence-tier: A
|
|
1333
|
+
```
|
|
1334
|
+
|
|
1335
|
+
**UI:**
|
|
1336
|
+
```
|
|
1337
|
+
AC-2: Dashboard shows user name after login
|
|
1338
|
+
Given: user logged in as "jason@example.com"
|
|
1339
|
+
When: navigate to /dashboard
|
|
1340
|
+
Then: heading 'Dashboard' visible, user menu shows "jason@example.com"
|
|
1341
|
+
Verify: browser_snapshot() → assert heading 'Dashboard' + text 'jason@example.com'
|
|
1342
|
+
Not: No dashboard data accuracy in this AC — just presence
|
|
1343
|
+
Evidence-tier: B
|
|
1344
|
+
```
|
|
1345
|
+
|
|
1346
|
+
**BUG-FIX:**
|
|
1347
|
+
```
|
|
1348
|
+
AC-3: EOL date shows 2027-09, not 2025-06
|
|
1349
|
+
Given: OCP Virt product in dashboard
|
|
1350
|
+
When: GET /api/dashboard
|
|
1351
|
+
Then: eolDate field = "2027-09"
|
|
1352
|
+
Verify: curl localhost:${DEV_API}/api/dashboard | jq -e '.products[] | select(.name=="OCP Virt") | .eolDate == "2027-09"'
|
|
1353
|
+
Not: Other product dates not in scope
|
|
1354
|
+
Assumption: If eol-dates.ts is reverted, this date returns to 2025-06
|
|
1355
|
+
Evidence-tier: S (must fail before fix, pass after, fail on revert)
|
|
1356
|
+
```
|
|
1357
|
+
|
|
1358
|
+
**ANTI:**
|
|
1359
|
+
```
|
|
1360
|
+
AC-A1: Legacy signals object no longer used
|
|
1361
|
+
Given: codebase after registry migration
|
|
1362
|
+
When: grep for signals.intelligence, signals.accountPlan, signals.cases
|
|
1363
|
+
Then: zero matches
|
|
1364
|
+
Verify: grep -rn "signals\.\(intelligence\|accountPlan\|cases\)" src/ | wc -l → 0
|
|
1365
|
+
Not: registrySignals usage is expected and correct
|
|
1366
|
+
Evidence-tier: A
|
|
1367
|
+
```
|
|
1368
|
+
|
|
1369
|
+
### Where ACs live
|
|
1370
|
+
|
|
1371
|
+
1. **Canonical source:** GitHub issue body (structured in Success Criteria section)
|
|
1372
|
+
2. **Parsed into:** workflow-state.json `acs[]` array at DISCOVERY phase
|
|
1373
|
+
3. **Filled into:** per-agent brief templates by brief-assembler
|
|
1374
|
+
4. **Verified by:** gate runner at ship gate (each AC has evidence + verdict)
|
|
1375
|
+
|
|
1376
|
+
The brief-assembler parses ACs from workflow-state.json, not from the issue body directly. The DA writes structured ACs during DISCOVERY and stores them in workflow-state.json. This is the canonical machine-readable format that all downstream steps read.
|
|
1377
|
+
|
|
1378
|
+
## Evidence Hierarchy
|
|
1379
|
+
|
|
1380
|
+
Evidence is scored by discriminating power — how well it distinguishes correct from incorrect implementation.
|
|
1381
|
+
|
|
1382
|
+
| Tier | Type | Example | Gate rule |
|
|
1383
|
+
|------|------|---------|-----------|
|
|
1384
|
+
| S | Dual-arm (fails before, passes after + negative control) | Reproduce bug → fix → verify pass → revert → verify fails again | Required for BUG-FIX ACs |
|
|
1385
|
+
| A | Execution-based (test output, API response) | `bun test` output: "24 pass 0 fail" | Required for CODE ACs |
|
|
1386
|
+
| B | Behavioral observation (a11y snapshot assertion) | browser_snapshot shows heading 'Dashboard' with correct values | Required for UI ACs |
|
|
1387
|
+
| C | Static analysis (typecheck, lint, format) | `bunx tsc --noEmit` exits 0 | Supplementary — never sole evidence |
|
|
1388
|
+
| D | Structural grep (string presence) | `grep -c "function validateInput" src/auth.ts` returns 1 | Max 25% of total evidence (existing ratio check) |
|
|
1389
|
+
| F | Self-attestation ("I verified it works") | Agent prose without command output | Always FAIL — zero evidentiary value |
|
|
1390
|
+
|
|
1391
|
+
### Evidence-to-tier mapping
|
|
1392
|
+
|
|
1393
|
+
The gate classifies evidence by its `evidenceMethod.type` field in workflow-state.json:
|
|
1394
|
+
|
|
1395
|
+
| evidenceMethod.type | Tier | Notes |
|
|
1396
|
+
|---------------------|------|-------|
|
|
1397
|
+
| `BUN_TEST` | A | Execution-based test output |
|
|
1398
|
+
| `curl`, `CLI`, `command` | A | Direct command execution |
|
|
1399
|
+
| `browser_snapshot` | B | A11y tree assertion (requires typed assertion, not just snapshot) |
|
|
1400
|
+
| `screenshot` | B only if assertion present | Without assertion → F (self-attestation) |
|
|
1401
|
+
| `tsc`, `lint`, `format` | C | Static analysis — never sole evidence |
|
|
1402
|
+
| `grep` | D | Structural presence check |
|
|
1403
|
+
| `prose`, `manual`, none | F | Self-attestation — always FAIL |
|
|
1404
|
+
|
|
1405
|
+
For BUG-FIX ACs, tier S requires BOTH: `evidenceMethod.type` returns PASS AND `proofOfFix.negativeControl` exists and shows FAIL on revert.
|
|
1406
|
+
|
|
1407
|
+
### Gate enforcement
|
|
1408
|
+
|
|
1409
|
+
- Each AC's evidence is scored against its type's minimum tier (see AC type classification)
|
|
1410
|
+
- OUTCOME ACs: tier A for new features, tier S for bug-fix outcomes
|
|
1411
|
+
- Overall evidence portfolio: tier D evidence capped at 25% (existing `evidence-type-ratio` check)
|
|
1412
|
+
- Tier C only: if ALL evidence for an AC is tier C (static analysis only) → FAIL "static analysis alone is insufficient"
|
|
1413
|
+
- Self-attestation (tier F) in any AC = automatic FAIL regardless of other evidence
|
|
1414
|
+
- Evidence without assertion = tier F (screenshot taken but nothing checked = self-attestation)
|
|
1415
|
+
|
|
1416
|
+
### Test deletion prevention
|
|
1417
|
+
|
|
1418
|
+
- Scope gate captures test count (pass + fail)
|
|
1419
|
+
- Verify gate compares: `post_test_count ≥ pre_test_count`
|
|
1420
|
+
- Test count DECREASE = FAIL with message "Tests removed — verify no gaming"
|
|
1421
|
+
- This catches the documented anti-pattern of agents deleting failing tests
|
|
1422
|
+
|
|
1423
|
+
## RCA Protocol
|
|
1424
|
+
|
|
1425
|
+
Required for all bug-fix issues. Included in all Marcus briefs (bug-fix AND feature). Gate enforcement only checks RCA fields on bug-fix issues — for features, the RCA section is guidance, not enforced.
|
|
1426
|
+
|
|
1427
|
+
### Issue type classification
|
|
1428
|
+
|
|
1429
|
+
The issue type determines RCA enforcement and evidence-tier minimums:
|
|
1430
|
+
|
|
1431
|
+
| Label on GitHub issue | `issueType` in workflow-state | RCA enforced? | Evidence minimum |
|
|
1432
|
+
|----------------------|-------------------------------|--------------|-----------------|
|
|
1433
|
+
| `bug` | `bug-fix` | Yes — gate checks RCA fields | S (dual-arm) |
|
|
1434
|
+
| `enhancement`, `feature` | `feature` | No — RCA included but not enforced | A (execution) |
|
|
1435
|
+
| `refactor` | `refactor` | No | A (execution) |
|
|
1436
|
+
| No label | `unknown` | No — DA should classify at DISCOVERY | A (execution) |
|
|
1437
|
+
|
|
1438
|
+
The DA sets `issueType` in workflow-state.json during DISCOVERY based on the GitHub issue label. If no label, DA classifies from the issue description and adds the label.
|
|
1439
|
+
|
|
1440
|
+
### Four-phase investigation (Hermes protocol, adapted)
|
|
1441
|
+
|
|
1442
|
+
```
|
|
1443
|
+
Phase 1: INVESTIGATE (mandatory before ANY code change)
|
|
1444
|
+
1. Reproduce the bug — exact steps/command that triggers it
|
|
1445
|
+
2. Trace data/control flow from symptom backward to origin
|
|
1446
|
+
3. Read the file(s) you intend to modify — full function, not just error line
|
|
1447
|
+
4. Read 2+ callers of the function you're modifying
|
|
1448
|
+
5. Read tests that cover the code you're about to change
|
|
1449
|
+
|
|
1450
|
+
Phase 2: PATTERN ANALYSIS
|
|
1451
|
+
1. Find WORKING examples of similar code in the codebase
|
|
1452
|
+
2. Diff working vs broken — identify ALL differences
|
|
1453
|
+
3. Map dependency chain from root cause to symptom
|
|
1454
|
+
|
|
1455
|
+
Phase 3: HYPOTHESIS (scientific method)
|
|
1456
|
+
1. State ONE falsifiable hypothesis: "The bug is caused by [X] because [Y]"
|
|
1457
|
+
2. PREDICT: if X is the root cause, what OTHER symptom should exist?
|
|
1458
|
+
3. Test the prediction — wrong prediction = wrong hypothesis → back to Phase 1
|
|
1459
|
+
4. Change ONE variable at a time
|
|
1460
|
+
|
|
1461
|
+
Phase 4: IMPLEMENT (only after Phases 1-3 complete)
|
|
1462
|
+
1. Write failing test that reproduces the bug BEFORE writing fix
|
|
1463
|
+
2. Minimal fix at the ROOT cause, not at the symptom
|
|
1464
|
+
3. Verify fix makes failing test pass + all existing tests still pass
|
|
1465
|
+
4. Guard at boundary where bad data first enters the system
|
|
1466
|
+
5. Audit for same pattern elsewhere: grep for similar code
|
|
1467
|
+
```
|
|
1468
|
+
|
|
1469
|
+
### Gate enforcement (RCA fields in workflow-state)
|
|
1470
|
+
|
|
1471
|
+
```json
|
|
1472
|
+
"rca": {
|
|
1473
|
+
"symptom": "EOL date shows 2025-06 instead of 2027-09",
|
|
1474
|
+
"reproduction": "curl localhost:${DEV_API}/api/dashboard | jq .eolDate",
|
|
1475
|
+
"rootCause": "src/data/eol-dates.ts:47 — hardcoded date not updated after vendor announcement",
|
|
1476
|
+
"prediction": "Other vendor dates in same file may also be stale",
|
|
1477
|
+
"predictionVerified": true,
|
|
1478
|
+
"predictionEvidence": "grep found 3 other stale dates in same file",
|
|
1479
|
+
"blastRadius": ["src/data/eol-dates.ts", "src/api/dashboard.ts", "test/eol.test.ts"]
|
|
1480
|
+
}
|
|
1481
|
+
```
|
|
1482
|
+
|
|
1483
|
+
Gate checks:
|
|
1484
|
+
- `rootCause` populated → not empty string
|
|
1485
|
+
- `prediction` populated → agent stated a falsifiable prediction
|
|
1486
|
+
- `predictionVerified` is true → agent tested the prediction
|
|
1487
|
+
- If ANY field empty on a bug-fix issue → WARN (not FAIL initially — graduate to FAIL after 30 days)
|
|
1488
|
+
|
|
1489
|
+
### Heuristic
|
|
1490
|
+
|
|
1491
|
+
If the "fix" is in the same file as the symptom, verify you haven't just suppressed the symptom. Root cause is usually 1-3 files upstream.
|
|
1492
|
+
|
|
1493
|
+
## Quinn Test Journey Format
|
|
1494
|
+
|
|
1495
|
+
Quinn receives typed test journeys — not prose instructions. Each step has explicit assertions, wait conditions, and flow control. No room for random exploration.
|
|
1496
|
+
|
|
1497
|
+
### Journey schema
|
|
1498
|
+
|
|
1499
|
+
```yaml
|
|
1500
|
+
test_journey:
|
|
1501
|
+
name: "checkout-flow"
|
|
1502
|
+
url: "http://localhost:${DEV_UI_PORT}/products"
|
|
1503
|
+
preconditions:
|
|
1504
|
+
- "User is logged in"
|
|
1505
|
+
- "Cart is empty"
|
|
1506
|
+
|
|
1507
|
+
steps:
|
|
1508
|
+
- id: "step-1"
|
|
1509
|
+
action: "Navigate to product listing page"
|
|
1510
|
+
wait_for: "heading 'Products' visible in snapshot"
|
|
1511
|
+
assert:
|
|
1512
|
+
- type: element_visible
|
|
1513
|
+
target: "heading 'Products'"
|
|
1514
|
+
- type: element_count
|
|
1515
|
+
target: "product cards"
|
|
1516
|
+
expected: ">= 1"
|
|
1517
|
+
on_fail: STOP
|
|
1518
|
+
evidence: snapshot
|
|
1519
|
+
|
|
1520
|
+
- id: "step-2"
|
|
1521
|
+
action: "Click 'Add to Cart' on first product"
|
|
1522
|
+
wait_for: "cart badge updates"
|
|
1523
|
+
assert:
|
|
1524
|
+
- type: text_changed
|
|
1525
|
+
target: "cart badge"
|
|
1526
|
+
from: "0"
|
|
1527
|
+
to: "1"
|
|
1528
|
+
- type: element_visible
|
|
1529
|
+
target: "notification 'Added to cart'"
|
|
1530
|
+
on_fail: STOP
|
|
1531
|
+
evidence: snapshot
|
|
1532
|
+
|
|
1533
|
+
verdict_rules:
|
|
1534
|
+
all_pass: JOURNEY_PASS
|
|
1535
|
+
any_critical_fail: JOURNEY_FAIL
|
|
1536
|
+
non_critical_fail: JOURNEY_WARN
|
|
1537
|
+
```
|
|
1538
|
+
|
|
1539
|
+
### Assertion types
|
|
1540
|
+
|
|
1541
|
+
| Type | What it checks | Example |
|
|
1542
|
+
|------|---------------|---------|
|
|
1543
|
+
| element_visible | Element exists in a11y tree | heading 'Dashboard' |
|
|
1544
|
+
| element_absent | Element should NOT be in a11y tree | error banner not present |
|
|
1545
|
+
| text_match | Element text equals expected | user menu shows 'jason@...' |
|
|
1546
|
+
| value_check | Element value satisfies condition | total price > $0.00 |
|
|
1547
|
+
| state_check | Element state (checked/disabled/expanded) | submit button disabled |
|
|
1548
|
+
| text_changed | Element text changed from X to Y | cart badge 0 → 1 |
|
|
1549
|
+
| element_count | Count of matching elements | product cards >= 1 |
|
|
1550
|
+
| no_errors | No error elements in snapshot + no new console errors | Clean state verification |
|
|
1551
|
+
| visual_check | Screenshot comparison (only when a11y can't verify) | layout, color, overlap |
|
|
1552
|
+
|
|
1553
|
+
`element_absent` catches cases where old UI should be removed after a fix. `no_errors` is weak alone but valuable as a baseline check at step start — catches error states before testing the feature.
|
|
1554
|
+
|
|
1555
|
+
### Journey constraints
|
|
1556
|
+
|
|
1557
|
+
- **Max 5-8 steps per journey.** Context degradation: instruction compliance decays from 73% at turn 5 to 33% by turn 16. Brief-assembler splits longer flows into multiple journeys with explicit entry conditions.
|
|
1558
|
+
- **Max steps enforced at generation time** — brief-assembler rejects journeys over 8 steps and splits automatically.
|
|
1559
|
+
- Each split journey has `preconditions` that reference the prior journey's end state.
|
|
1560
|
+
|
|
1561
|
+
### Perception hierarchy
|
|
1562
|
+
|
|
1563
|
+
1. **Primary: browser_snapshot() (a11y tree)** — 2-5KB, deterministic, 10-100x cheaper than screenshots, no vision model needed
|
|
1564
|
+
2. **Fallback: browser_take_screenshot()** — ONLY for visual checks (layout, color, overlap, canvas/SVG) or on FAIL
|
|
1565
|
+
3. **Never: raw DOM/HTML** — noisy, expensive, brittle
|
|
1566
|
+
|
|
1567
|
+
### Decision tree (embedded in Quinn's prompt)
|
|
1568
|
+
|
|
1569
|
+
```
|
|
1570
|
+
BEFORE EACH STEP:
|
|
1571
|
+
Am I on the right URL? → NO → navigate, retry once → still NO → FAIL
|
|
1572
|
+
Can I find target element in snapshot? → NO → wait 3s, re-snapshot → still NO → FAIL
|
|
1573
|
+
Is page in expected state? → NO → check for error banners → report actual state → FAIL
|
|
1574
|
+
|
|
1575
|
+
AFTER EACH ACTION:
|
|
1576
|
+
Did snapshot change? → NO → action may not have fired → retry once → still NO → FAIL
|
|
1577
|
+
Does new state match expected? → YES → PASS with evidence → NO → FAIL with diff
|
|
1578
|
+
Am I still on expected URL? → NO → unexpected navigation → WARN
|
|
1579
|
+
```
|
|
1580
|
+
|
|
1581
|
+
### Circuit breaker
|
|
1582
|
+
|
|
1583
|
+
- 3 consecutive FAIL steps → ABORT journey with partial results
|
|
1584
|
+
- Total elapsed > timeout (300s) → ABORT
|
|
1585
|
+
- Same step retried 2x → FAIL definitively, move on
|
|
1586
|
+
|
|
1587
|
+
### Evidence capture rules
|
|
1588
|
+
|
|
1589
|
+
| Event | Snapshot (a11y) | Screenshot | Console log |
|
|
1590
|
+
|-------|----------------|------------|-------------|
|
|
1591
|
+
| Step start (BEFORE) | ALWAYS | Only if visual step | Check for pre-existing errors |
|
|
1592
|
+
| After action (AFTER) | ALWAYS | Only on FAIL or visual step | Check for new errors |
|
|
1593
|
+
| Assertion FAIL | Attach diff | ALWAYS | ALWAYS |
|
|
1594
|
+
| Assertion PASS | Store ref only | Skip | Skip |
|
|
1595
|
+
| Journey end | Final state | ALWAYS (final proof) | Full log |
|
|
1596
|
+
|
|
1597
|
+
### Journey generation and storage
|
|
1598
|
+
|
|
1599
|
+
The brief-assembler generates Quinn journeys from ACs with UI type:
|
|
1600
|
+
- AC's `evidenceMethod.type` = browser_snapshot or screenshot → generates a journey step
|
|
1601
|
+
- AC's threshold becomes the assertion expected value
|
|
1602
|
+
- AC's contextFiles become the navigation targets
|
|
1603
|
+
- `${DEV_UI_PORT}` from rungate.json fills URLs
|
|
1604
|
+
|
|
1605
|
+
**Where journeys live:**
|
|
1606
|
+
1. **Generated at:** ship time by brief-assembler
|
|
1607
|
+
2. **Stored in:** `~/.rungate/{slug}/quinn-journey.yaml` (per-issue)
|
|
1608
|
+
3. **Read by:** Quinn's prompt template references the journey file path
|
|
1609
|
+
4. **Evidence stored in:** `~/.rungate/{slug}/evidence/journey-{step-id}.json` (per-step)
|
|
1610
|
+
|
|
1611
|
+
Journey files are ephemeral — they live in the slug directory and follow slug lifecycle (archived after 24h, deleted after 30 days).
|
|
1612
|
+
|
|
1613
|
+
## Blast Radius Analysis
|
|
1614
|
+
|
|
1615
|
+
Required before implementation. The brief-assembler generates a blast radius section from CODE-MAP data.
|
|
1616
|
+
|
|
1617
|
+
### Pre-implementation checklist (in Marcus brief)
|
|
1618
|
+
|
|
1619
|
+
```
|
|
1620
|
+
BEFORE writing any code change:
|
|
1621
|
+
1. DIRECT DEPENDENTS: What imports/calls the code I'm changing?
|
|
1622
|
+
→ grep -r "import.*{module}" src/ && grep -r "{function}" src/
|
|
1623
|
+
2. TRANSITIVE: What depends on those dependents? (2 levels deep)
|
|
1624
|
+
3. TEST COVERAGE: Do tests exist for each dependent?
|
|
1625
|
+
→ NO → write tests for dependents BEFORE making the change
|
|
1626
|
+
4. INTERFACE CONTRACT: Changing a function signature or return type?
|
|
1627
|
+
→ YES → update every caller in the same PR
|
|
1628
|
+
5. CROSS-BOUNDARY: Change crosses module/package boundary?
|
|
1629
|
+
→ YES → flag for review, do not proceed autonomously
|
|
1630
|
+
```
|
|
1631
|
+
|
|
1632
|
+
### Gate enforcement
|
|
1633
|
+
|
|
1634
|
+
```json
|
|
1635
|
+
"blastRadius": {
|
|
1636
|
+
"filesRead": ["src/api/auth.ts", "src/middleware/jwt.ts", "test/auth.test.ts"],
|
|
1637
|
+
"filesChanged": ["src/api/auth.ts"],
|
|
1638
|
+
"readToWriteRatio": 3.0,
|
|
1639
|
+
"unlisted": []
|
|
1640
|
+
}
|
|
1641
|
+
```
|
|
1642
|
+
|
|
1643
|
+
- `filesRead.length ≥ filesChanged.length` — read more than you write
|
|
1644
|
+
- `readToWriteRatio ≥ 3.0` — computed by gate from filesRead.length / filesChanged.length (file count, not token count — simpler and available without instrumentation)
|
|
1645
|
+
- `unlisted` not empty → WARN "Files changed outside brief: {list}" with explanation required
|
|
1646
|
+
- `filesRead` empty → WARN "No files read before implementation — likely symptom-fixing"
|
|
1647
|
+
|
|
1648
|
+
## Prevention-Oriented Fix Protocol
|
|
1649
|
+
|
|
1650
|
+
Beyond fixing the current bug — every fix should make the same class of bug harder to introduce. This section is included in ALL Marcus briefs (bug-fix and feature). Guard/type-narrow/pattern-audit are universal. Gate enforcement of the `prevention` JSON block only applies to bug-fix issues.
|
|
1651
|
+
|
|
1652
|
+
### Post-fix checklist (required before marking AC complete)
|
|
1653
|
+
|
|
1654
|
+
```
|
|
1655
|
+
AFTER fixing any bug, before marking complete:
|
|
1656
|
+
|
|
1657
|
+
1. GUARD AT BOUNDARY: Add input validation/assertion at the point where
|
|
1658
|
+
bad data FIRST enters the system — not where it crashes
|
|
1659
|
+
→ typeof checks for unexpected types
|
|
1660
|
+
→ Range checks for numeric boundaries
|
|
1661
|
+
→ Null guards where null propagation caused the bug
|
|
1662
|
+
→ Schema validation (Zod) at API/module boundaries
|
|
1663
|
+
|
|
1664
|
+
2. TYPE NARROWING: Can the type system prevent this class of bug?
|
|
1665
|
+
→ Discriminated unions over string literals
|
|
1666
|
+
→ Branded types for IDs that shouldn't be interchangeable
|
|
1667
|
+
→ NonNullable<T> where null caused the failure
|
|
1668
|
+
→ Exhaustive switch/case (default branch = compile error)
|
|
1669
|
+
|
|
1670
|
+
3. INVARIANT ASSERTION: Add runtime assertion that would catch this
|
|
1671
|
+
bug class EVEN IF the specific fix regresses
|
|
1672
|
+
→ assert(condition, "Invariant: [what must be true and why]")
|
|
1673
|
+
→ Place at function entry, not deep in the logic
|
|
1674
|
+
→ The assertion should fire on ANY variant of this bug, not just this instance
|
|
1675
|
+
|
|
1676
|
+
4. PATTERN AUDIT: grep for the same pattern elsewhere in codebase
|
|
1677
|
+
→ grep -rn "[pattern that caused bug]" src/
|
|
1678
|
+
→ If the same mistake exists elsewhere, fix ALL instances in this PR
|
|
1679
|
+
→ Report count: "Found N instances, fixed N"
|
|
1680
|
+
|
|
1681
|
+
5. BOUNDARY TEST: Write a test that sends the EXACT bad input
|
|
1682
|
+
that caused this bug
|
|
1683
|
+
→ Test must FAIL without your fix and PASS with it
|
|
1684
|
+
→ Test name should describe the bug class, not the ticket number
|
|
1685
|
+
→ Good: "rejects_negative_quantity_in_cart"
|
|
1686
|
+
→ Bad: "fix_issue_1234"
|
|
1687
|
+
|
|
1688
|
+
6. BUG CLASS DOCUMENTATION: Name the class in the PR/commit
|
|
1689
|
+
→ Null propagation, off-by-one, race condition, type coercion,
|
|
1690
|
+
stale cache, missing await, unhandled edge case, etc.
|
|
1691
|
+
→ Enables pattern-matching in future reviews
|
|
1692
|
+
→ Gate records bug class in workflow-state for trend analysis
|
|
1693
|
+
```
|
|
1694
|
+
|
|
1695
|
+
### Gate enforcement
|
|
1696
|
+
|
|
1697
|
+
```json
|
|
1698
|
+
"prevention": {
|
|
1699
|
+
"guardAdded": true,
|
|
1700
|
+
"guardLocation": "src/api/auth.ts:23 — Zod schema validates login payload",
|
|
1701
|
+
"patternAuditCount": 3,
|
|
1702
|
+
"patternAuditFixed": 3,
|
|
1703
|
+
"bugClass": "null-propagation",
|
|
1704
|
+
"boundaryTestName": "test/auth.test.ts:rejects_missing_email_field"
|
|
1705
|
+
}
|
|
1706
|
+
```
|
|
1707
|
+
|
|
1708
|
+
- `guardAdded` = false → WARN "No boundary guard added — same bug class can recur"
|
|
1709
|
+
- `patternAuditCount` > `patternAuditFixed` → **FAIL** "N unfixed instances of same pattern remain — fix all or explain why not"
|
|
1710
|
+
- `bugClass` empty → WARN "Bug class not documented — can't track trends"
|
|
1711
|
+
- All fields populated → prevention score logged for trend analysis
|
|
1712
|
+
|
|
1713
|
+
`patternAuditCount > patternAuditFixed` is FAIL not WARN because: you FOUND the same bug elsewhere and chose not to fix it. That's not a judgment call — it's a known gap being shipped. If there's a legitimate reason to skip (e.g., different module owner), add to `scopeOut` with explanation.
|
|
1714
|
+
|
|
1715
|
+
### Why this matters
|
|
1716
|
+
|
|
1717
|
+
AI-generated code has a 44% security flaw rate (Veracode 2026). Without prevention-oriented fixes, each bug is fixed in isolation and the same class recurs. The pattern audit step alone (grep for siblings) caught 3+ additional instances in prior sessions that would have become separate bug reports.
|
|
1718
|
+
|
|
1719
|
+
## Proof-of-Fix Protocol
|
|
1720
|
+
|
|
1721
|
+
Extends existing /prove skill with negative control step. Required for S-tier (dual-arm) evidence.
|
|
1722
|
+
|
|
1723
|
+
### When each step applies
|
|
1724
|
+
|
|
1725
|
+
| Step | Bug-fix issues | Feature issues | Rationale |
|
|
1726
|
+
|------|---------------|----------------|-----------|
|
|
1727
|
+
| 1. REPRODUCE | Required | Skip (no "before") | Can't reproduce what didn't exist |
|
|
1728
|
+
| 2. APPLY | Required | Required | Both need the fix/feature applied |
|
|
1729
|
+
| 3. VERIFY-FIX | Required | Required | Both need verification |
|
|
1730
|
+
| 4. VERIFY-REGRESSION | Required | Required | Both can cause regressions |
|
|
1731
|
+
| 5. NEGATIVE CONTROL | Required | Skip | No "before" state to revert to |
|
|
1732
|
+
|
|
1733
|
+
### Protocol
|
|
1734
|
+
|
|
1735
|
+
```
|
|
1736
|
+
1. REPRODUCE: Run reproduction steps on buggy code → capture FAIL
|
|
1737
|
+
Skip when: issueType != "bug-fix"
|
|
1738
|
+
2. APPLY: Apply candidate fix
|
|
1739
|
+
3. VERIFY-FIX: Run same reproduction steps → capture PASS
|
|
1740
|
+
4. VERIFY-REGRESSION: Run full test suite → no new failures
|
|
1741
|
+
5. NEGATIVE CONTROL: Revert fix → confirm bug returns (FAIL again)
|
|
1742
|
+
Skip when: issueType != "bug-fix"
|
|
1743
|
+
```
|
|
1744
|
+
|
|
1745
|
+
### Evidence in workflow-state.json
|
|
1746
|
+
|
|
1747
|
+
```json
|
|
1748
|
+
"proofOfFix": {
|
|
1749
|
+
"issueType": "bug-fix",
|
|
1750
|
+
"reproduction_before": "curl output showing eolDate=2025-06",
|
|
1751
|
+
"reproduction_after": "curl output showing eolDate=2027-09",
|
|
1752
|
+
"regression_suite": "bun test: 47 pass, 0 fail",
|
|
1753
|
+
"negative_control": "reverted → curl shows eolDate=2025-06 (bug returned)",
|
|
1754
|
+
"commit_sha": "abc1234",
|
|
1755
|
+
"verdict": "PROVEN"
|
|
1756
|
+
}
|
|
1757
|
+
```
|
|
1758
|
+
|
|
1759
|
+
This maps directly to `prove-evidence.json` written by the /prove skill. The gate reads `proofOfFix` from workflow-state and cross-validates against prove-evidence.json.
|
|
1760
|
+
|
|
1761
|
+
For feature issues, `proofOfFix` contains only steps 2-4 (no reproduction_before, no negative_control) and `verdict` is based on VERIFY-FIX + VERIFY-REGRESSION only.
|
|
1762
|
+
|
|
1763
|
+
Step 5 (negative control) is what distinguishes real proof from coincidence. If the bug doesn't return when the fix is reverted, either the reproduction is flawed or the fix didn't address the root cause.
|
|
1764
|
+
|
|
1765
|
+
## Template Architecture
|
|
1766
|
+
|
|
1767
|
+
Templates are three-layered. Each layer adds specificity. The brief-assembler merges all three.
|
|
1768
|
+
|
|
1769
|
+
### Layer 1: Harness-generic (ships with rungate)
|
|
1770
|
+
|
|
1771
|
+
Universal templates that work for any project:
|
|
1772
|
+
- RCA protocol (4 phases)
|
|
1773
|
+
- AC format (Given/When/Then + oracle)
|
|
1774
|
+
- Evidence hierarchy (S through F)
|
|
1775
|
+
- Quinn decision tree
|
|
1776
|
+
- Blast radius checklist
|
|
1777
|
+
- Regression prevention checklist
|
|
1778
|
+
- Read-before-write protocol
|
|
1779
|
+
- Prevention-oriented fix template (guard, type narrow, pattern audit)
|
|
1780
|
+
- Coding principles (deep modules, boundary validation, Zod at boundaries, assertNever, branded types)
|
|
1781
|
+
- Testing strategy (PBT, contract tests, tautological trap, boundary testing, test evidence per AC type)
|
|
1782
|
+
- Escalation decision tree (research tools, when to stop iterating)
|
|
1783
|
+
|
|
1784
|
+
These never reference specific files, ports, routes, or technologies.
|
|
1785
|
+
|
|
1786
|
+
### Layer 1 knowledge problem: training data goes stale
|
|
1787
|
+
|
|
1788
|
+
Saying "use deep modules" in a brief doesn't teach the agent what deep modules ARE. Models are trained on data up to a cutoff — principles published after that cutoff aren't in their weights. The harness solves this in three ways:
|
|
1789
|
+
|
|
1790
|
+
1. **Inline the principle, not just the name.** The `coding-principles.md` template doesn't say "use deep modules." It says:
|
|
1791
|
+
```
|
|
1792
|
+
DEEP MODULES: A module should do a lot of work behind a simple interface.
|
|
1793
|
+
- A function with 3 parameters that handles 15 edge cases internally = deep
|
|
1794
|
+
- 15 small functions each handling 1 case, requiring the caller to orchestrate = shallow
|
|
1795
|
+
- When adding functionality: extend an existing deep module rather than adding a new shallow one
|
|
1796
|
+
- Test: can a caller use this module without understanding its internals? If no, the interface is too complex.
|
|
1797
|
+
```
|
|
1798
|
+
The principle is EXPLAINED, not referenced. The agent has the content, not a pointer.
|
|
1799
|
+
|
|
1800
|
+
2. **Context7 MCP for current library docs.** When an agent needs framework-specific guidance (Hono routing patterns, Zod schema design, Bun test runners), Context7 fetches CURRENT documentation — not the model's training data. This is why Context7 is declared in the research tools section: it's the escape hatch from stale training data.
|
|
1801
|
+
|
|
1802
|
+
3. **Harness update cycle.** When new research emerges (like the findings from this session's 8 researchers), it gets written into Layer 1 templates. Every project gets the updated templates on next `bun install`. The harness is the distribution mechanism for current best practices.
|
|
1803
|
+
|
|
1804
|
+
**The flow:**
|
|
1805
|
+
```
|
|
1806
|
+
New research/principle discovered
|
|
1807
|
+
→ Written into harness prompts/ template (Layer 1)
|
|
1808
|
+
→ Harness version bumped
|
|
1809
|
+
→ Projects run bun install → get new templates
|
|
1810
|
+
→ Next ship cycle → brief-assembler generates briefs with updated content
|
|
1811
|
+
→ Agent reads the actual principle, not just a name
|
|
1812
|
+
```
|
|
1813
|
+
|
|
1814
|
+
This is why Layer 1 templates contain the full explanation, not links. An agent can't click a link to "A Philosophy of Software Design." But it CAN read a 10-line summary of the principle with specific do/don't examples embedded in its brief.
|
|
1815
|
+
|
|
1816
|
+
### Layer 2: Project-config (from rungate.json + CODE-MAP)
|
|
1817
|
+
|
|
1818
|
+
Project-specific values injected into templates:
|
|
1819
|
+
- Test command (`bun test` vs `pytest` vs `go test`)
|
|
1820
|
+
- Port numbers (from rungate.json environments)
|
|
1821
|
+
- File paths (from CODE-MAP)
|
|
1822
|
+
- Framework details (Hono, React, etc.)
|
|
1823
|
+
- Page URLs (from rungate.json pages)
|
|
1824
|
+
- Consumer modules (from rungate.json consumers)
|
|
1825
|
+
|
|
1826
|
+
### Layer 3: Issue-specific (generated per brief by brief-assembler)
|
|
1827
|
+
|
|
1828
|
+
Per-issue values:
|
|
1829
|
+
- ACs for this specific issue
|
|
1830
|
+
- Files this issue touches (from brief)
|
|
1831
|
+
- Governing spec for this issue
|
|
1832
|
+
- Quinn journey steps for this UI change
|
|
1833
|
+
- RCA fields pre-populated from issue description
|
|
1834
|
+
|
|
1835
|
+
### Merge order
|
|
1836
|
+
|
|
1837
|
+
```
|
|
1838
|
+
harness-generic (Layer 1)
|
|
1839
|
+
+ project-config (Layer 2) → variables filled
|
|
1840
|
+
+ issue-specific (Layer 3) → ACs, files, journeys added
|
|
1841
|
+
= Complete agent brief
|
|
1842
|
+
```
|
|
1843
|
+
|
|
1844
|
+
### Per-agent template sections
|
|
1845
|
+
|
|
1846
|
+
| Agent | Layer 1 sections | Layer 2 additions | Layer 3 additions |
|
|
1847
|
+
|-------|-----------------|-------------------|-------------------|
|
|
1848
|
+
| Marcus | RCA protocol, AC format, blast radius, regression prevention, read-before-write, coding principles, testing strategy, prevention-oriented fix | test command, source dirs, consumers | ACs, files, governing spec |
|
|
1849
|
+
| Quinn | Decision tree, journey schema, evidence capture, circuit breaker | ports, pages, viewport | Typed journey steps from UI ACs |
|
|
1850
|
+
| Rook | Security checklist, OWASP patterns, secret detection | security baseline path | Changed files to scan |
|
|
1851
|
+
| Serena | Architecture review, cross-boundary analysis, coding principles | ADR location, module map | Structural changes proposed |
|
|
1852
|
+
| Aditi | Component spec format, a11y requirements | component paths, design system | UI components affected |
|
|
1853
|
+
|
|
1854
|
+
### Improvement flow
|
|
1855
|
+
|
|
1856
|
+
Template improvement in Layer 1 → every project gets it on next re-scaffold. No per-project configuration needed for fundamental best practices.
|
|
1857
|
+
|
|
1858
|
+
### Brief assembly order (SC-140, SC-145)
|
|
1859
|
+
|
|
1860
|
+
Research: "Lost in the Middle" (Stanford, TACL 2024) shows 30-50% accuracy drop for content in the middle of context vs start/end. Instruction Stacking Collapse (arXiv:2608.02639) shows compliance drops from 96.4% to 57.7% at 16 stacked rules.
|
|
1861
|
+
|
|
1862
|
+
The brief-assembler MUST order sections to put critical instructions at the positions models attend to most:
|
|
1863
|
+
|
|
1864
|
+
```
|
|
1865
|
+
1. IDENTITY (role, constraints, output format) ← START (highest attention)
|
|
1866
|
+
2. CORE PRINCIPLES (universal 6-line block) ← near start
|
|
1867
|
+
3. METHODOLOGY (Layer 1 templates, 10 rules max) ← middle (lowest attention — OK because gates enforce)
|
|
1868
|
+
4. PROJECT CONTEXT (Layer 2, from rungate.json) ← middle
|
|
1869
|
+
5. ACs + VERIFICATION COMMANDS (Layer 3) ← END (high attention, recency bias)
|
|
1870
|
+
```
|
|
1871
|
+
|
|
1872
|
+
This exploits the U-shaped attention curve: models attend most to start and end, least to middle. Methodology in the middle is acceptable because gates mechanically enforce what the model might miss. ACs at the end get recency bias — the thing the agent works on most is what it sees last.
|
|
1873
|
+
|
|
1874
|
+
### Template format requirements (SC-141, SC-142)
|
|
1875
|
+
|
|
1876
|
+
Research on instruction format effectiveness:
|
|
1877
|
+
|
|
1878
|
+
1. **Positive framing > negative.** "Use `Result<T, E>` for error handling" beats "Don't throw exceptions." Cap negative instructions at 10 per brief. Past that, models confuse what NOT to do with what to do.
|
|
1879
|
+
|
|
1880
|
+
2. **Examples > descriptions.** One code snippet showing the pattern beats three paragraphs describing it. Every Layer 1 template must include at least one concrete code example per principle.
|
|
1881
|
+
|
|
1882
|
+
3. **One concept per bullet.** Multi-action instructions get merged or reinterpreted. Break into single-action items.
|
|
1883
|
+
|
|
1884
|
+
4. **Calm tone.** "CRITICAL!", "YOU MUST", "NEVER EVER" overtriggers newer models and produces WORSE results than direct instructions (Anthropic context engineering guide). Emphasize one line, not many.
|
|
1885
|
+
|
|
1886
|
+
5. **Three-tier rule authority (SC-146).** Every rule in an agent brief is categorized by action authority. This tells the agent what it can do autonomously vs what needs approval vs what is forbidden. Research: "never do" sections are the most frequently skipped and cause the most production failures — making them explicit and separate improves compliance.
|
|
1887
|
+
|
|
1888
|
+
```
|
|
1889
|
+
## Always Do (autonomous — no approval needed)
|
|
1890
|
+
- Run `bun test` after every change
|
|
1891
|
+
- Read AGENTS.md before starting work
|
|
1892
|
+
- Verify before asserting
|
|
1893
|
+
|
|
1894
|
+
## Ask First (needs DA approval before proceeding)
|
|
1895
|
+
- Modifying files outside the brief's listed files
|
|
1896
|
+
- Adding new dependencies
|
|
1897
|
+
- Changing public interfaces
|
|
1898
|
+
|
|
1899
|
+
## Never Do (hard stops — gate FAIL if violated)
|
|
1900
|
+
- Self-attest evidence (tier F)
|
|
1901
|
+
- Skip ACs without rationale
|
|
1902
|
+
- Commit secrets or credentials
|
|
1903
|
+
```
|
|
1904
|
+
|
|
1905
|
+
### Activation modes (SC-143)
|
|
1906
|
+
|
|
1907
|
+
Industry convergence on four activation modes (Cursor, Devin, Cline all implement these). Each template declares its mode:
|
|
1908
|
+
|
|
1909
|
+
| Mode | When loaded | Example |
|
|
1910
|
+
|------|------------|---------|
|
|
1911
|
+
| always-on | Every request, unconditionally | Core Principles block, AGENTS.md |
|
|
1912
|
+
| file-pattern | When matching files in context | Security checklist when .env files touched |
|
|
1913
|
+
| agent-decided | Agent reads description, decides relevance | RCA protocol (only for bug-fix issues) |
|
|
1914
|
+
| manual | Only on explicit invocation | Full spec reference |
|
|
1915
|
+
|
|
1916
|
+
**Budget rule:** Always-on content under 200 words total. Every word costs on every request. File-pattern and agent-decided have no hard cap but 10 essential rules per template.
|
|
1917
|
+
|
|
1918
|
+
### Rule count caps (SC-138)
|
|
1919
|
+
|
|
1920
|
+
Hard data: models reliably satisfy ~3 concurrent constraints (Instruction Complexity Cliff research). At 16 stacked instructions, compliance drops 50%+ (arXiv:2608.02639). ETH Zurich: 10 essential rules outperform 200 generic ones.
|
|
1921
|
+
|
|
1922
|
+
**Per-template caps:**
|
|
1923
|
+
- Universal Core Principles: 6 rules (always-on, every agent)
|
|
1924
|
+
- Per-role rules in agent definition: 4-6 rules
|
|
1925
|
+
- Layer 1 methodology template: 10 rules max
|
|
1926
|
+
- Total rules an agent sees at brief time: ≤25
|
|
1927
|
+
|
|
1928
|
+
The 25-rule ceiling is a design target, not a gate FAIL. Gate WARNS if brief exceeds 25 distinct rules. The intent: force prioritization. If you can't fit a principle in 25 rules, the principle isn't essential — move it to a gate check instead.
|
|
1929
|
+
|
|
1930
|
+
### Rule lifecycle (SC-144)
|
|
1931
|
+
|
|
1932
|
+
Rules go stale. Production systems report compounding stale-rule problems: old patterns in code outnumber new ones, agents imitate what they find, deprecated patterns survive test suites, each copy creates another call site for future agents.
|
|
1933
|
+
|
|
1934
|
+
**Rule hygiene protocol:**
|
|
1935
|
+
1. Every rule in AGENTS.md Hard Constraints gets a `<!-- since: YYYY-MM-DD -->` comment
|
|
1936
|
+
2. Re-scaffold flags rules older than 90 days without revalidation: WARN "Rule '{name}' last validated {N} days ago — review or reconfirm"
|
|
1937
|
+
3. "Second occurrence" principle: only add a rule after the same mistake happens twice. First occurrence = noise. Twice = pattern.
|
|
1938
|
+
4. Quarterly audit (manual, not automated): was it used recently? Does it solve a real problem? Can it be simplified? Does it conflict with other rules?
|
|
1939
|
+
|
|
1940
|
+
### Mid-session compliance (15 tool-call cliff)
|
|
1941
|
+
|
|
1942
|
+
Research: past 15 tool calls, system prompt constraints lose influence from attention dilution. Ship cycles easily exceed 15 tool calls.
|
|
1943
|
+
|
|
1944
|
+
**Mitigation strategy (mechanical, not behavioral):**
|
|
1945
|
+
1. Gates run AFTER the work, not during — they don't depend on the agent remembering rules mid-session
|
|
1946
|
+
2. Gate checks are deterministic code, not LLM judgment — compliance is verified, not trusted
|
|
1947
|
+
3. Brief structure puts ACs at the END (recency bias) — the verification commands are what the agent sees last
|
|
1948
|
+
4. Circuit breaker at iteration 3 forces a reset — fresh context, rules re-read
|
|
1949
|
+
|
|
1950
|
+
This is why the entire harness architecture is gate-first: we EXPECT agents to drift past 15 tool calls. The gates catch drift mechanically. The brief optimizations (ordering, caps, positive framing) reduce drift but don't eliminate it.
|
|
1951
|
+
|
|
1952
|
+
## Spec Drift Enforcement
|
|
1953
|
+
|
|
1954
|
+
Spec content hashes are tracked. When a spec changes, tests written against it are stale.
|
|
1955
|
+
|
|
1956
|
+
### Mechanism
|
|
1957
|
+
|
|
1958
|
+
1. Each testable spec's content is hashed (SHA-256 of SC section)
|
|
1959
|
+
2. Hash stored in `test/spec-compliance-auto.test.ts` as comment
|
|
1960
|
+
3. At gate time (scope + verify), current spec hash compared to stored hash
|
|
1961
|
+
4. **Mismatch = FAIL** — "spec changed since tests written, update tests"
|
|
1962
|
+
5. `bunx rungate sync-tests .` regenerates tests from current spec → hash updates
|
|
1963
|
+
|
|
1964
|
+
### Why FAIL not WARN
|
|
1965
|
+
|
|
1966
|
+
WARN is behavioral — the agent sees it, notes it, proceeds anyway. FAIL is mechanical — gate blocks, work can't ship until tests match spec. This is D-18.
|
|
1967
|
+
|
|
1968
|
+
## Parallel Work Isolation
|
|
1969
|
+
|
|
1970
|
+
When multiple agents or sessions work on the same repo concurrently, four conflict types emerge. The harness handles each mechanically.
|
|
1971
|
+
|
|
1972
|
+
### Conflict types and solutions
|
|
1973
|
+
|
|
1974
|
+
| Conflict type | What breaks | Solution | Enforcement |
|
|
1975
|
+
|---|---|---|---|
|
|
1976
|
+
| Code overlap | Two agents modify same file | Git worktrees (file isolation) + file-set overlap detection | SC-46, SC-69 |
|
|
1977
|
+
| Container sharing | Two agents rebuild/test simultaneously | Container lock in Makefile | SC-45, SC-47 |
|
|
1978
|
+
| Port collision | Two dev servers on same port | Port/namespace allocation per worktree | SC-70 |
|
|
1979
|
+
| Stale accumulation | Old worktrees, branches, slugs pile up | Slug lifecycle + worktree cleanup | SC-42, SC-43 |
|
|
1980
|
+
|
|
1981
|
+
### Git worktree integration
|
|
1982
|
+
|
|
1983
|
+
Each parallel issue gets its own worktree. The harness manages the lifecycle:
|
|
1984
|
+
|
|
1985
|
+
```
|
|
1986
|
+
CREATION (at ship SCOPE):
|
|
1987
|
+
1. Branch created: git branch issue-{N}-{slug} main
|
|
1988
|
+
2. Worktree created: git worktree add .worktrees/issue-{N} issue-{N}-{slug}
|
|
1989
|
+
3. Slug directory: ~/.rungate/{slug}/ (shared — not in worktree)
|
|
1990
|
+
4. Agent spawned with cwd = worktree path
|
|
1991
|
+
|
|
1992
|
+
BRANCH NAMING:
|
|
1993
|
+
Pattern: issue-{number}-{slug}
|
|
1994
|
+
Examples: issue-1452-bootstrap-fix, issue-1453-quinn-journey
|
|
1995
|
+
Deterministic from issue number — no collisions
|
|
1996
|
+
|
|
1997
|
+
DURING WORK:
|
|
1998
|
+
- Agent works entirely within its worktree
|
|
1999
|
+
- bun test runs against worktree's copy (isolated)
|
|
2000
|
+
- All file paths in workflow-state.json are relative to worktree root
|
|
2001
|
+
- Container lock checked before any make rebuild / make prove-up
|
|
2002
|
+
|
|
2003
|
+
MERGE (after ship + prove PASS):
|
|
2004
|
+
1. PR created from worktree branch → main
|
|
2005
|
+
2. CI runs on PR branch
|
|
2006
|
+
3. Human review (or auto-merge for LIGHT tier if configured)
|
|
2007
|
+
4. Squash merge into main
|
|
2008
|
+
5. Rebase subsequent worktree branches against updated main:
|
|
2009
|
+
git -C .worktrees/issue-{next} rebase main
|
|
2010
|
+
|
|
2011
|
+
CLEANUP (after merge):
|
|
2012
|
+
1. Worktree removed: git worktree remove .worktrees/issue-{N}
|
|
2013
|
+
2. Branch deleted: git branch -d issue-{N}-{slug}
|
|
2014
|
+
3. Slug archived (existing lifecycle — SC-42)
|
|
2015
|
+
4. If no changes made, worktree auto-cleaned (Claude Code native behavior)
|
|
2016
|
+
```
|
|
2017
|
+
|
|
2018
|
+
### File-set overlap detection (SC-69)
|
|
2019
|
+
|
|
2020
|
+
Before assigning concurrent issues to parallel agents:
|
|
2021
|
+
|
|
2022
|
+
```
|
|
2023
|
+
1. Each issue's brief lists expected files to modify
|
|
2024
|
+
2. Compare file sets across all active worktrees
|
|
2025
|
+
3. DISJOINT → safe to parallelize
|
|
2026
|
+
4. OVERLAP on shared files (utils, config, types) → WARN with list
|
|
2027
|
+
5. OVERLAP on same module → BLOCK — serialize these issues
|
|
2028
|
+
|
|
2029
|
+
Detection runs at SCOPE gate when a new ship cycle starts while other cycles are active.
|
|
2030
|
+
```
|
|
2031
|
+
|
|
2032
|
+
### Port/namespace isolation (SC-70)
|
|
2033
|
+
|
|
2034
|
+
Git worktrees isolate files but NOT runtime resources:
|
|
2035
|
+
|
|
2036
|
+
```
|
|
2037
|
+
PORT ALLOCATION:
|
|
2038
|
+
- Each worktree gets a port offset from base: base + (issue_number % 100)
|
|
2039
|
+
- Dev server: DEV_UI_PORT = 5173 + offset
|
|
2040
|
+
- API server: DEV_API_PORT = 7778 + offset
|
|
2041
|
+
- Stored in worktree-local .env or passed via env var
|
|
2042
|
+
- rungate.json in worktree updated with allocated ports
|
|
2043
|
+
|
|
2044
|
+
CONTAINER ISOLATION:
|
|
2045
|
+
- Container lock prevents concurrent rebuilds (SC-45)
|
|
2046
|
+
- Queued issues wait for lock release (SC-47)
|
|
2047
|
+
- Alternative: separate container per worktree (heavier but fully isolated)
|
|
2048
|
+
|
|
2049
|
+
TEST ISOLATION:
|
|
2050
|
+
- Each worktree runs its own bun test (file-isolated by worktree)
|
|
2051
|
+
- Database tests: use test-scoped DB name or transaction rollback
|
|
2052
|
+
- Cache: each worktree gets its own cache namespace
|
|
2053
|
+
```
|
|
2054
|
+
|
|
2055
|
+
### Sequential merge protocol (SC-72)
|
|
2056
|
+
|
|
2057
|
+
Parallel work creates parallel branches. Merging is ALWAYS sequential:
|
|
2058
|
+
|
|
2059
|
+
```
|
|
2060
|
+
1. Merge PR for issue A → main
|
|
2061
|
+
2. CI runs on updated main → PASS
|
|
2062
|
+
3. Rebase issue B's branch against new main
|
|
2063
|
+
4. Merge PR for issue B → main
|
|
2064
|
+
5. CI runs on updated main → PASS
|
|
2065
|
+
6. Continue for each remaining issue
|
|
2066
|
+
|
|
2067
|
+
NEVER merge two PRs simultaneously — the second merge
|
|
2068
|
+
must see the first's changes to detect conflicts.
|
|
2069
|
+
```
|
|
2070
|
+
|
|
2071
|
+
### Agent concurrency cap (SC-71)
|
|
2072
|
+
|
|
2073
|
+
- Maximum 5 concurrent agents per session (Google Research: diminishing returns beyond this)
|
|
2074
|
+
- Coordination overhead scales quadratically: 10 agents = 45 potential conflict pairs
|
|
2075
|
+
- The cap applies to parallel ship cycles, not to within-cycle agents (Marcus + Quinn + Rook can all run for one issue)
|
|
2076
|
+
|
|
2077
|
+
## Test Baseline and Pre-existing Failures
|
|
2078
|
+
|
|
2079
|
+
### Scope gate captures baseline
|
|
2080
|
+
|
|
2081
|
+
At scope gate (beginning of ship cycle), the test baseline is captured:
|
|
2082
|
+
|
|
2083
|
+
```json
|
|
2084
|
+
"beforeState": {
|
|
2085
|
+
"testBaseline": {
|
|
2086
|
+
"total": 50,
|
|
2087
|
+
"pass": 47,
|
|
2088
|
+
"fail": 3,
|
|
2089
|
+
"failingTests": ["test-a", "test-b", "test-c"],
|
|
2090
|
+
"capturedAt": "2026-09-18T..."
|
|
2091
|
+
}
|
|
2092
|
+
}
|
|
2093
|
+
```
|
|
2094
|
+
|
|
2095
|
+
### Verify gate compares against baseline
|
|
2096
|
+
|
|
2097
|
+
```
|
|
2098
|
+
fail count AFTER ≤ fail count BEFORE = PASS (no regressions)
|
|
2099
|
+
fail count AFTER > fail count BEFORE = FAIL (regression introduced)
|
|
2100
|
+
total count AFTER < total count BEFORE = FAIL (tests removed — possible gaming)
|
|
2101
|
+
```
|
|
2102
|
+
|
|
2103
|
+
### Pre-existing failures are tracked, not ignored
|
|
2104
|
+
|
|
2105
|
+
When scope gate finds pre-existing test failures:
|
|
2106
|
+
|
|
2107
|
+
1. Each failing test not already tracked in an open issue → auto-filed:
|
|
2108
|
+
```
|
|
2109
|
+
gh issue create --title "Pre-existing: {test name} failing"
|
|
2110
|
+
--label "p3-can-wait,pre-existing"
|
|
2111
|
+
--body "Found during ship cycle for #{currentIssue}. Not from current changes."
|
|
2112
|
+
```
|
|
2113
|
+
2. Pre-existing failures don't block current work
|
|
2114
|
+
3. They exist in the backlog with the `pre-existing` label
|
|
2115
|
+
|
|
2116
|
+
### Fix-on-find enforcement
|
|
2117
|
+
|
|
2118
|
+
When an agent discovers any issue during a ship cycle:
|
|
2119
|
+
|
|
2120
|
+
| Can fix in <10 min, no design decision? | Fix it now — same session |
|
|
2121
|
+
| Can't fix quickly? | `gh issue create` with `found-during-#{issue}` label |
|
|
2122
|
+
| Neither? | WARN — must explicitly scope it out in `scopeOut` |
|
|
2123
|
+
|
|
2124
|
+
Verify gate checks: every found issue is either (a) fixed, (b) filed as issue, or (c) in scopeOut. Silently dropping it = **FAIL** — unaddressed gaps are not acceptable.
|
|
2125
|
+
|
|
2126
|
+
## Makefile Fallback Chain
|
|
2127
|
+
|
|
2128
|
+
Not every project has a Makefile. The scan tries sources in priority order:
|
|
2129
|
+
|
|
2130
|
+
| Field | Source 1 (Makefile) | Source 2 (package.json) | Source 3 (null) |
|
|
2131
|
+
|-------|--------------------|-----------------------|-----------------|
|
|
2132
|
+
| dev.start | `make dev-all` target | `scripts.dev` → `bun run dev` | null → skip dev server steps |
|
|
2133
|
+
| dev.testCmd | `make test` target | `scripts.test` → `bun test` | `bun test` (safe default) |
|
|
2134
|
+
| prod.rebuild | `make rebuild` target | `scripts.build` → `bun run build` | null → skip container steps entirely |
|
|
2135
|
+
| prod.smokeTest | `make smoke` target | `scripts.smoke` | null → skip smoke tests |
|
|
2136
|
+
| test.rebuild | `make test-rebuild` target | none | null → skip test container |
|
|
2137
|
+
| test.start | `make prove-up` target | none | null → skip prove container |
|
|
2138
|
+
| test.stop | `make prove-down` target | none | null → skip prove cleanup |
|
|
2139
|
+
|
|
2140
|
+
When a field is null, the workflow step that uses it is SKIPPED — no agent spawned, no command run. The workflow records `environments.{env}.{step} = "SKIP"` with `skipReason: "no config"`.
|
|
2141
|
+
|
|
2142
|
+
## Phase 1.5 — Knowledge Extraction (non-inferrable rules from docs)
|
|
2143
|
+
|
|
2144
|
+
Code scans produce structure. But projects accumulate organic knowledge in docs, ADRs, and specs that agents can't discover from code — intentional anti-patterns, deploy gotchas, disabled features, safety boundaries.
|
|
2145
|
+
|
|
2146
|
+
### Source Scan
|
|
2147
|
+
|
|
2148
|
+
Bootstrap scans these locations for non-inferrable rule candidates:
|
|
2149
|
+
|
|
2150
|
+
| Source | What to Look For |
|
|
2151
|
+
|--------|-----------------|
|
|
2152
|
+
| `docs/adr/*.md` | Decisions marked "accepted" — these are intentional choices |
|
|
2153
|
+
| `docs/*.md` | Patterns: "intentional", "by design", "do not change", "never", "must not" |
|
|
2154
|
+
| `specs/*.md` | Constraints section, anti-patterns section |
|
|
2155
|
+
| `ARCHITECTURE.md` | "Looks like X but is actually Y", intentional coupling |
|
|
2156
|
+
| `PRINCIPLES.md` | Hard rules, pre-flight checks |
|
|
2157
|
+
|
|
2158
|
+
### Extraction Rules
|
|
2159
|
+
|
|
2160
|
+
1. Grep all docs for signal phrases: `intentional|by design|do not|never|must not|anti-pattern|permanently disabled|looks like.*but`
|
|
2161
|
+
2. Extract the sentence + surrounding context (3 lines)
|
|
2162
|
+
3. Deduplicate against existing Hard Constraints in AGENTS.md (content match) and rejected-constraints.md (content-hash)
|
|
2163
|
+
4. Present new candidates to the user for confirmation
|
|
2164
|
+
5. Confirmed candidates → appended to AGENTS.md Hard Constraints section between AUTO-EXTRACTED markers
|
|
2165
|
+
6. Rejected candidates → logged to `reference/rejected-constraints.md` with content-hash so they don't resurface
|
|
2166
|
+
|
|
2167
|
+
### Execution
|
|
2168
|
+
|
|
2169
|
+
- Separate command: `bunx rungate extract-constraints /path/to/project`
|
|
2170
|
+
- NOT auto-run from scaffold (70% false positive rate — human review mandatory)
|
|
2171
|
+
- On re-scaffold: HYGIENE-6 conformity check warns about unreviewed candidates
|
|
2172
|
+
- `--dry-run` is the default. `--apply` writes confirmed candidates to AGENTS.md
|
|
2173
|
+
- On first run: full scan of all docs → present all candidates
|
|
2174
|
+
- On re-run: incremental scan (only docs modified since last scan) → present new candidates only
|
|
2175
|
+
- Existing Hard Constraints are NEVER removed by re-scan — only added to
|
|
2176
|
+
|
|
2177
|
+
### Doc Lifecycle (archive stale, keep current)
|
|
2178
|
+
|
|
2179
|
+
During extraction, docs are also classified:
|
|
2180
|
+
|
|
2181
|
+
| Classification | Criteria | Action |
|
|
2182
|
+
|---------------|----------|--------|
|
|
2183
|
+
| ACTIVE | Git log shows commit within threshold (per-type below) | Keep in `docs/` |
|
|
2184
|
+
| ARCHIVED | Git log older than threshold OR manually moved | Move to `reference/` |
|
|
2185
|
+
|
|
2186
|
+
**Per-doc-type staleness thresholds** (council decision — single 90d threshold flags 84% of DDB docs):
|
|
2187
|
+
|
|
2188
|
+
| Doc type | Path pattern | Threshold | Rationale |
|
|
2189
|
+
|----------|-------------|-----------|-----------|
|
|
2190
|
+
| ADR | `docs/adr/` | NEVER time-staled | Durable decisions — stale only by `status: superseded` |
|
|
2191
|
+
| Spec | `specs/` | 90 days | Active governance docs should be current |
|
|
2192
|
+
| Guide/runbook | `docs/` | 180 days | Operational docs change less frequently |
|
|
2193
|
+
|
|
2194
|
+
**Staleness signal:** Primary = `git log -1 --format=%ci` per file. Frontmatter `last-verified` is optional human override — if present and within threshold, file is ACTIVE regardless of git log. Frontmatter `updated` is NOT used (bulk-stamp dates prove unreliability).
|
|
2195
|
+
|
|
2196
|
+
## Success Criteria
|
|
2197
|
+
|
|
2198
|
+
<!-- Phase subheaders route SCs to test files automatically. -->
|
|
2199
|
+
<!-- The meta-test (test/meta-sc-coverage.test.ts) reads these headers. -->
|
|
2200
|
+
<!-- To add a new SC: paste it under the correct ### Phase header. -->
|
|
2201
|
+
|
|
2202
|
+
### Phase 0 — Scaffold Output
|
|
2203
|
+
|
|
2204
|
+
- [ ] SC-1: Bootstrap Phase 1 (CODE-MAP + rungate) completes before Phase 2 (AGENTS.md + agents/)
|
|
2205
|
+
- [ ] SC-2: rungate.json consumers field populated from CODE-MAP consumer scan, not hardcoded patterns
|
|
2206
|
+
- [ ] SC-3: AGENTS.md environment section reads from rungate.json (not hardcoded)
|
|
2207
|
+
- [ ] SC-4: Zero hardcoded ports/URLs in ship.js — all from rungate.json fields
|
|
2208
|
+
- [ ] SC-5: Zero hardcoded commands in ship.js — all from rungate.json fields
|
|
2209
|
+
- [ ] SC-6: Zero hardcoded project names in ship.js — all from args or rungate.json
|
|
2210
|
+
- [ ] SC-8: All ship.js inline prompts >5 lines extracted to prompts/*.md templates
|
|
2211
|
+
- [ ] SC-11: Re-running bootstrap on an existing project updates AGENTS.md with current CODE-MAP and rungate data
|
|
2212
|
+
- [ ] SC-15: rungate.json generation falls back from Makefile → package.json → null for each field
|
|
2213
|
+
- [ ] SC-16: Null config fields cause workflow steps to SKIP (not error, not spawn rogue agents)
|
|
2214
|
+
- [ ] SC-17: AGENTS.md is under 150 lines — scaffold WARNS if over
|
|
2215
|
+
- [ ] SC-18: Empty sections omitted from AGENTS.md (no consumers = no Consumers section)
|
|
2216
|
+
- [ ] SC-19: CLAUDE.md contains @AGENTS.md bridge for cross-tool compatibility
|
|
2217
|
+
- [ ] SC-20: New code projects get src/index.ts stub as agent starting point
|
|
2218
|
+
- [ ] SC-21: rungate.json has no hardcoded port numbers — all detected from Makefile/config or null
|
|
2219
|
+
- [ ] SC-24: Conformity test calls all 9 suites (scaffold, spec-discovery, spec-drift, doc-hygiene, agent-validation, fallow, constraint-candidates, package-validation, tsconfig-validation)
|
|
2220
|
+
- [ ] SC-25: AGENTS.md Commands section includes all harness commands (bun test, tsc, conformity, sync-tests, create-spec, create-adr, extract-constraints, re-scaffold)
|
|
2221
|
+
- [ ] SC-26: Spec template in project matches harness version — scaffold updates on re-run
|
|
2222
|
+
- [ ] SC-27: sync-spec-tests reads project's specs/ directory (not just harness specs)
|
|
2223
|
+
- [ ] SC-37: Every agent brief includes Core Principles block (verify-before-asserting, no gaps, read AGENTS.md)
|
|
2224
|
+
- [ ] SC-39: CODE-MAP.md staleness detected by git SHA comparison — regenerated when src/ has commits since last scan, not by arbitrary time/count thresholds
|
|
2225
|
+
- [ ] SC-40: Scaffold creates/verifies .gitignore with full security template (node_modules, dist, .env*, .rungate, *.pem, *.key, credentials.json, service-account*)
|
|
2226
|
+
- [ ] SC-73: package.json has required fields (name, type:module, scripts.test, devDependencies.rungate) — scaffold adds missing, never overwrites existing
|
|
2227
|
+
- [ ] SC-74: tsconfig.json has recommended fields (strict:true) — scaffold WARNs if missing, never overwrites
|
|
2228
|
+
- [ ] SC-75: Spec frontmatter has required fields (doc-type, status, testable, governs) — Spec Discovery FAILs if missing
|
|
2229
|
+
- [ ] SC-76: ADR frontmatter has required fields (doc-type:adr, status, created) — Doc Hygiene FAILs if missing
|
|
2230
|
+
- [ ] SC-77: `bunx rungate create-spec` creates spec with all required frontmatter fields pre-populated
|
|
2231
|
+
- [ ] SC-78: `bunx rungate create-adr` creates ADR with all required frontmatter fields pre-populated
|
|
2232
|
+
- [ ] SC-80: Phase 0 creates all 10 directories + all static files BEFORE any scan runs (Phase 1)
|
|
2233
|
+
- [ ] SC-82: Scaffold creates .github/workflows/ci.yml — harness-owned, always regenerated from template + rungate.json
|
|
2234
|
+
- [ ] SC-83: Scaffold creates .github/workflows/gates.yml — harness-owned, always regenerated from template + rungate.json
|
|
2235
|
+
- [ ] SC-84: CI workflow runs all conformity suites that `bun test` runs (same checks, CI-enforced)
|
|
2236
|
+
- [ ] SC-85: Gates workflow includes secret scan (HYGIENE-12) as CI check on every PR
|
|
2237
|
+
- [ ] SC-86: CI workflows read runner, bunVersion, branches from rungate.json ci section — defaults used when null
|
|
2238
|
+
- [ ] SC-87: CI workflow files include "Managed by rungate" header directing users to rungate.json for customization
|
|
2239
|
+
- [ ] SC-88: All 9 ship workflow prompt templates exist in harness prompts/ (discovery, marcus, quinn, rook, serena, aditi, environment, container-rebuild, container-verify)
|
|
2240
|
+
- [ ] SC-89: All 11 Layer 1 methodology templates exist in harness prompts/ (rca, blast-radius, prevention, regression, read-before-write, quinn-decision-tree, evidence-hierarchy, ac-format, coding-principles, testing-strategy, escalation-decision-tree)
|
|
2241
|
+
- [ ] SC-90: Brief-assembler embeds relevant Layer 1 methodology templates into per-agent briefs at ship time
|
|
2242
|
+
- [ ] SC-91: Scaffold creates git pre-commit hook with secret scan (HYGIENE-12) — in-repo, not global
|
|
2243
|
+
- [ ] SC-92: Scaffold creates git pre-push hook with conformity test — in-repo, not global
|
|
2244
|
+
- [ ] SC-93: All project hooks, scripts, and workflows are self-contained in repo — no global dependencies
|
|
2245
|
+
- [ ] SC-94: AGENTS.md "Where to Create Things" section lists correct location for every file type agents create
|
|
2246
|
+
- [ ] SC-95: AGENTS.md "Available MCP Servers" section lists MCP servers configured for the project with tool names
|
|
2247
|
+
- [ ] SC-97: Pre-flight checks git repo (git init if not), package.json (bun init if not), .gitignore (create if not)
|
|
2248
|
+
- [ ] SC-98: Pre-flight runs BEFORE Phase 0 — .gitignore exists before any git add
|
|
2249
|
+
- [ ] SC-99: Post-scaffold commits all harness files — "scaffold: initialize" for new, "scaffold: update" for existing
|
|
2250
|
+
- [ ] SC-100: Post-scaffold commit only runs if there are actual changes (no empty commits)
|
|
2251
|
+
- [ ] SC-101: rungate.json uses field-level merge — auto-detected fields overwritten, manual fields preserved
|
|
2252
|
+
- [ ] SC-107: Makefile fallback chain: each field tries Makefile → package.json → null in order
|
|
2253
|
+
- [ ] SC-114: AGENTS.md "Research Tools" section lists available tools with usage examples
|
|
2254
|
+
- [ ] SC-115: Coding principles reference doc (deep modules, boundary validation, etc.) ships with harness and is referenced in Marcus briefs
|
|
2255
|
+
- [ ] SC-116: Coding principles doc auto-updated via Context7 or research on harness update — not static training data
|
|
2256
|
+
- [ ] SC-117: Testing strategy reference doc (PBT, contract tests, tautological trap) ships with harness and is referenced in Marcus briefs
|
|
2257
|
+
- [ ] SC-118: Writer/verifier separation enforced — gate checks that test author agent != code author agent (SC-66 mechanical)
|
|
2258
|
+
- [ ] SC-119: Test assertions reference AC thresholds, not implementation details — gate WARNs on test files with zero AC-ID references
|
|
2259
|
+
- [ ] SC-120: Context loading follows 5-layer architecture — CLAUDE.md → AGENTS.md → skills (on demand) → agent frontmatter → brief
|
|
2260
|
+
- [ ] SC-121: Agent frontmatter `skills` field preloads role-specific methodology — Marcus gets coding-principles + testing-strategy, Quinn gets journey-format
|
|
2261
|
+
- [ ] SC-122: AGENTS.md contains ONLY non-inferrable details — no info agents could discover by reading src/ or running commands
|
|
2262
|
+
- [ ] SC-123: .github/copilot-instructions.md created if missing, NEVER overwritten if exists — scaffold skips on existing
|
|
2263
|
+
- [ ] SC-124: rungate.json envVars field populated from .env.example if it exists — null if no .env.example
|
|
2264
|
+
- [ ] SC-125: CODE-MAP.md frontmatter includes scan-paths array listing directories scanned
|
|
2265
|
+
- [ ] SC-126: AGENTS.md Code Style section auto-detected from runtime (Bun/Node), language (TS/JS), module system (ESM/CJS)
|
|
2266
|
+
- [ ] SC-128: AGENTS.md includes Harness-Managed Files section with file/customize/don't table
|
|
2267
|
+
- [ ] SC-138: Agent briefs cap at 10 essential rules per Layer 1 template — instruction stacking beyond 16 degrades compliance 50%+
|
|
2268
|
+
- [ ] SC-139: AGENTS.md rules content (excluding tables/headers) stays under 200 words — always-on context must be lean
|
|
2269
|
+
- [ ] SC-140: Brief structure: Core Principles at START, ACs at END — methodology in middle (mitigates "Lost in the Middle" attention decay)
|
|
2270
|
+
- [ ] SC-141: Templates use positive framing ("use X" not "don't use Y") — negative instructions capped at 10 per brief
|
|
2271
|
+
- [ ] SC-142: Every Layer 1 template includes at least one code example per principle — examples outperform descriptions
|
|
2272
|
+
- [ ] SC-143: Each template declares activation mode: always-on, file-pattern, agent-decided, or manual
|
|
2273
|
+
- [ ] SC-144: Rule lifecycle: every rule in AGENTS.md has a `since` date in comment — rules older than 90 days without revalidation flagged at re-scaffold
|
|
2274
|
+
- [ ] SC-145: Brief-assembler orders sections: identity (role) → principles → methodology → ACs → verification commands — never methodology after ACs
|
|
2275
|
+
- [ ] SC-146: Every agent brief categorizes rules into three tiers: Always Do (autonomous), Ask First (needs approval), Never Do (hard stops) — gate validates "Never Do" items mechanically
|
|
2276
|
+
- [ ] SC-147: rungate.json includes `harnessVersion` (semver from harness package.json) and `scaffoldedAt` (ISO timestamp) — projects know what version they were scaffolded with
|
|
2277
|
+
- [ ] SC-148: Conformity test WARNs when installed harness version differs from `harnessVersion` in rungate.json — signals re-scaffold needed
|
|
2278
|
+
- [ ] SC-159: AGENTS.md Commands table includes `Check findings | cat .rungate/conformity-findings.json` row
|
|
2279
|
+
- [ ] SC-160: AGENTS.md Quick Reference includes "After test failures, read `.rungate/conformity-findings.json` for structured findings with fix commands"
|
|
2280
|
+
|
|
2281
|
+
### Phase 1 — Knowledge Extraction + Doc Hygiene
|
|
2282
|
+
|
|
2283
|
+
- [ ] SC-12: Bootstrap scans docs for non-inferrable rule candidates using signal phrases
|
|
2284
|
+
- [ ] SC-13: Extracted candidates presented for user confirmation before writing to AGENTS.md
|
|
2285
|
+
- [ ] SC-14: Staleness uses git log date with per-type thresholds (ADR exempt, spec 90d, guide 180d)
|
|
2286
|
+
- [ ] SC-22: Re-scaffold reports stale constraints (Hard Constraints referencing deleted files)
|
|
2287
|
+
- [ ] SC-23: Re-scaffold reports line count if AGENTS.md exceeds 150 lines
|
|
2288
|
+
- [ ] SC-30: Misplaced spec/ADR/doc files are FAIL not WARN (HYGIENE-7/8/9)
|
|
2289
|
+
- [ ] SC-106: Doc lifecycle: stale docs (beyond threshold) flagged for archive to reference/ — per-type thresholds applied
|
|
2290
|
+
- [ ] SC-137: Rejected constraint candidates logged to reference/rejected-constraints.md with content-hash dedup — same candidate never resurfaces
|
|
2291
|
+
- [ ] SC-149: Conformity findings written to `.rungate/conformity-findings.json` with structured format (ruleId, severity, file, message, fixCommand)
|
|
2292
|
+
- [ ] SC-150: Every FAIL finding includes a `fixCommand` — agents can execute it directly to resolve the issue
|
|
2293
|
+
- [ ] SC-154: Unified findings report includes `constraintCandidates` array with rule, source, hash, and status (pending/applied/rejected)
|
|
2294
|
+
- [ ] SC-155: Unified findings report includes `staleness` array with file, daysSince, threshold, and type
|
|
2295
|
+
- [ ] SC-156: HYGIENE-6 pipes extract-constraints results (candidates + staleness) into unified findings report — nothing lost to stdout
|
|
2296
|
+
- [ ] SC-157: Gate runner reads `.rungate/conformity-findings.json` after `bun test` completes and prints structured output (findings with fix commands, candidates, stale docs)
|
|
2297
|
+
- [ ] SC-158: Gate runner stores conformityFindings in workflow-state.json — agents read fix commands from workflow-state, not test output
|
|
2298
|
+
- [ ] SC-151: HYGIENE-7 detects spec files at root (not in specs/) and produces FAIL with `mv` fix command
|
|
2299
|
+
- [ ] SC-152: HYGIENE-8 detects ADR files outside docs/adr/ and produces FAIL with `mv` fix command
|
|
2300
|
+
- [ ] SC-153: HYGIENE-9 detects doc files at root (not in docs/, not allowlisted) and produces FAIL with `mv` fix command
|
|
2301
|
+
|
|
2302
|
+
### Phase 2 — Gate Enforcement + Ship Behavior
|
|
2303
|
+
|
|
2304
|
+
- [ ] SC-7: Container-rebuild agent NOT spawned when prod.rebuild is null
|
|
2305
|
+
- [ ] SC-9: rungate-schema.ts includes test.rebuild, test.start, test.stop, test.apiBase fields
|
|
2306
|
+
- [ ] SC-10: Schema fields dev.typeCheck, prod.smokeTest, contextDocs read by at least one workflow
|
|
2307
|
+
- [ ] SC-28: Scope gate runs project conformity tests — FAIL blocks scope
|
|
2308
|
+
- [ ] SC-29: Verify gate runs project conformity tests — FAIL blocks verify
|
|
2309
|
+
- [ ] SC-31: Scope gate captures test baseline (pass/fail count + failing test names)
|
|
2310
|
+
- [ ] SC-32: Verify gate compares test count — fail increase = regression = FAIL
|
|
2311
|
+
- [ ] SC-33: Pre-existing test failures auto-filed as GitHub issues with "pre-existing" label
|
|
2312
|
+
- [ ] SC-34: Found issues during ship cycle either fixed, filed, or explicitly scoped out
|
|
2313
|
+
- [ ] SC-35: `bunx rungate create-spec` creates spec at specs/ with correct frontmatter
|
|
2314
|
+
- [ ] SC-36: `bunx rungate create-adr` creates ADR at docs/adr/ with auto-incremented number
|
|
2315
|
+
- [ ] SC-38: Ship cycle runs scaffold after verify gate PASS — auto-refresh CODE-MAP, briefs, AGENTS.md
|
|
2316
|
+
- [ ] SC-41: Conformity checks for secret patterns in staged/committed files (HYGIENE-12) — catches secrets on ALL tiers including LIGHT
|
|
2317
|
+
- [ ] SC-42: Completed slugs (DONE + proven) auto-archived after 24h
|
|
2318
|
+
- [ ] SC-43: Archived slugs deleted after 30 days
|
|
2319
|
+
- [ ] SC-44: Ship cycle checks for existing slug by issue number before creating new one
|
|
2320
|
+
- [ ] SC-48: ACs use Given/When/Then format with four required fields: trigger, output, verify command, exclusions
|
|
2321
|
+
- [ ] SC-49: Every AC has an executable verification command (oracle) — not prose description
|
|
2322
|
+
- [ ] SC-50: Evidence hierarchy enforced per AC — minimum tier: CODE→A, UI→B, BUG-FIX→S, static→C supplementary only
|
|
2323
|
+
- [ ] SC-51: Self-attestation (tier F) in any AC evidence = automatic FAIL regardless of other evidence
|
|
2324
|
+
- [ ] SC-52: Garbage test applied to every AC — "Could garbage data pass this?" If yes, AC is rejected
|
|
2325
|
+
- [ ] SC-53: RCA section required in Marcus brief for bug-fix issues — rootCause, prediction, predictionVerified
|
|
2326
|
+
- [ ] SC-54: Gate checks RCA fields populated on bug-fix issues — empty rootCause = WARN (graduates to FAIL after 30d)
|
|
2327
|
+
- [ ] SC-55: Quinn briefs use typed journey format — steps with action, wait_for, typed assertions, on_fail
|
|
2328
|
+
- [ ] SC-56: Quinn primary perception is browser_snapshot (a11y tree), screenshots only on FAIL or visual checks
|
|
2329
|
+
- [ ] SC-57: Quinn circuit breaker — 3 consecutive FAIL steps abort journey with partial results
|
|
2330
|
+
- [ ] SC-58: Proof-of-fix includes negative control (revert fix, confirm bug returns) for S-tier evidence
|
|
2331
|
+
- [ ] SC-59: Blast radius section in Marcus brief — filesRead ≥ filesChanged enforced
|
|
2332
|
+
- [ ] SC-60: Read-before-write ratio ≥ 3:1 — gate checks read vs write token count
|
|
2333
|
+
- [ ] SC-61: Brief-assembler generates per-role templates (Marcus, Quinn, Rook, Serena, Aditi) from three layers
|
|
2334
|
+
- [ ] SC-62: Templates are three-layered: harness-generic + project-config (rungate.json) + issue-specific (ACs)
|
|
2335
|
+
- [ ] SC-63: Spec content hash tracked — hash mismatch at gate = FAIL (tests must match current spec)
|
|
2336
|
+
- [ ] SC-64: Test count post ≥ pre at verify gate — decrease = FAIL "tests removed, verify no gaming"
|
|
2337
|
+
- [ ] SC-65: Prevention-oriented fixes: guard at boundary, type narrowing, pattern audit for same-bug-class
|
|
2338
|
+
- [ ] SC-66: Writer and verifier are separate agents — same agent cannot write code AND verify its own ACs
|
|
2339
|
+
- [ ] SC-67: Quinn journey steps auto-generated from UI-type ACs by brief-assembler
|
|
2340
|
+
- [ ] SC-68: Evidence without assertion = tier F — screenshot taken but nothing checked = self-attestation
|
|
2341
|
+
- [ ] SC-79: Conformity includes `runPackageValidation` check for package.json required fields
|
|
2342
|
+
- [ ] SC-81: Phase 0 file creation is additive — never overwrites existing content, only adds missing fields/entries
|
|
2343
|
+
- [ ] SC-96: rungate.json `mcp` section declares MCP servers available to agents — scaffold includes in agent briefs
|
|
2344
|
+
- [ ] SC-102: Re-scaffold auto-fixes broken refs, unlisted specs, missing CODE-MAP ref, and pages drift — not just reports them
|
|
2345
|
+
- [ ] SC-103: Evidence-to-tier mapping implemented in gate — evidenceMethod.type maps to correct tier per mapping table
|
|
2346
|
+
- [ ] SC-104: File ownership model enforced — harness-owned files ARE regenerated, co-owned files are NOT overwritten
|
|
2347
|
+
- [ ] SC-105: Scaffold is idempotent — running from scratch produces same result as re-scaffold (Core Principle 6)
|
|
2348
|
+
- [ ] SC-108: Quinn journey stored in slug directory (~/.rungate/{slug}/quinn-journey.yaml) — ephemeral, follows slug lifecycle
|
|
2349
|
+
- [ ] SC-109: Fix-on-find: silently dropped issues = FAIL at verify gate, not WARN
|
|
2350
|
+
- [ ] SC-110: Port allocation for parallel worktrees checks for collisions before assigning offset
|
|
2351
|
+
- [ ] SC-111: Escalation decision tree embedded in every agent brief — agents know research tools exist
|
|
2352
|
+
- [ ] SC-112: Circuit breaker iteration 2 requires research tool invocation before next attempt — gate checks research evidence exists before allowing iteration 3
|
|
2353
|
+
- [ ] SC-113: rungate.json `research` section declares available research tools — brief-assembler includes only available tools
|
|
2354
|
+
- [ ] SC-127: All 3 gate prompt templates exist in harness prompts/ (ac-adversary.md, evidence-validator.md, prove-reproducer.md)
|
|
2355
|
+
- [ ] SC-129: Gate WARNs if a new module file has more exported functions than internal functions (shallow module signal)
|
|
2356
|
+
- [ ] SC-130: Gate WARNs if a module with >3 importers has no contract test
|
|
2357
|
+
- [ ] SC-131: Anti-criteria (SC-A*) use absence verification — Verify command must confirm absence, not presence
|
|
2358
|
+
- [ ] SC-132: Tier D evidence (grep) capped at 25% of total evidence portfolio per issue — gate FAIL if exceeded
|
|
2359
|
+
- [ ] SC-133: Brief-assembler rejects Quinn journeys over 8 steps and auto-splits into multiple journeys with entry conditions
|
|
2360
|
+
- [ ] SC-134: Files changed outside brief's listed files → WARN "Files changed outside brief: {list}"
|
|
2361
|
+
- [ ] SC-135: Gate cross-validates proofOfFix in workflow-state.json against prove-evidence.json — mismatch = FAIL
|
|
2362
|
+
|
|
2363
|
+
### Phase 3 — Parallel Work
|
|
2364
|
+
|
|
2365
|
+
- [ ] SC-45: Container lock checked mechanically (Makefile target, not behavioral rule) — blocks rebuild/prove-up if lock exists
|
|
2366
|
+
- [ ] SC-46: Ship workflow supports worktree isolation for parallel execution
|
|
2367
|
+
- [ ] SC-47: Container lock released mechanically after test completion (cleanup triggers next queued issue)
|
|
2368
|
+
- [ ] SC-69: Parallel work: file-set overlap detection before assigning concurrent issues to agents
|
|
2369
|
+
- [ ] SC-70: Parallel work: port/namespace isolation beyond git worktree file isolation
|
|
2370
|
+
- [ ] SC-71: Parallel work: cap concurrent agents at 5 per session (diminishing returns beyond)
|
|
2371
|
+
- [ ] SC-72: Parallel work: sequential merge with CI verification between each PR
|
|
2372
|
+
- [ ] SC-136: Worktree branches use deterministic naming: issue-{N}-{slug}
|
|
2373
|
+
|
|
2374
|
+
### Anti-Criteria
|
|
2375
|
+
|
|
2376
|
+
- [ ] SC-A1: No workflow file imports or references values from a specific project (DDB, asaCommandCenter, etc.)
|
|
2377
|
+
- [ ] SC-A2: No hardcoded file paths in rungate.json generation (no `dashboard/src/App.tsx`, no `callGemini` patterns)
|
|
2378
|
+
- [ ] SC-A3: No references to "PAI" in lock files, work dirs, or runtime paths — all use "rungate" naming
|
|
2379
|
+
## Evaluated Concerns
|
|
2380
|
+
|
|
2381
|
+
Concerns raised during spec review, evaluated, and documented for future reference:
|
|
2382
|
+
|
|
2383
|
+
| Concern | Verdict | Reasoning |
|
|
2384
|
+
|---------|---------|-----------|
|
|
2385
|
+
| Framework changes (Hono→Express) | Not a gap | Re-scaffold + CODE-MAP detects whatever's in src/. Agent briefs regenerate with new structure |
|
|
2386
|
+
| Monorepo support | Future work | Single-project repos are current target. Nested AGENTS.md per package is future scope |
|
|
2387
|
+
| CI/CD setup | Handled | Harness creates ci.yml + gates.yml with quality gates. Custom CI (deploy, release) is project-specific |
|
|
2388
|
+
| Branch protection / PR templates | Out of scope | Project-specific GitHub settings, not harness concern |
|
|
2389
|
+
| Version migration (harness major bump) | Handled | `bun update rungate` — conformity tests catch breaking changes immediately |
|
|
2390
|
+
| Collaborator onboarding | Handled | `bun install` gets devDep, `bun test` verifies conformity. No separate setup step |
|
|
2391
|
+
| First git push | Fixed by SC-40 | .gitignore ensures clean first push |
|
|
2392
|
+
| Dependency management (bun.lockb) | Handled | Standard bun behavior — lockb committed, node_modules gitignored |
|
|
2393
|
+
| Mutation testing for test quality | Future work | AI tests: 91% coverage but 34% mutation score. Meta's approach raised to 89.5%. Valuable but heavyweight — test count comparison is minimum viable gate for now |
|
|
2394
|
+
| LLM-as-judge for verification | Not adopted | 4 known bias modes (length, position, self-preference, non-determinism). Mechanical gates preferred over LLM judgment |
|
|
2395
|
+
| Agent teams / swarm mode | Future work | Claude Code agent teams in research preview. Current harness uses single-agent-per-role. Parallel work via worktrees |
|
|
2396
|
+
| Merge queue tooling (Graphite/Mergify) | Future work | Merge queue becomes bottleneck at scale. Current: sequential manual merge. Worth evaluating when parallel work is routine |
|
|
2397
|
+
| Visual regression testing (Percy/Chromatic) | Future work | Pixel-diff tools exist but add infra complexity. A11y-first assertion covers functional; visual regression is supplementary |
|
|
2398
|
+
| Context window management for long journeys | Handled by design | Quinn journeys capped at 5-8 steps. Instruction compliance decays from 73% at turn 5 to 33% by turn 16 (Gamage 2026) |
|
|
2399
|
+
|
|
2400
|
+
## Cautions
|
|
2401
|
+
|
|
2402
|
+
- Bootstrap Phase 1 depends on fallow being installed. If fallow fails, CODE-MAP.md should still generate from other scans (routes, components, dirs) — fallow sections empty, not entire file missing.
|
|
2403
|
+
- rungate.json consumers field: auto-detection from CODE-MAP may miss consumers that are dynamically imported or configured at runtime. Manual override must be preserved on re-scan.
|
|
2404
|
+
- Hard Constraints in AGENTS.md must survive regeneration — they're the one section that can't be auto-detected from code. **CRITICAL (council finding):** The preservation regex must handle Hard Constraints as the last section — the lookahead `(?=\n## |\s*$)` handles this (fixed).
|
|
2405
|
+
- Prompt templates with ${VAR} placeholders: if a variable is undefined (field missing from config), the template should show a clear placeholder or instruction, not leave a raw ${VAR} string in the agent prompt.
|
|
2406
|
+
- `rejected-constraints.md` must use content-hash dedup so rejections survive line-number changes across doc edits. Prevents the same false positive from resurfacing on every re-scan.
|
|
2407
|
+
- No `--auto-accept` flag on extract-constraints. 70% false positive rate means human review is mandatory. `--dry-run` is the default.
|
|
2408
|
+
- AGENTS.md over 150 lines: research shows 20-23% inference cost increase with no behavior improvement. LLM-generated context files decrease performance by 3% (ETH Zurich). Keep it concise, human-written, and focused on non-inferrable details.
|
|
2409
|
+
- Scaffold partial failure: if scaffold fails halfway (e.g., CODE-MAP generated but rungate.json errors out), the project may have stale data. Recovery: re-run scaffold — it's idempotent, each file is independently valid. No rollback mechanism needed because each step overwrites cleanly.
|
|
2410
|
+
- Secrets on LIGHT tier: LIGHT tier skips Rook (security scan). Without HYGIENE-12 (secret pattern check in conformity), an agent could commit an API key through LIGHT without any mechanical catch. HYGIENE-12 runs on ALL tiers via `bun test`, closing this gap.
|
|
2411
|
+
- .gitignore on new projects: pre-flight creates .gitignore from full security template BEFORE `bun init` runs, ensuring secrets are excluded from the first commit. On existing projects, scaffold verifies required entries and appends missing ones.
|
|
2412
|
+
- RCA WARN→FAIL graduation: SC-54 starts as WARN to avoid blocking existing workflows. After 30 days of data collection, evaluate false positive rate. If <10%, graduate to FAIL. If >10%, refine the check before graduating.
|
|
2413
|
+
- Evidence hierarchy adoption: existing workflows have grep-heavy evidence. SC-50 enforcement should not retroactively break passing issues. Apply to NEW issues only — existing workflow-state.json files are grandfathered.
|
|
2414
|
+
- Quinn a11y tree dependency: browser_snapshot() requires Playwright MCP server running. If MCP unavailable, Quinn falls back to screenshots with WARN. Never block QA on infra failure.
|
|
2415
|
+
- Negative control cost: Step 5 (revert + re-test) adds ~30-60 seconds per prove cycle. Worth it for bug fixes (46% of validations are non-discriminating without it). Can skip for feature work where there's no "before" state.
|
|
2416
|
+
- Typed journey generation: brief-assembler generates journeys from ACs with UI type. If AC threshold is too vague for typed assertion (e.g., "looks good"), the journey step gets `type: visual_check` which requires screenshot + human review — it doesn't silently skip.
|
|
2417
|
+
- Three-layer merge conflicts: if project-config (Layer 2) and harness-generic (Layer 1) conflict, Layer 2 wins. Project knows its own constraints better than the generic template. Issue-specific (Layer 3) can override both — but gate still enforces minimums.
|
|
2418
|
+
- Parallel work file-overlap detection (SC-69) is advisory at first. False positives (two issues legitimately touching shared utils) should not block. Start as WARN, graduate to FAIL after calibration.
|
|
2419
|
+
- AI-generated code security: 44% of AI-generated tasks produce known security flaws (Veracode 2026). Rook scan on EVERY build cycle is not optional — it's the only mechanical catch for a 44% defect rate.
|
|
2420
|
+
- Iterative refinement paradox: asking an agent to "improve" its own code 5+ times increases vulnerabilities 37.6% (IEEE-ISTAS 2025). The harness's circuit breaker (3 iterations max, research required at iteration 2) exists partly for this reason.
|
|
2421
|
+
- Read-before-write enforcement: the 3:1 ratio (SC-60) is measured from workflow-state.json token counts, not file counts. Agents that read 1 file and change 3 are likely patching symptoms. The ratio is a heuristic — some surgical fixes legitimately have low ratios.
|
|
2422
|
+
- Rule count caps (SC-138): the 25-rule ceiling is a design target, not a gate FAIL. The data shows compliance degrades at 16+, but the gate-first architecture compensates — the rules most likely to be forgotten are methodology steps, which gates enforce mechanically anyway.
|
|
2423
|
+
- Brief assembly order (SC-140, SC-145): the U-shaped attention curve is real but varies by model and context length. The ordering is a best-effort optimization, not a guarantee. The primary defense remains mechanical gates, not attention management.
|
|
2424
|
+
- Always-on 200-word budget (SC-139): measured as prose content in AGENTS.md, excluding table formatting, headers, and code blocks. Tables are information-dense and parse differently than prose rules.
|
|
2425
|
+
- Activation mode declarations (SC-143): activation modes are metadata for the brief-assembler, not runtime config. The brief-assembler decides what to include based on mode + current context. A template with `mode: agent-decided` is included only when the brief-assembler judges it relevant to the issue type.
|
|
2426
|
+
- Rule lifecycle 90-day flag (SC-144): the WARN is advisory, not blocking. Some rules are permanent (security constraints, architectural invariants). The flag is a prompt for human review, not an expiration.
|
|
2427
|
+
- 15 tool-call compliance cliff: this is why the harness uses gates (post-hoc verification) rather than relying on agents to self-enforce rules during long sessions. The cliff validates the architectural decision, not a gap to fix.
|
|
2428
|
+
|
|
2429
|
+
## Research Sources
|
|
2430
|
+
|
|
2431
|
+
Research informing design decisions D-12 through D-23 and SCs 48-145:
|
|
2432
|
+
|
|
2433
|
+
| Topic | Key finding | Source |
|
|
2434
|
+
|-------|------------|--------|
|
|
2435
|
+
| AC format effectiveness | Human-refined specs: ~50% error reduction | Augment Code, ClearSpec |
|
|
2436
|
+
| AC constrainedness vs acceptance | Well-constrained: 82.1% vs ambiguous: 66.1% | ArXiv PR Acceptance Study (7,156 PRs) |
|
|
2437
|
+
| Evidence discrimination | 46% of validation events non-discriminating | BSG-VA framework (arXiv July 2026) |
|
|
2438
|
+
| Self-attestation failure | 31.7% semantic drift miss rate same-agent | Independent-Checker Pattern |
|
|
2439
|
+
| Test deletion gaming | Documented anti-pattern across multiple teams | DEV Community, FreeCodeCamp |
|
|
2440
|
+
| Mutation testing | 91% coverage, 34% mutation score in AI tests | MutGen study, Meta FSE 2025 |
|
|
2441
|
+
| SWE-bench false positives | 50% of passing PRs wouldn't merge | METR March 2026 |
|
|
2442
|
+
| A11y tree vs screenshots | Degraded a11y: 80%→42% success | CHI 2026 (UC Berkeley + U Michigan) |
|
|
2443
|
+
| Quinn failure pattern | Explore→Explore→Explore loop = failure | ASE 2025 trajectory research |
|
|
2444
|
+
| Read-before-write | Context efficiency r=0.928 with solve rate | SWE-Explore (June 2026) |
|
|
2445
|
+
| Token ratio | Top performers: 10:1 read-to-write | Epoch AI SWE-bench evaluation |
|
|
2446
|
+
| Context file quality | LLM-generated: -3% success, +23% cost | ETH Zurich (Feb 2026) |
|
|
2447
|
+
| Instruction compliance decay | 73% at turn 5 → 33% by turn 16 | Gamage et al. 2026 |
|
|
2448
|
+
| AI code defect rate | 1.7-2.7x more defects than human | Multiple 2026 studies |
|
|
2449
|
+
| AI security flaw rate | 44% of tasks produce known flaws | Veracode July 2026 |
|
|
2450
|
+
| Change failure rate | Up ~30% since AI adoption | Cortex 2026 Benchmark |
|
|
2451
|
+
| Iterative refinement risk | 37.6% vulnerability increase after 5 rounds | IEEE-ISTAS 2025 |
|
|
2452
|
+
| Adversarial review | 3 agents with structured disagreement beat 5 | ICML 2026 Workshop |
|
|
2453
|
+
| RCA vs symptom-fix | 63% of top scores retrieve, not derive | Cursor reward-hacking study 2026 |
|
|
2454
|
+
| Hermes 4-phase protocol | Production-tested systematic debugging | NousResearch |
|
|
2455
|
+
| Agentless pipeline | 27.33% SWE-bench Lite at $0.34/bug | FSE 2025 |
|
|
2456
|
+
| AgentTrace | 94.9% Hit@1, 69x faster than LLM diagnosis | ICLR 2026 Workshop |
|
|
2457
|
+
| Parallel agent conflicts | 41.7% cross-agent conflict rate | 33,596 PR study |
|
|
2458
|
+
| Worktree isolation | Universal standard, 3-5 agents optimal | Claude Code, Codex, Devin |
|
|
2459
|
+
| Merge queue bottleneck | Max 48 PRs/day at 30min CI | Industry analysis |
|
|
2460
|
+
| Agent PR quality | AI PRs break main 1.9% vs human 4.4% | Mergify 200K+ merges |
|
|
2461
|
+
| Deep module violations | AI agents: 10-50x more boundary violations/session | Ousterhout observation, SWE-bench analysis |
|
|
2462
|
+
| Property-based testing | 984 bugs across 100 packages, 56% true bugs | Anthropic PBT research |
|
|
2463
|
+
| AI test quality gap | 91% coverage, 34% mutation score | MutGen study |
|
|
2464
|
+
| Mutation testing ROI | 55% fewer production bugs | Meta FSE 2025 (mutation score raised to 89.5%) |
|
|
2465
|
+
| Testing pyramid inversion | PBT > contract > integration > unit for AI code | Multiple studies synthesis |
|
|
2466
|
+
| Tautological testing | AI tests verify implementation, not specification | DEV Community, FreeCodeCamp pattern analysis |
|
|
2467
|
+
| Consumer-driven contracts | Horizontal boundary testing catches 3x more AI-code bugs | Contract testing studies |
|
|
2468
|
+
| Context loading architecture | Subagents load CLAUDE.md hierarchy + preloaded skills | Anthropic Claude Code docs (Context7) |
|
|
2469
|
+
| Template compression | >150 lines methodology = 20-23% cost, no behavior improvement | ETH Zurich (Feb 2026) |
|
|
2470
|
+
| Context budget | Heaviest agent (Marcus) uses 3.1% of 200k window; full cycle 11.1% | Rungate context budget analysis |
|
|
2471
|
+
| Instruction stacking collapse | 96.4% → 57.7% compliance at 16 stacked rules; 44.7% at 20 | arXiv:2608.02639 (2026) |
|
|
2472
|
+
| IFScale benchmark | 94% at 1-3 instructions → 68% at 500 | Jaroslawicz et al. 2025 |
|
|
2473
|
+
| Instruction complexity cliff | Models reliably satisfy ~3 concurrent constraints | TianPan.co research |
|
|
2474
|
+
| Lost in the Middle | 30-50% accuracy drop for middle-positioned content | Stanford TACL 2024, Veseli et al. 2025 |
|
|
2475
|
+
| Context rot | All 18 models degraded as input length grew; degradation at 25% capacity | Chroma study (July 2025) |
|
|
2476
|
+
| AGENTS.md length vs performance | >500 lines underperforms 200-line files on instruction-following | DeployHQ 2026 |
|
|
2477
|
+
| AGENTS.md empirical study | Human-written: +4% success, -35-55% bugs; LLM-generated: -3% success | Gloaguen et al. 2026 |
|
|
2478
|
+
| AGENTS.md adoption | 60,000+ repos, 25+ tools, Linux Foundation AAIF governance | agents.md, AAIF (Dec 2025) |
|
|
2479
|
+
| Cursor activation modes | 4 modes: always-on, glob, agent-decided, manual | Cursor docs |
|
|
2480
|
+
| Codex config separation | .codex/config.toml (how) vs AGENTS.md (what) | OpenAI Codex docs |
|
|
2481
|
+
| Anthropic system prompt reduction | Removed 80% of Claude Code system prompt for Claude 5 without loss | Anthropic engineering blog |
|
|
2482
|
+
| Second occurrence rule | Only add rule after same mistake happens twice | Anthropic Claude Code best practices |
|
|
2483
|
+
| Aggressive language counterproductive | "CRITICAL!", "YOU MUST" produces worse results than calm instructions | Anthropic context engineering guide |
|
|
2484
|
+
| Format: examples > descriptions | Specific tool mentions increase agent usage from 0.01 to 1.6x/task | Addy Osmani research |
|
|
2485
|
+
| Positive > negative framing | Cap negative instructions at 10-15; past that, model confuses constraints | VirtusLab, Dust research |
|
|
2486
|
+
| 15 tool-call compliance cliff | System prompt constraints lose influence past ~15 tool calls | Engineer documentation, attention dilution |
|
|
2487
|
+
| Runtime compliance (C-Trace) | 97.7% recovery with semantic feedback vs 31.4% with bare denials | arXiv:2606.19242 |
|
|
2488
|
+
| Microsoft Agent Governance | Sub-millisecond external policy enforcement at runtime | Microsoft Agent Governance Toolkit (April 2026) |
|
|
2489
|
+
| Sourcegraph context engineering | 1,281 runs: more context degrades performance; smarter selection, not bigger windows | Sourcegraph CodeScaleBench (May 2026) |
|
|
2490
|
+
| Spec-Driven Development | Dominant coding agent pattern: structured specs feed agent briefs | GitHub Spec Kit, Kiro (AWS), Claude Code Skills |
|
|
2491
|
+
| Multi-agent production survival | 73% Fortune 500 run multi-agent; 62% of early deployments fail to reach prod | Multi-agent production survey 2026 |
|
|
2492
|
+
| Compact at 60-75% capacity | Not 95%; primary lever for long-horizon agents | Anthropic effective harnesses blog |
|
|
2493
|
+
| Google ADK context layers | Static/Turn/User/Cache — most structured production context architecture | Google ADK docs |
|