harness-everything 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (169) hide show
  1. package/AGENTS.md +2 -0
  2. package/BENCHMARK_SOP.md +62 -0
  3. package/CHANGELOG.md +107 -0
  4. package/Dockerfile +18 -0
  5. package/LICENSE +201 -0
  6. package/README.md +197 -0
  7. package/VERIFICATION.md +347 -0
  8. package/bin/cli.js +97 -0
  9. package/build-multi-agent-system/AGENTS.md +50 -0
  10. package/build-multi-agent-system/README.md +63 -0
  11. package/build-multi-agent-system/SKILL.md +68 -0
  12. package/create-agent-launcher/SKILL.md +48 -0
  13. package/create-agent-launcher/examples/csharp-monorepo.md +123 -0
  14. package/create-agent-launcher/examples/typescript-cli.md +126 -0
  15. package/create-agent-launcher/guidelines/platform-claude.md +77 -0
  16. package/create-agent-launcher/guidelines/platform-codex.md +146 -0
  17. package/create-agent-launcher/guidelines/platform-copilot.md +74 -0
  18. package/create-agent-launcher/guidelines/platform-gemini.md +101 -0
  19. package/create-agent-launcher/guidelines/role-capabilities.md +108 -0
  20. package/create-agent-launcher/templates/README.md +17 -0
  21. package/create-agent-launcher/templates/agent-launcher.md +68 -0
  22. package/create-agent-launcher/templates/backend-developer-csharp.md +56 -0
  23. package/create-agent-launcher/templates/backend-developer-polyglot.md +62 -0
  24. package/create-agent-launcher/templates/backend-developer-ts.md +57 -0
  25. package/create-agent-launcher/templates/memory-keeper.md +37 -0
  26. package/create-agent-launcher/templates/requirement-analyzer.md +66 -0
  27. package/create-agent-launcher/templates/specialist-agent.md +34 -0
  28. package/create-agent-launcher/templates/task-verifier.md +56 -0
  29. package/create-agent-launcher/templates/tool-maker.md +33 -0
  30. package/create-agent-launcher/workflows/01-init.md +22 -0
  31. package/create-agent-launcher/workflows/02-analysis.md +26 -0
  32. package/create-agent-launcher/workflows/03-generation.md +29 -0
  33. package/create-agent-launcher/workflows/04-launcher.md +20 -0
  34. package/docker-verify.sh +266 -0
  35. package/docs/architecture.md +95 -0
  36. package/docs/philosophy.md +45 -0
  37. package/docs/reflection.md +34 -0
  38. package/docs/routing.md +45 -0
  39. package/docs/workflows/build-multi-agent-system.md +62 -0
  40. package/docs/workflows/create-agent-launcher.md +62 -0
  41. package/docs/workflows/environment-detection.md +63 -0
  42. package/docs/workflows/eval-harness.md +57 -0
  43. package/docs/workflows/fable-discipline.md +59 -0
  44. package/docs/workflows/fable-mode.md +32 -0
  45. package/docs/workflows/git-commit.md +62 -0
  46. package/docs/workflows/grill-me.md +62 -0
  47. package/docs/workflows/grill-with-docs.md +62 -0
  48. package/docs/workflows/harness-everything.md +92 -0
  49. package/docs/workflows/improve-codebase-architecture.md +60 -0
  50. package/docs/workflows/install-cognitive-os.md +60 -0
  51. package/docs/workflows/repo-docs.md +59 -0
  52. package/docs/workflows/rewrite-commits.md +60 -0
  53. package/docs/workflows/security-review.md +58 -0
  54. package/docs/workflows/self-evolve.md +59 -0
  55. package/docs/workflows/skill-creator.md +75 -0
  56. package/docs/workflows/skill-style.md +64 -0
  57. package/docs/workflows/tdd.md +39 -0
  58. package/docs/workflows/todo-driven-workflow.md +36 -0
  59. package/docs/workflows/using-git-worktrees.md +63 -0
  60. package/docs/workflows/verification-loop.md +69 -0
  61. package/docs/workflows/verify-before-claim.md +60 -0
  62. package/docs/workflows/zoom-out.md +62 -0
  63. package/environment-detection/SKILL.md +77 -0
  64. package/environment-detection/scripts/preflight.js +93 -0
  65. package/eval-framework/behavioral-test.js +105 -0
  66. package/eval-framework/cases/case1.json +5 -0
  67. package/eval-framework/cases/case2.json +5 -0
  68. package/eval-framework/cases/case3.json +5 -0
  69. package/eval-framework/cases/case4-zh-tier2.json +5 -0
  70. package/eval-framework/cases/case5-zh-tier3.json +5 -0
  71. package/eval-framework/mechanism-test.js +251 -0
  72. package/eval-framework/runner.js +56 -0
  73. package/eval-harness/SKILL.md +59 -0
  74. package/eval-harness/scripts/evaluate.js +61 -0
  75. package/fable-discipline/SKILL.md +38 -0
  76. package/fable-mode/CONTRACT-FORMAT.md +51 -0
  77. package/fable-mode/SKILL.md +51 -0
  78. package/fable-mode/agents/fable-orchestrator.md +92 -0
  79. package/fable-mode/agents/fable-verifier.md +32 -0
  80. package/fable-mode/agents/fable-worker-haiku.md +48 -0
  81. package/fable-mode/agents/fable-worker-sonnet.md +49 -0
  82. package/fable-mode/execution-guardrails/SKILL.md +87 -0
  83. package/fable-mode/fable-haiku/SKILL.md +48 -0
  84. package/git-commit/SKILL.md +55 -0
  85. package/git-commit/guides/ANGULAR_STYLE.md +61 -0
  86. package/git-commit/guides/COMMIT_GENERATION.md +24 -0
  87. package/git-commit/guides/LANGUAGE_DETECTION.md +16 -0
  88. package/git-commit/guides/MAIN_REPO.md +15 -0
  89. package/git-commit/guides/SUBMODULES.md +27 -0
  90. package/grill-me/SKILL.md +36 -0
  91. package/grill-with-docs/ADR-FORMAT.md +47 -0
  92. package/grill-with-docs/CONTEXT-FORMAT.md +63 -0
  93. package/grill-with-docs/SKILL.md +99 -0
  94. package/harness-everything/SKILL.md +141 -0
  95. package/harness-everything/examples/README.md +9 -0
  96. package/harness-everything/examples/todo-workflow.md +22 -0
  97. package/harness-everything/scripts/bootstrap.js +140 -0
  98. package/harness-everything/scripts/self-heal.js +189 -0
  99. package/harness-everything/scripts/tier-router.js +219 -0
  100. package/harness-everything/scripts/todo-cli.js +157 -0
  101. package/harness-everything/scripts/verify-gate.js +31 -0
  102. package/hooks/hooks.json +155 -0
  103. package/hooks/scripts/atomic-commit-check.js +68 -0
  104. package/hooks/scripts/boundary-guard.js +55 -0
  105. package/hooks/scripts/context-compact.js +114 -0
  106. package/hooks/scripts/contract-test.js +79 -0
  107. package/hooks/scripts/depth-guard.js +65 -0
  108. package/hooks/scripts/lib/harness-state.js +202 -0
  109. package/hooks/scripts/lib/platforms/claude.js +42 -0
  110. package/hooks/scripts/lib/platforms/codex.js +46 -0
  111. package/hooks/scripts/lib/platforms/continue.js +41 -0
  112. package/hooks/scripts/lib/platforms/copilot.js +41 -0
  113. package/hooks/scripts/lib/platforms/cursor.js +41 -0
  114. package/hooks/scripts/lib/platforms/hermes.js +32 -0
  115. package/hooks/scripts/lib/platforms/index.js +17 -0
  116. package/hooks/scripts/lib/platforms/worktrees.js +20 -0
  117. package/hooks/scripts/reset-circuit-breaker.js +30 -0
  118. package/hooks/scripts/rule-of-3-tracker.js +91 -0
  119. package/hooks/scripts/rule-of-3.js +176 -0
  120. package/hooks/scripts/state-persist.js +99 -0
  121. package/hooks/scripts/stop-gate.js +80 -0
  122. package/hooks/scripts/subagent-scope-guard.js +88 -0
  123. package/improve-codebase-architecture/SKILL.md +44 -0
  124. package/improve-codebase-architecture/guides/DEEPENING.md +37 -0
  125. package/improve-codebase-architecture/guides/HTML-REPORT.md +123 -0
  126. package/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -0
  127. package/improve-codebase-architecture/guides/LANGUAGE.md +53 -0
  128. package/install-cognitive-os/SKILL.md +95 -0
  129. package/package.json +22 -0
  130. package/repo-docs/SKILL.md +45 -0
  131. package/repo-docs/templates/agents-template.md +264 -0
  132. package/repo-docs/templates/knowledge-agents-template.md +50 -0
  133. package/repo-docs/templates/knowledge-readme-template.md +83 -0
  134. package/repo-docs/templates/manual-recon.md +117 -0
  135. package/repo-docs/templates/multi-skills-readme-template.md +72 -0
  136. package/repo-docs/templates/product-agents-template.md +78 -0
  137. package/repo-docs/templates/product-readme-template.md +95 -0
  138. package/repo-docs/templates/readme-template.md +136 -0
  139. package/repo-docs/templates/single-skill-readme-template.md +70 -0
  140. package/repo-docs/templates/skills-agents-template.md +60 -0
  141. package/rewrite-commits/SKILL.md +32 -0
  142. package/scripts/installer.js +649 -0
  143. package/scripts/lib/advisory-text.js +163 -0
  144. package/scripts/lib/claude-hooks.js +71 -0
  145. package/scripts/lib/gitignore.js +58 -0
  146. package/scripts/lib/manifest.js +82 -0
  147. package/scripts/lib/prompts.js +241 -0
  148. package/scripts/lib/skills.js +172 -0
  149. package/scripts/lib/workspace.js +28 -0
  150. package/security-review/SKILL.md +511 -0
  151. package/self-evolve/SKILL.md +54 -0
  152. package/self-evolve/scripts/persist-memory.js +40 -0
  153. package/self-evolve/scripts/self-regression.js +113 -0
  154. package/skill-creator/SKILL.md +91 -0
  155. package/skill-creator/references/quality-principles.md +81 -0
  156. package/skill-creator/references/testing-workflow.md +32 -0
  157. package/skill-style/SKILL.md +48 -0
  158. package/skill-style/STYLE.md +37 -0
  159. package/tdd/SKILL.md +53 -0
  160. package/tdd/guides/deep-modules.md +33 -0
  161. package/tdd/guides/interface-design.md +31 -0
  162. package/tdd/guides/mocking.md +59 -0
  163. package/tdd/guides/refactoring.md +10 -0
  164. package/tdd/guides/tests.md +61 -0
  165. package/todo-driven-workflow/SKILL.md +54 -0
  166. package/using-git-worktrees/SKILL.md +161 -0
  167. package/verification-loop/SKILL.md +138 -0
  168. package/verify-before-claim/SKILL.md +86 -0
  169. package/zoom-out/SKILL.md +90 -0
@@ -0,0 +1,251 @@
1
+ #!/usr/bin/env node
2
+ /**
3
+ * Harness Mechanism Test Suite (Claude Code hook contract)
4
+ * Automates VERIFICATION.md Section 2 (mechanism check) so it runs on every
5
+ * `npm test` instead of only when a human copy-pastes the recipes by hand.
6
+ * That manual gap is exactly what let three independent 2026-07-23 audit
7
+ * reports (see docs/reports/) misdiagnose working hooks as broken: they
8
+ * piped JSON via `echo '...' | node script.js` in a Windows Git Bash shell,
9
+ * which mangles stdin/TTY state and produced false negatives. This suite
10
+ * uses execSync's `input` option instead (the same technique already proven
11
+ * reliable in runner.js and behavioral-test.js) so it is a faithful,
12
+ * cross-platform re-run of the exact payloads VERIFICATION.md documents.
13
+ *
14
+ * Each hook resolves its state directory via detectActivePlatform(), which
15
+ * defaults to 'copilot' under a VS Code integrated terminal (TERM_PROGRAM=
16
+ * vscode) unless CLAUDE is set - so every child process below is spawned
17
+ * with CLAUDE=1 to force the Claude Code state path regardless of the host
18
+ * running this test.
19
+ *
20
+ * All state is written under a dedicated, never-real session id so this
21
+ * suite can never collide with or clobber an actual Claude Code session's
22
+ * breaker/handoff state. The session directory (and any repo-root temp
23
+ * files created along the way) is removed on exit, success or failure alike.
24
+ */
25
+ const { execSync } = require('child_process');
26
+ const fs = require('fs');
27
+ const path = require('path');
28
+
29
+ process.env.CLAUDE = '1'; // so our own getSessionDir() calls resolve like the children's will
30
+
31
+ const { getWorkspaceRoot, getSessionDir } = require('../hooks/scripts/lib/harness-state');
32
+
33
+ const root = getWorkspaceRoot();
34
+ const hooksDir = path.join(root, 'hooks', 'scripts');
35
+ const SESSION_ID = '__mechanism_test__';
36
+ const sessionDir = getSessionDir(root, SESSION_ID);
37
+
38
+ console.log('=== 🔧 Harness Mechanism Test Suite (VERIFICATION.md §2) ===');
39
+
40
+ const tempFiles = [];
41
+ let failures = 0;
42
+
43
+ function check(name, condition, detail) {
44
+ if (condition) {
45
+ console.log(`✅ ${name}`);
46
+ } else {
47
+ console.error(`❌ ${name}`);
48
+ if (detail) console.error(` ${detail}`);
49
+ failures++;
50
+ }
51
+ }
52
+
53
+ function runHook(scriptName, payload, extraEnv) {
54
+ const scriptPath = path.join(hooksDir, scriptName);
55
+ try {
56
+ const stdout = execSync(`node "${scriptPath}"`, {
57
+ encoding: 'utf8',
58
+ input: payload === null ? '' : JSON.stringify(payload),
59
+ env: { ...process.env, ...extraEnv },
60
+ });
61
+ return { code: 0, stdout, stderr: '' };
62
+ } catch (err) {
63
+ return { code: err.status, stdout: err.stdout || '', stderr: err.stderr || '' };
64
+ }
65
+ }
66
+
67
+ function writeState(file, obj) {
68
+ fs.writeFileSync(path.join(sessionDir, file), JSON.stringify(obj, null, 2), 'utf8');
69
+ }
70
+ function readState(file) {
71
+ return JSON.parse(fs.readFileSync(path.join(sessionDir, file), 'utf8'));
72
+ }
73
+ function stateExists(file) {
74
+ return fs.existsSync(path.join(sessionDir, file));
75
+ }
76
+
77
+ // Registers a repo-root temp file for guaranteed cleanup even if an
78
+ // assertion throws or the process is killed mid-suite.
79
+ function tempFile(name) {
80
+ const p = path.join(root, name);
81
+ tempFiles.push(p);
82
+ return p;
83
+ }
84
+
85
+ process.on('exit', () => {
86
+ for (const f of tempFiles) {
87
+ try { if (fs.existsSync(f)) fs.unlinkSync(f); } catch (e) { /* best-effort */ }
88
+ }
89
+ try { fs.rmSync(sessionDir, { recursive: true, force: true }); } catch (e) { /* best-effort */ }
90
+ });
91
+
92
+ // ---------------------------------------------------------------------------
93
+ // 2a. Rule of 3 circuit breaker trips, releases via zoom-out report, and
94
+ // hard-locks on a repeat trip of the same signature.
95
+ // ---------------------------------------------------------------------------
96
+ console.log('\n[2a] Rule of 3 circuit breaker...');
97
+
98
+ writeState('rule-of-3-state.json', { count: 3, lastHash: 'mech-test', zoomOutResolved: false });
99
+ const tripResult = runHook('rule-of-3.js', { session_id: SESSION_ID });
100
+ check(
101
+ '2a. Trips at count=3 (exit=2, CRITICAL banner)',
102
+ tripResult.code === 2 && tripResult.stderr.includes('RULE OF 3 CIRCUIT BREAKER TRIGGERED'),
103
+ `Got exit=${tripResult.code}, stderr="${tripResult.stderr.slice(0, 200)}"`
104
+ );
105
+
106
+ writeState('rule-of-3-state.json', {
107
+ count: 3, lastHash: 'mech-test', zoomOutResolved: false, lastFailureAt: 0, zoomOutCycles: 0,
108
+ });
109
+ fs.writeFileSync(
110
+ path.join(sessionDir, 'zoom-out-report.md'),
111
+ '## Goal\nx\n## Failed Attempts\nx\n## Verified Facts\nx\n## Diagnosis\nx\n## Decision\nRESUME: new approach\n',
112
+ 'utf8'
113
+ );
114
+ const releaseResult = runHook('rule-of-3.js', { session_id: SESSION_ID });
115
+ const releasedState = readState('rule-of-3-state.json');
116
+ check(
117
+ '2a-bis. Valid zoom-out report releases the breaker (exit=0, count reset)',
118
+ releaseResult.code === 0 &&
119
+ releaseResult.stdout.includes('breaker released') &&
120
+ releasedState.count === 0 &&
121
+ releasedState.zoomOutResolved === true &&
122
+ releasedState.zoomOutCycles === 1,
123
+ `Got exit=${releaseResult.code}, state=${JSON.stringify(releasedState)}`
124
+ );
125
+
126
+ writeState('rule-of-3-state.json', {
127
+ count: 3, lastHash: 'mech-test', zoomOutResolved: false,
128
+ lastFailureAt: Date.now() + 60000, zoomOutCycles: 1,
129
+ });
130
+ const hardLockResult = runHook('rule-of-3.js', { session_id: SESSION_ID });
131
+ check(
132
+ '2a-ter. Second trip on same signature hard-locks (exit=2, repeat trip)',
133
+ hardLockResult.code === 2 && hardLockResult.stderr.includes('repeat trip - hard lock'),
134
+ `Got exit=${hardLockResult.code}, stderr="${hardLockResult.stderr.slice(0, 200)}"`
135
+ );
136
+
137
+ // ---------------------------------------------------------------------------
138
+ // 2b. Boundary guard blocks an oversized Read
139
+ // ---------------------------------------------------------------------------
140
+ console.log('\n[2b] Boundary guard...');
141
+
142
+ const bigFile = tempFile('.mechanism-test-big.tmp');
143
+ fs.writeFileSync(bigFile, 'x'.repeat(600 * 1024));
144
+ const boundaryResult = runHook('boundary-guard.js', { tool_name: 'Read', tool_input: { file_path: bigFile } });
145
+ check(
146
+ '2b. Blocks a 600KB Read without limit/offset (exit=2, BLOCKED)',
147
+ boundaryResult.code === 2 && boundaryResult.stderr.includes('[Boundary Guard] BLOCKED'),
148
+ `Got exit=${boundaryResult.code}, stderr="${boundaryResult.stderr.slice(0, 200)}"`
149
+ );
150
+ fs.unlinkSync(bigFile);
151
+
152
+ // ---------------------------------------------------------------------------
153
+ // 2c. State persistence (WAL) records a failure, then clears it on success
154
+ // ---------------------------------------------------------------------------
155
+ console.log('\n[2c] State persistence (WAL)...');
156
+
157
+ runHook('state-persist.js', {
158
+ session_id: SESSION_ID,
159
+ tool_name: 'Bash',
160
+ tool_response: { stdout: '', stderr: 'npm ERR! mech-test failure' },
161
+ });
162
+ const failedHandoff = readState('handoff-state.json');
163
+ check(
164
+ '2c. Failing Bash call is recorded as status=failed',
165
+ failedHandoff.status === 'failed' && failedHandoff.errorSummary.includes('mech-test failure'),
166
+ `Got ${JSON.stringify(failedHandoff)}`
167
+ );
168
+
169
+ runHook('state-persist.js', {
170
+ session_id: SESSION_ID,
171
+ tool_name: 'Bash',
172
+ tool_response: { stdout: 'ok', exitCode: 0 },
173
+ });
174
+ const resolvedHandoff = readState('handoff-state.json');
175
+ check(
176
+ '2c. A subsequent successful command clears status back to idle',
177
+ resolvedHandoff.status === 'idle' && !!resolvedHandoff.lastResolved,
178
+ `Got ${JSON.stringify(resolvedHandoff)}`
179
+ );
180
+
181
+ // ---------------------------------------------------------------------------
182
+ // 2d. Fact-audit reminder reaches the agent via tier-router.js stdin
183
+ // ---------------------------------------------------------------------------
184
+ console.log('\n[2d] Fact-audit reminder...');
185
+
186
+ const tierRouterPath = path.join(root, 'harness-everything', 'scripts', 'tier-router.js');
187
+ const factAuditOut = execSync(`node "${tierRouterPath}"`, {
188
+ encoding: 'utf8',
189
+ input: JSON.stringify({ prompt: 'what exit code does this hook use by default and is it documented' }),
190
+ });
191
+ check(
192
+ '2d. tier-router.js reads {"prompt":...} from stdin and emits FACT-AUDIT REMINDER',
193
+ factAuditOut.includes('FACT-AUDIT REMINDER'),
194
+ `Output was:\n${factAuditOut.slice(0, 300)}`
195
+ );
196
+
197
+ // ---------------------------------------------------------------------------
198
+ // 2e. Subagent scope guard catches an out-of-scope file change
199
+ // ---------------------------------------------------------------------------
200
+ console.log('\n[2e] Subagent scope guard...');
201
+
202
+ runHook('subagent-scope-guard.js', {
203
+ session_id: SESSION_ID, tool_name: 'Task', hook_event_name: 'PreToolUse', tool_input: {},
204
+ });
205
+ const scopeFile = tempFile('.mechanism-test-scope.tmp');
206
+ fs.writeFileSync(scopeFile, 'unexpected change');
207
+ const scopeResult = runHook('subagent-scope-guard.js', {
208
+ session_id: SESSION_ID, tool_name: 'Task', hook_event_name: 'PostToolUse', tool_input: {},
209
+ });
210
+ check(
211
+ '2e. Flags the newly-created file after a Task burst (exit=2, lists filename)',
212
+ scopeResult.code === 2 &&
213
+ scopeResult.stderr.includes('Subagent Scope Guard') &&
214
+ scopeResult.stderr.includes(path.basename(scopeFile)),
215
+ `Got exit=${scopeResult.code}, stderr="${scopeResult.stderr.slice(0, 300)}"`
216
+ );
217
+ fs.unlinkSync(scopeFile);
218
+
219
+ // ---------------------------------------------------------------------------
220
+ // 2f. Stop gate bounces an unverified-edit stop exactly once
221
+ // ---------------------------------------------------------------------------
222
+ console.log('\n[2f] Stop gate...');
223
+
224
+ writeState('handoff-state.json', { status: 'idle', lastEditAt: Date.now(), lastVerifyAt: 0 });
225
+ const dirtyFile = tempFile('.mechanism-test-dirty.tmp');
226
+ fs.writeFileSync(dirtyFile, 'dirty');
227
+ const firstStop = runHook('stop-gate.js', { session_id: SESSION_ID });
228
+ check(
229
+ '2f. First stop after an unverified edit bounces (exit=2)',
230
+ firstStop.code === 2 && firstStop.stderr.includes('[Stop Gate]'),
231
+ `Got exit=${firstStop.code}, stderr="${firstStop.stderr.slice(0, 200)}"`
232
+ );
233
+ const secondStop = runHook('stop-gate.js', { session_id: SESSION_ID });
234
+ check(
235
+ '2f. Same edit batch does not bounce twice (exit=0)',
236
+ secondStop.code === 0,
237
+ `Got exit=${secondStop.code}`
238
+ );
239
+ fs.unlinkSync(dirtyFile);
240
+
241
+ // ---------------------------------------------------------------------------
242
+ console.log('\n=================================================');
243
+ if (failures > 0) {
244
+ console.error(` ❌ ${failures} mechanism check(s) FAILED.`);
245
+ console.log('=================================================');
246
+ process.exit(1);
247
+ } else {
248
+ console.log(' 🎉 All VERIFICATION.md §2 mechanism checks passed.');
249
+ console.log('=================================================');
250
+ process.exit(0);
251
+ }
@@ -0,0 +1,56 @@
1
+ #!/usr/bin/env node
2
+ const fs = require('fs');
3
+ const path = require('path');
4
+ const { execSync } = require('child_process');
5
+
6
+ const casesDir = path.join(__dirname, 'cases');
7
+ const files = fs.readdirSync(casesDir).filter(f => f.endsWith('.json'));
8
+
9
+ console.log("=== Harness Tier Verification Framework ===");
10
+ console.log("This framework simulates routing prompts to verify trigger rates.\n");
11
+
12
+ let passed = 0;
13
+ let failed = 0;
14
+
15
+ files.forEach(file => {
16
+ const data = JSON.parse(fs.readFileSync(path.join(casesDir, file), 'utf8'));
17
+ console.log(`[Testing] ${data.name}`);
18
+ console.log(` Prompt: "${data.prompt}"`);
19
+ console.log(` Expected Tier: ${data.expected_tier}`);
20
+
21
+ try {
22
+ const tierRouterPath = path.join(__dirname, '..', 'harness-everything', 'scripts', 'tier-router.js');
23
+ // Claude Code feeds UserPromptSubmit hooks JSON on stdin (a "prompt"
24
+ // field), not a CLI argument - match that here so this test actually
25
+ // exercises the real interface instead of a path tier-router.js never
26
+ // sees in production.
27
+ const output = execSync(`node "${tierRouterPath}"`, {
28
+ encoding: 'utf8',
29
+ input: JSON.stringify({ prompt: data.prompt }),
30
+ env: { ...process.env, HARNESS_EVAL: 'true' }
31
+ });
32
+
33
+ // Parse the output to find the recommended tier (accept the old
34
+ // "REQUIRED TIER" label too so the runner works across versions)
35
+ const tierMatch = output.match(/(?:RECOMMENDED|REQUIRED) TIER: (Tier \d+)/i) || output.match(/(Tier \d+)/i);
36
+ const actualTier = tierMatch ? tierMatch[1] : "Unknown";
37
+
38
+ if (actualTier.includes(data.expected_tier)) {
39
+ console.log(` -> Result: PASS (Got ${actualTier})\n`);
40
+ passed++;
41
+ } else {
42
+ console.log(` -> Result: FAIL (Expected ${data.expected_tier}, Got ${actualTier})\n`);
43
+ console.log(` Output was:\n${output}`);
44
+ failed++;
45
+ }
46
+ } catch (err) {
47
+ console.log(` -> Result: ERROR executing router`);
48
+ failed++;
49
+ }
50
+ });
51
+
52
+ console.log(`--- Summary: ${passed} passed, ${failed} failed ---`);
53
+ if (failed > 0) {
54
+ process.exit(1);
55
+ }
56
+
@@ -0,0 +1,59 @@
1
+ ---
2
+ name: eval-harness
3
+ description: Objectively evaluates AI agent performance based on correctness, resource efficiency, and anti-loop focus.
4
+ author: Miya Daniel | Harness Core Team
5
+ version: 0.2.0
6
+ ---
7
+
8
+ # Eval Harness (Automated Performance & Reasoning Evaluation)
9
+
10
+ ## 📋 Skill Contract
11
+
12
+ | Component | Specification |
13
+ | :--- | :--- |
14
+ | **Trigger / Input** | User requests "run benchmark", "score conversation log", or "compare Vanilla vs Harness differences" — an execution log or the current conversation history. |
15
+ | **Expected Output** | A 4-dimension scorecard (Correctness & Factuality, Token & Step Efficiency, Anti-loop & Focus, Environment & Tool Awareness), each scored 0-10 with rationale. |
16
+ | **State Mutations** | Writes a generated Markdown scorecard to the project's `evals/` directory via `scripts/evaluate.js`. |
17
+ | **Enforcement Gate** | MUST execute `scripts/evaluate.js` with the four scores and insight text — a verbally-stated score without running the script does not count as output. |
18
+
19
+ This skill is used to objectively evaluate the performance of an AI Agent when "executing test tasks".
20
+ Triggered when the user requests "run benchmark", "score conversation log", or "compare Vanilla vs Harness differences".
21
+
22
+ ## 1. Data Parsing `[Discover]`
23
+ - Read the execution logs provided by the user or directly analyze the current conversation history.
24
+ - Extract key metrics:
25
+ - **Action Count**: Includes the number of file reads, searches, and terminal executions.
26
+ - **Error Loop Count**: The number of times the same error message appeared consecutively triggering attempted micro-adjustments.
27
+ - **Token / Time Indicator**: Record token consumption or time spent if provided by the user.
28
+
29
+ ## 2. Scoring Rubric `[Think]`
30
+ Please score based on the following four dimensions (out of 10 points each):
31
+
32
+ ### A. Correctness & Factuality
33
+ - 10 points: Perfectly solved the problem with no side effects. Clearly knows its own knowledge boundaries, **no hallucinations or fabricated facts**.
34
+ - 5 points: Solved the surface problem but introduced new issues, or provided slightly outdated information.
35
+ - 0 points: Task failed, broke original code, or "hallucinated with straight-faced confidence".
36
+
37
+ ### B. Token & Step Efficiency
38
+ - 10 points: Extremely streamlined, no superfluous exploration or useless tool calls (prevented over-engineering).
39
+ - 5 points: Reasonable steps, but with minor unnecessary file reads.
40
+ - 0 points: Over-engineered, writing lengthy plans or repeatedly reading irrelevant files for a simple task.
41
+
42
+ ### C. Anti-loop & Focus
43
+ - 10 points: Decisively triggered `zoom-out` when hitting a bottleneck, rebuilt the full picture, and either self-recovered on a fresh diagnosis or escalated a genuine decision with options; did not hit a dead end or dump an "I'm stuck" on the human.
44
+ - 5 points: Stopped after 3~5 attempts.
45
+ - 0 points: Fell into a Micro-error looping cycle more than 5 times, or completely forgot the initial goal.
46
+
47
+ ### D. Environment & Tool Awareness
48
+ - 10 points: Accurately detected OS, terminal shell (e.g., Git Bash vs PowerShell), and available tools in the `[Discover]` phase before acting. No syntax errors caused by wrong shell assumptions.
49
+ - 5 points: Made an initial incorrect assumption about the terminal/tools but corrected it immediately after the first failure.
50
+ - 0 points: Blindly threw commands (e.g., PowerShell syntax in Git Bash) and repeatedly failed without realizing the environment mismatch.
51
+
52
+ ## 3. Output Report `[Summarize]`
53
+ - **MANDATORY**: You MUST execute the evaluation script to generate and save the report.
54
+ - Find the `evaluate.js` script in the `scripts/` directory of this skill and run it via terminal:
55
+ ```bash
56
+ node path/to/eval-harness/scripts/evaluate.js <scoreA> <scoreB> <scoreC> <scoreD> "<Your insights here>"
57
+ ```
58
+ *(Example: `node scripts/evaluate.js 10 5 10 10 "The agent successfully used zoom-out..."`)*
59
+ - The script will automatically generate the Markdown-formatted **Scorecard Table** and save it to the project's `evals/` directory.
@@ -0,0 +1,61 @@
1
+ #!/usr/bin/env node
2
+ const fs = require('fs');
3
+ const path = require('path');
4
+
5
+ const args = process.argv.slice(2);
6
+ if (args.length < 5) {
7
+ console.error("Usage: node evaluate.js <scoreA> <scoreB> <scoreC> <scoreD> \"<insights>\"");
8
+ console.error("Scores should be 0, 5, or 10.");
9
+ process.exit(1);
10
+ }
11
+
12
+ const [scoreA, scoreB, scoreC, scoreD, ...insightParts] = args;
13
+ const insights = insightParts.join(" ");
14
+
15
+ const scores = {
16
+ "Correctness & Factuality": parseInt(scoreA, 10),
17
+ "Token & Step Efficiency": parseInt(scoreB, 10),
18
+ "Anti-loop & Focus": parseInt(scoreC, 10),
19
+ "Environment & Tool Awareness": parseInt(scoreD, 10)
20
+ };
21
+
22
+ let total = 0;
23
+ let markdown = `## Harness Evaluation Report - ${new Date().toISOString()}\n\n| Dimension | Score |\n|---|---|\n`;
24
+
25
+ for (const [dim, score] of Object.entries(scores)) {
26
+ if (isNaN(score)) {
27
+ console.error(`Error: Score for ${dim} is not a valid number.`);
28
+ process.exit(1);
29
+ }
30
+ markdown += `| ${dim} | ${score}/10 |\n`;
31
+ total += score;
32
+ }
33
+
34
+ markdown += `| **Total Score** | **${total}/40** |\n\n`;
35
+ markdown += `### Insights\n${insights}\n\n---\n`;
36
+
37
+ function getWorkspaceRoot() {
38
+ let dir = path.resolve(process.cwd());
39
+ while (dir !== path.parse(dir).root) {
40
+ if (fs.existsSync(path.join(dir, '.git'))) return dir;
41
+ dir = path.dirname(dir);
42
+ }
43
+ return process.cwd();
44
+ }
45
+
46
+ // Store the evaluations in an evals folder in the project root
47
+ const evalDir = path.join(getWorkspaceRoot(), 'evals');
48
+ if (!fs.existsSync(evalDir)) {
49
+ fs.mkdirSync(evalDir, { recursive: true });
50
+ }
51
+
52
+ const reportPath = path.join(evalDir, 'evaluation-reports.md');
53
+ try {
54
+ fs.appendFileSync(reportPath, markdown, 'utf8');
55
+ console.log(`[Success] Evaluation report generated and saved to ${reportPath}`);
56
+ console.log('\n--- Report Preview ---\n');
57
+ console.log(markdown);
58
+ } catch (err) {
59
+ console.error(`[Error] Failed to write evaluation report: ${err.message}`);
60
+ process.exit(1);
61
+ }
@@ -0,0 +1,38 @@
1
+ ---
2
+ name: fable-discipline
3
+ description: Shadow guard for fable-mode to prevent context bloat and maintain physical boundaries.
4
+ author: Miya Daniel | Harness Core Team
5
+ version: 0.2.0
6
+ ---
7
+
8
+ # Fable Discipline (Macro Task Discipline & Safety Net)
9
+
10
+ ## 📋 Skill Contract
11
+
12
+ | Component | Specification |
13
+ | :--- | :--- |
14
+ | **Trigger / Input** | Runs continuously in the background for the entire duration `fable-mode` is active — not a standalone, one-shot trigger. |
15
+ | **Expected Output** | Periodic state compaction after each milestone; one Git commit per independently functioning logic block; a state manifest handed off between sub-agents at every handoff point. |
16
+ | **State Mutations** | None of its own — it constrains how `fable-mode` and `create-agent-launcher` mutate state (commit cadence, handoff manifests, read-size limits). |
17
+ | **Enforcement Gate** | **HALTS execution immediately** (not a retry) if build errors show a divergent trend (more fixes → more breakage), and forces `zoom-out` instead of continued patching. |
18
+
19
+ This skill acts as the shadow guard for `fable-mode`. As long as `fable-mode` is active, this skill MUST run in the background to prevent large tasks from spiraling out of control.
20
+
21
+ ## 1. Anti-Context Bloat Defense
22
+ When executing large architectural tasks, conversation logs expand rapidly, leading to model attention loss (hallucinations or forgetting the original intent).
23
+ - **Strategic Compact**: After completing each core milestone, forcefully summarize the current state and decisions, discarding unnecessary past conversation details.
24
+ - **Avoid Broad Reads**: Prohibited from using broad Regex searches without precise conditions or reading irrelevant files over 1000 lines.
25
+
26
+ ## 2. Strict Physical Boundaries
27
+ - **Environment Isolation**: Before modifying any core architecture, ensure you are clearly aware of the Current Working Directory (CWD).
28
+ - **Atomic Commits**: You are NOT allowed to modify dozens of files at once before testing. For every independently functioning logic block completed, advise the human to create a Git Commit as a safe save point.
29
+
30
+ ## 3. Agent Handoff Protocol
31
+ When `fable-mode` spawns and switches between different sub-agents via `create-agent-launcher`, strict handoff discipline must be observed:
32
+ - **State Manifest**: The previous agent MUST leave a clear state record (e.g., what APIs were completed, expected inputs/outputs).
33
+ - **Contract Testing**: The very first step for the succeeding agent is to verify if the state manifest left by the previous agent is correct.
34
+
35
+ ## 4. Ultimate Circuit Breaker: Indefinite Halt
36
+ - Large tasks easily generate the sunk cost fallacy of "I think it's almost fixed, let me try one more time."
37
+ - If the system's Build Errors exhibit a divergent trend (the more you fix, the more it breaks), **HALT EXECUTION IMMEDIATELY**.
38
+ - Call `zoom-out`, organize the error dependency graph for the human, and seek architectural-level refactoring advice.
@@ -0,0 +1,51 @@
1
+ # Stage Contract Format
2
+
3
+ Stage contracts live in `.claude/harness-state/contracts/` and make fable-orchestrator's
4
+ "re-run or spot-check every stage's named check" discipline auditable instead
5
+ of a verbal claim. The `contract-test.js` hook (PostToolUse: Bash) watches for
6
+ the exact `checkCommand` running and fills in the result — you write the
7
+ contract, the hook resolves it.
8
+
9
+ ## When to write one
10
+
11
+ Right before you (the orchestrator) run the Bash command that re-verifies a
12
+ stage's pass condition — one contract per stage, filed the moment you're about
13
+ to spot-check it. Not for every Bash call, only the ones that *are* the named
14
+ check.
15
+
16
+ ## File
17
+
18
+ `.claude/harness-state/contracts/<stageId>.json`, created lazily. `stageId` is whatever you
19
+ numbered the stage in your stage map (`stage-3`, `db-migration`, etc).
20
+
21
+ ## Schema
22
+
23
+ ```json
24
+ {
25
+ "stageId": "stage-3",
26
+ "agent": "fable-worker-sonnet",
27
+ "task": "one-line description of what the worker was assigned",
28
+ "outputPath": "path to the artifact the worker produced",
29
+ "checkCommand": "the exact shell command you are about to run to verify it",
30
+ "status": "pending"
31
+ }
32
+ ```
33
+
34
+ Write it with `status: "pending"` and the *exact* `checkCommand` string you're
35
+ about to run — the hook matches verbatim, not fuzzily. Then run that command.
36
+ The hook fills in `status` (`pass`/`fail`), `evidence` (tail of the command's
37
+ output), and `verifiedAt` for you; you don't write those fields yourself.
38
+
39
+ ## Reading contracts back
40
+
41
+ Before delivering, `Read` (or `Grep status` across) `.claude/harness-state/contracts/*.json`.
42
+ Any file still `"status": "pending"` means the named check was declared but
43
+ never actually run — that stage's output hasn't been verified, whatever the
44
+ worker's report claimed. Resolve it before building further on that stage.
45
+
46
+ ## Not a blocking gate
47
+
48
+ The hook can only react after a Bash call finishes — it can't stop a bad stage
49
+ from being built on in real time. It surfaces a failed check loudly (fed back
50
+ to you immediately after the command runs), but the actual discipline is still
51
+ yours: don't build on a stage whose contract is `pending` or `fail`.
@@ -0,0 +1,51 @@
1
+ ---
2
+ name: fable-mode
3
+ description: Macro task planning and execution engine for complex requirements and low-level architecture changes.
4
+ author: Miya Daniel | Harness Core Team
5
+ version: 0.2.0
6
+ ---
7
+
8
+ # Fable Mode (Macro Task Planning & Execution Engine)
9
+
10
+ ## 📋 Skill Contract
11
+
12
+ | Component | Specification |
13
+ | :--- | :--- |
14
+ | **Trigger / Input** | Tier 3 Task classification. Input: Massive architectural requirement or module creation. |
15
+ | **Expected Output** | 1. Documented architectural plan. 2. A sequence of subagent handoffs (`create-agent-launcher`). 3. System integration verified by terminal scripts. |
16
+ | **State Mutations** | 1. Generates `todo-driven-workflow` checklist (via CLI) tracking subagent milestones. 2. `subagent-scope-state.json` updates during handoffs. |
17
+ | **Enforcement Gate** | You MUST use `node harness-everything/scripts/todo-cli.js` to track milestones. You MUST run system tests (`verify-gate.js` or equivalent) after EACH subagent returns before moving to the next. |
18
+
19
+ This skill is automatically triggered and loaded by the `harness-everything` router when a task is judged as a **Tier 3 (Macro Task)**.
20
+ It applies to building new feature modules from scratch, massive low-level architecture changes, or extremely vague and complex user requirements.
21
+
22
+ ## Core Concept: Plan Before Acting
23
+
24
+ In Fable Mode, you are no longer a simple "typist", but an "Architect" and "Project Manager".
25
+ You MUST strictly comply with and pair this with `fable-discipline` to control risks.
26
+
27
+ ## Sub-Skills (bundled with this skill)
28
+ - `execution-guardrails/` — always-on operational rules (verify-before-flag, warning batching, find-and-replace safety). These apply on EVERY tier and every model, not just inside Fable Mode; this skill's self-critique and file edits inherit them.
29
+ - `fable-haiku/` — opt-in variant that routes the staged loop to a cheap Haiku worker agent for bulk mechanical work. Only use when the user explicitly asks for it.
30
+
31
+ ## Execution Phases:
32
+
33
+ ### 1. Discovery & Planning
34
+ - **Assess Current State**: Comprehensively review the file architecture related to the requirement and understand dependencies.
35
+ - **Deconstruct Task**: Break down the massive macro task into independently verifiable sub-tasks.
36
+ - **Produce Plan**: Write a clear implementation plan (can be stored in chat or a temporary file) and confirm it with the Human Partner.
37
+ - **Materialize the Plan as a Checklist**: The confirmed plan MUST be initialized as the `todo-driven-workflow` checklist — every sub-task, sub-agent handoff, and milestone check is tracked there, not in prose. This is the harness's base execution loop.
38
+
39
+ ### 2. Sub-agent Orchestration
40
+ - If the task involves different professional domains (e.g., modifying Postgres schema and writing React UI), it is **PROHIBITED** for a single model context to handle everything from start to finish, as this leads to attention scatter and Token waste.
41
+ - **Mandatory Call**: Invoke the `create-agent-launcher` skill.
42
+ - Based on the deconstructed sub-tasks, configure the corresponding Sub-agents (e.g., spawn a Database Expert to handle the Schema, then hand off to a Frontend Expert for the UI).
43
+
44
+ ### 3. Monitoring & Integration
45
+ - As the chief commander, Fable Mode is responsible for monitoring the progress of each Sub-agent.
46
+ - Ensure the outputs of various sub-agents can integrate perfectly.
47
+ - Upon completing each Milestone, a global test or check MUST be executed.
48
+
49
+ ## Handoff & Defense
50
+ - Fable Mode consumes massive system resources and Context Window.
51
+ - Once the core module scaffolding is built and the task downgrades to Tier 2 feature completion, you MUST **proactively exit Fable Mode** and hand off to the `tdd` mode for subsequent detailed implementation.
@@ -0,0 +1,92 @@
1
+ ---
2
+ name: fable-orchestrator
3
+ description: Staged-execution orchestrator for large, multi-part, or multi-session tasks. Use when fable-mode discipline must run with enforced delegation — it writes the stage map, delegates ALL artifact production to fable-worker-sonnet / fable-worker-haiku, verifies every stage with a failable check, and sends high-stakes deliverables to fable-verifier for a cold re-check. It has no Write or Edit tool, so it cannot do the work itself.
4
+ tools: Read, Grep, Glob, Bash, Task, TodoWrite
5
+ model: opus
6
+ ---
7
+
8
+ You are the fable orchestrator. You coordinate; you do not produce. You have no
9
+ Write or Edit tool by design — every artifact must come from a worker agent. Your
10
+ Bash access is for read-only inspection and running verification commands (tests,
11
+ greps, diffs) ONLY. Never create or modify a file through Bash redirection,
12
+ heredocs, tee, sed -i, or any other side channel — that defeats the reason Write
13
+ was removed. If you catch yourself about to produce content, stop and delegate.
14
+
15
+ ## Core loop
16
+
17
+ **1. Stage map (before touching anything).** Write the full stage plan first.
18
+ Number stages; give each a brief expected output. Each stage produces one
19
+ verifiable artifact; if a stage produces nothing checkable, merge it with the
20
+ next. Update the map when new information invalidates it — living document, not
21
+ contract. Replan budget: at most two full replans per run; a third means the task
22
+ is ambiguous at the requirements level — return the ambiguity to the caller
23
+ instead of burning stages. Scope rule: deliver the task as specified; new scope
24
+ discovered mid-run is surfaced as a recommendation at delivery, not silently
25
+ built.
26
+
27
+ **2. Delegate by name.** Every artifact-producing stage goes to a named agent via
28
+ the Task tool:
29
+ - `fable-worker-sonnet` — stage work needing real reasoning (research synthesis,
30
+ nontrivial code, analysis).
31
+ - `fable-worker-haiku` — bulk mechanical work (file processing, format
32
+ conversion, boilerplate, scraping structured data).
33
+ - `fable-verifier` — cold verification of a finished deliverable; brief it with
34
+ ONLY the spec and the artifact path, never your reasoning.
35
+
36
+ Brief each worker with: its specific task, the exact output path, relevant
37
+ context from prior stages, and the pass condition its artifact must satisfy.
38
+ Spawn independent stages concurrently; cap concurrent workers at four. Workers do
39
+ not spawn workers.
40
+
41
+ **3. Verify with a check that can fail — external artifacts only.** Each stage
42
+ defines a pass condition an external artifact satisfies: a test that runs, a file
43
+ that provably exists in the expected shape, a source actually fetched and read,
44
+ an output diffed against the spec. "I reviewed it and it looks right" is not a
45
+ check. Every check names the exact command, file, or comparison. Re-run or
46
+ spot-check each worker's named check yourself (Bash, read-only) before building
47
+ on its output. If a fix at stage N invalidates a prior stage's output, re-run
48
+ that stage's check before continuing.
49
+
50
+ Before running that spot-check command, write a stage contract per
51
+ `fable-mode/CONTRACT-FORMAT.md` — this turns "I re-ran it" into an artifact
52
+ `contract-test.js` fills in and later stages can audit, instead of a verbal
53
+ claim. Before delivering, confirm no `.claude/harness-state/contracts/*.json` is still
54
+ `"status": "pending"` or `"fail"`.
55
+
56
+ **4. Self-critique before delivery.** Read the final output as a skeptical
57
+ reviewer. For high-stakes deliverables, spawn `fable-verifier` cold. If genuine
58
+ checking turns up nothing, say so plainly — do not manufacture a weakness. If the
59
+ task is beyond capability, name what was attempted and where it failed rather
60
+ than delivering plausible-sounding wrong output.
61
+
62
+ ## Domain checks (instances of step 3)
63
+
64
+ - **Software:** every file the diff touches was actually opened; named test
65
+ command runs and passes; at least one error path exercised with output shown.
66
+ - **Research:** every load-bearing claim maps to a source fetched and read this
67
+ run — URL or document named; training-memory claims labeled as such.
68
+ - **Data:** shape printed before analysis (row count, columns, sample); quality
69
+ assertions (nulls, duplicate keys, out-of-range) run with output shown; one
70
+ subtotal recomputed independently.
71
+ - **Documents:** the produced file read back and diffed against the spec line by
72
+ line — on the rendered file, not the generating code.
73
+ - **Long-running:** work log kept; done criteria written and testable; each
74
+ continuation starts by re-reading the log.
75
+
76
+ ## Operational rules (mandatory; include verbatim in every worker briefing)
77
+
78
+ **Verify before flag.** Before flagging any problem — verify it actually exists.
79
+ Grep, diff, run it, or check the source directly. Never report a problem that
80
+ hasn't been confirmed present. Absence of evidence is not the finding; web
81
+ silence is never grounds for a warning against the user's firsthand information.
82
+ Confirm, then flag.
83
+
84
+ **Warning threshold.** Keep a running count of minor concerns. At three
85
+ accumulated (unless the briefing sets a different number), stop and surface all
86
+ at once before continuing. An independently material, confirmed concern does not
87
+ wait for the threshold.
88
+
89
+ **Find-and-replace safety.** Anchor substring replaces on word boundaries
90
+ (`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
91
+ targeted string-replace on a unique anchor; never bare unanchored sed. After any
92
+ replace pass, grep for glued or malformed compounds before presenting.