harness-everything 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +2 -0
- package/BENCHMARK_SOP.md +62 -0
- package/CHANGELOG.md +107 -0
- package/Dockerfile +18 -0
- package/LICENSE +201 -0
- package/README.md +197 -0
- package/VERIFICATION.md +347 -0
- package/bin/cli.js +97 -0
- package/build-multi-agent-system/AGENTS.md +50 -0
- package/build-multi-agent-system/README.md +63 -0
- package/build-multi-agent-system/SKILL.md +68 -0
- package/create-agent-launcher/SKILL.md +48 -0
- package/create-agent-launcher/examples/csharp-monorepo.md +123 -0
- package/create-agent-launcher/examples/typescript-cli.md +126 -0
- package/create-agent-launcher/guidelines/platform-claude.md +77 -0
- package/create-agent-launcher/guidelines/platform-codex.md +146 -0
- package/create-agent-launcher/guidelines/platform-copilot.md +74 -0
- package/create-agent-launcher/guidelines/platform-gemini.md +101 -0
- package/create-agent-launcher/guidelines/role-capabilities.md +108 -0
- package/create-agent-launcher/templates/README.md +17 -0
- package/create-agent-launcher/templates/agent-launcher.md +68 -0
- package/create-agent-launcher/templates/backend-developer-csharp.md +56 -0
- package/create-agent-launcher/templates/backend-developer-polyglot.md +62 -0
- package/create-agent-launcher/templates/backend-developer-ts.md +57 -0
- package/create-agent-launcher/templates/memory-keeper.md +37 -0
- package/create-agent-launcher/templates/requirement-analyzer.md +66 -0
- package/create-agent-launcher/templates/specialist-agent.md +34 -0
- package/create-agent-launcher/templates/task-verifier.md +56 -0
- package/create-agent-launcher/templates/tool-maker.md +33 -0
- package/create-agent-launcher/workflows/01-init.md +22 -0
- package/create-agent-launcher/workflows/02-analysis.md +26 -0
- package/create-agent-launcher/workflows/03-generation.md +29 -0
- package/create-agent-launcher/workflows/04-launcher.md +20 -0
- package/docker-verify.sh +266 -0
- package/docs/architecture.md +95 -0
- package/docs/philosophy.md +45 -0
- package/docs/reflection.md +34 -0
- package/docs/routing.md +45 -0
- package/docs/workflows/build-multi-agent-system.md +62 -0
- package/docs/workflows/create-agent-launcher.md +62 -0
- package/docs/workflows/environment-detection.md +63 -0
- package/docs/workflows/eval-harness.md +57 -0
- package/docs/workflows/fable-discipline.md +59 -0
- package/docs/workflows/fable-mode.md +32 -0
- package/docs/workflows/git-commit.md +62 -0
- package/docs/workflows/grill-me.md +62 -0
- package/docs/workflows/grill-with-docs.md +62 -0
- package/docs/workflows/harness-everything.md +92 -0
- package/docs/workflows/improve-codebase-architecture.md +60 -0
- package/docs/workflows/install-cognitive-os.md +60 -0
- package/docs/workflows/repo-docs.md +59 -0
- package/docs/workflows/rewrite-commits.md +60 -0
- package/docs/workflows/security-review.md +58 -0
- package/docs/workflows/self-evolve.md +59 -0
- package/docs/workflows/skill-creator.md +75 -0
- package/docs/workflows/skill-style.md +64 -0
- package/docs/workflows/tdd.md +39 -0
- package/docs/workflows/todo-driven-workflow.md +36 -0
- package/docs/workflows/using-git-worktrees.md +63 -0
- package/docs/workflows/verification-loop.md +69 -0
- package/docs/workflows/verify-before-claim.md +60 -0
- package/docs/workflows/zoom-out.md +62 -0
- package/environment-detection/SKILL.md +77 -0
- package/environment-detection/scripts/preflight.js +93 -0
- package/eval-framework/behavioral-test.js +105 -0
- package/eval-framework/cases/case1.json +5 -0
- package/eval-framework/cases/case2.json +5 -0
- package/eval-framework/cases/case3.json +5 -0
- package/eval-framework/cases/case4-zh-tier2.json +5 -0
- package/eval-framework/cases/case5-zh-tier3.json +5 -0
- package/eval-framework/mechanism-test.js +251 -0
- package/eval-framework/runner.js +56 -0
- package/eval-harness/SKILL.md +59 -0
- package/eval-harness/scripts/evaluate.js +61 -0
- package/fable-discipline/SKILL.md +38 -0
- package/fable-mode/CONTRACT-FORMAT.md +51 -0
- package/fable-mode/SKILL.md +51 -0
- package/fable-mode/agents/fable-orchestrator.md +92 -0
- package/fable-mode/agents/fable-verifier.md +32 -0
- package/fable-mode/agents/fable-worker-haiku.md +48 -0
- package/fable-mode/agents/fable-worker-sonnet.md +49 -0
- package/fable-mode/execution-guardrails/SKILL.md +87 -0
- package/fable-mode/fable-haiku/SKILL.md +48 -0
- package/git-commit/SKILL.md +55 -0
- package/git-commit/guides/ANGULAR_STYLE.md +61 -0
- package/git-commit/guides/COMMIT_GENERATION.md +24 -0
- package/git-commit/guides/LANGUAGE_DETECTION.md +16 -0
- package/git-commit/guides/MAIN_REPO.md +15 -0
- package/git-commit/guides/SUBMODULES.md +27 -0
- package/grill-me/SKILL.md +36 -0
- package/grill-with-docs/ADR-FORMAT.md +47 -0
- package/grill-with-docs/CONTEXT-FORMAT.md +63 -0
- package/grill-with-docs/SKILL.md +99 -0
- package/harness-everything/SKILL.md +141 -0
- package/harness-everything/examples/README.md +9 -0
- package/harness-everything/examples/todo-workflow.md +22 -0
- package/harness-everything/scripts/bootstrap.js +140 -0
- package/harness-everything/scripts/self-heal.js +189 -0
- package/harness-everything/scripts/tier-router.js +219 -0
- package/harness-everything/scripts/todo-cli.js +157 -0
- package/harness-everything/scripts/verify-gate.js +31 -0
- package/hooks/hooks.json +155 -0
- package/hooks/scripts/atomic-commit-check.js +68 -0
- package/hooks/scripts/boundary-guard.js +55 -0
- package/hooks/scripts/context-compact.js +114 -0
- package/hooks/scripts/contract-test.js +79 -0
- package/hooks/scripts/depth-guard.js +65 -0
- package/hooks/scripts/lib/harness-state.js +202 -0
- package/hooks/scripts/lib/platforms/claude.js +42 -0
- package/hooks/scripts/lib/platforms/codex.js +46 -0
- package/hooks/scripts/lib/platforms/continue.js +41 -0
- package/hooks/scripts/lib/platforms/copilot.js +41 -0
- package/hooks/scripts/lib/platforms/cursor.js +41 -0
- package/hooks/scripts/lib/platforms/hermes.js +32 -0
- package/hooks/scripts/lib/platforms/index.js +17 -0
- package/hooks/scripts/lib/platforms/worktrees.js +20 -0
- package/hooks/scripts/reset-circuit-breaker.js +30 -0
- package/hooks/scripts/rule-of-3-tracker.js +91 -0
- package/hooks/scripts/rule-of-3.js +176 -0
- package/hooks/scripts/state-persist.js +99 -0
- package/hooks/scripts/stop-gate.js +80 -0
- package/hooks/scripts/subagent-scope-guard.js +88 -0
- package/improve-codebase-architecture/SKILL.md +44 -0
- package/improve-codebase-architecture/guides/DEEPENING.md +37 -0
- package/improve-codebase-architecture/guides/HTML-REPORT.md +123 -0
- package/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -0
- package/improve-codebase-architecture/guides/LANGUAGE.md +53 -0
- package/install-cognitive-os/SKILL.md +95 -0
- package/package.json +22 -0
- package/repo-docs/SKILL.md +45 -0
- package/repo-docs/templates/agents-template.md +264 -0
- package/repo-docs/templates/knowledge-agents-template.md +50 -0
- package/repo-docs/templates/knowledge-readme-template.md +83 -0
- package/repo-docs/templates/manual-recon.md +117 -0
- package/repo-docs/templates/multi-skills-readme-template.md +72 -0
- package/repo-docs/templates/product-agents-template.md +78 -0
- package/repo-docs/templates/product-readme-template.md +95 -0
- package/repo-docs/templates/readme-template.md +136 -0
- package/repo-docs/templates/single-skill-readme-template.md +70 -0
- package/repo-docs/templates/skills-agents-template.md +60 -0
- package/rewrite-commits/SKILL.md +32 -0
- package/scripts/installer.js +649 -0
- package/scripts/lib/advisory-text.js +163 -0
- package/scripts/lib/claude-hooks.js +71 -0
- package/scripts/lib/gitignore.js +58 -0
- package/scripts/lib/manifest.js +82 -0
- package/scripts/lib/prompts.js +241 -0
- package/scripts/lib/skills.js +172 -0
- package/scripts/lib/workspace.js +28 -0
- package/security-review/SKILL.md +511 -0
- package/self-evolve/SKILL.md +54 -0
- package/self-evolve/scripts/persist-memory.js +40 -0
- package/self-evolve/scripts/self-regression.js +113 -0
- package/skill-creator/SKILL.md +91 -0
- package/skill-creator/references/quality-principles.md +81 -0
- package/skill-creator/references/testing-workflow.md +32 -0
- package/skill-style/SKILL.md +48 -0
- package/skill-style/STYLE.md +37 -0
- package/tdd/SKILL.md +53 -0
- package/tdd/guides/deep-modules.md +33 -0
- package/tdd/guides/interface-design.md +31 -0
- package/tdd/guides/mocking.md +59 -0
- package/tdd/guides/refactoring.md +10 -0
- package/tdd/guides/tests.md +61 -0
- package/todo-driven-workflow/SKILL.md +54 -0
- package/using-git-worktrees/SKILL.md +161 -0
- package/verification-loop/SKILL.md +138 -0
- package/verify-before-claim/SKILL.md +86 -0
- package/zoom-out/SKILL.md +90 -0
|
@@ -0,0 +1,251 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* Harness Mechanism Test Suite (Claude Code hook contract)
|
|
4
|
+
* Automates VERIFICATION.md Section 2 (mechanism check) so it runs on every
|
|
5
|
+
* `npm test` instead of only when a human copy-pastes the recipes by hand.
|
|
6
|
+
* That manual gap is exactly what let three independent 2026-07-23 audit
|
|
7
|
+
* reports (see docs/reports/) misdiagnose working hooks as broken: they
|
|
8
|
+
* piped JSON via `echo '...' | node script.js` in a Windows Git Bash shell,
|
|
9
|
+
* which mangles stdin/TTY state and produced false negatives. This suite
|
|
10
|
+
* uses execSync's `input` option instead (the same technique already proven
|
|
11
|
+
* reliable in runner.js and behavioral-test.js) so it is a faithful,
|
|
12
|
+
* cross-platform re-run of the exact payloads VERIFICATION.md documents.
|
|
13
|
+
*
|
|
14
|
+
* Each hook resolves its state directory via detectActivePlatform(), which
|
|
15
|
+
* defaults to 'copilot' under a VS Code integrated terminal (TERM_PROGRAM=
|
|
16
|
+
* vscode) unless CLAUDE is set - so every child process below is spawned
|
|
17
|
+
* with CLAUDE=1 to force the Claude Code state path regardless of the host
|
|
18
|
+
* running this test.
|
|
19
|
+
*
|
|
20
|
+
* All state is written under a dedicated, never-real session id so this
|
|
21
|
+
* suite can never collide with or clobber an actual Claude Code session's
|
|
22
|
+
* breaker/handoff state. The session directory (and any repo-root temp
|
|
23
|
+
* files created along the way) is removed on exit, success or failure alike.
|
|
24
|
+
*/
|
|
25
|
+
const { execSync } = require('child_process');
|
|
26
|
+
const fs = require('fs');
|
|
27
|
+
const path = require('path');
|
|
28
|
+
|
|
29
|
+
process.env.CLAUDE = '1'; // so our own getSessionDir() calls resolve like the children's will
|
|
30
|
+
|
|
31
|
+
const { getWorkspaceRoot, getSessionDir } = require('../hooks/scripts/lib/harness-state');
|
|
32
|
+
|
|
33
|
+
const root = getWorkspaceRoot();
|
|
34
|
+
const hooksDir = path.join(root, 'hooks', 'scripts');
|
|
35
|
+
const SESSION_ID = '__mechanism_test__';
|
|
36
|
+
const sessionDir = getSessionDir(root, SESSION_ID);
|
|
37
|
+
|
|
38
|
+
console.log('=== 🔧 Harness Mechanism Test Suite (VERIFICATION.md §2) ===');
|
|
39
|
+
|
|
40
|
+
const tempFiles = [];
|
|
41
|
+
let failures = 0;
|
|
42
|
+
|
|
43
|
+
function check(name, condition, detail) {
|
|
44
|
+
if (condition) {
|
|
45
|
+
console.log(`✅ ${name}`);
|
|
46
|
+
} else {
|
|
47
|
+
console.error(`❌ ${name}`);
|
|
48
|
+
if (detail) console.error(` ${detail}`);
|
|
49
|
+
failures++;
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
function runHook(scriptName, payload, extraEnv) {
|
|
54
|
+
const scriptPath = path.join(hooksDir, scriptName);
|
|
55
|
+
try {
|
|
56
|
+
const stdout = execSync(`node "${scriptPath}"`, {
|
|
57
|
+
encoding: 'utf8',
|
|
58
|
+
input: payload === null ? '' : JSON.stringify(payload),
|
|
59
|
+
env: { ...process.env, ...extraEnv },
|
|
60
|
+
});
|
|
61
|
+
return { code: 0, stdout, stderr: '' };
|
|
62
|
+
} catch (err) {
|
|
63
|
+
return { code: err.status, stdout: err.stdout || '', stderr: err.stderr || '' };
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
function writeState(file, obj) {
|
|
68
|
+
fs.writeFileSync(path.join(sessionDir, file), JSON.stringify(obj, null, 2), 'utf8');
|
|
69
|
+
}
|
|
70
|
+
function readState(file) {
|
|
71
|
+
return JSON.parse(fs.readFileSync(path.join(sessionDir, file), 'utf8'));
|
|
72
|
+
}
|
|
73
|
+
function stateExists(file) {
|
|
74
|
+
return fs.existsSync(path.join(sessionDir, file));
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
// Registers a repo-root temp file for guaranteed cleanup even if an
|
|
78
|
+
// assertion throws or the process is killed mid-suite.
|
|
79
|
+
function tempFile(name) {
|
|
80
|
+
const p = path.join(root, name);
|
|
81
|
+
tempFiles.push(p);
|
|
82
|
+
return p;
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
process.on('exit', () => {
|
|
86
|
+
for (const f of tempFiles) {
|
|
87
|
+
try { if (fs.existsSync(f)) fs.unlinkSync(f); } catch (e) { /* best-effort */ }
|
|
88
|
+
}
|
|
89
|
+
try { fs.rmSync(sessionDir, { recursive: true, force: true }); } catch (e) { /* best-effort */ }
|
|
90
|
+
});
|
|
91
|
+
|
|
92
|
+
// ---------------------------------------------------------------------------
|
|
93
|
+
// 2a. Rule of 3 circuit breaker trips, releases via zoom-out report, and
|
|
94
|
+
// hard-locks on a repeat trip of the same signature.
|
|
95
|
+
// ---------------------------------------------------------------------------
|
|
96
|
+
console.log('\n[2a] Rule of 3 circuit breaker...');
|
|
97
|
+
|
|
98
|
+
writeState('rule-of-3-state.json', { count: 3, lastHash: 'mech-test', zoomOutResolved: false });
|
|
99
|
+
const tripResult = runHook('rule-of-3.js', { session_id: SESSION_ID });
|
|
100
|
+
check(
|
|
101
|
+
'2a. Trips at count=3 (exit=2, CRITICAL banner)',
|
|
102
|
+
tripResult.code === 2 && tripResult.stderr.includes('RULE OF 3 CIRCUIT BREAKER TRIGGERED'),
|
|
103
|
+
`Got exit=${tripResult.code}, stderr="${tripResult.stderr.slice(0, 200)}"`
|
|
104
|
+
);
|
|
105
|
+
|
|
106
|
+
writeState('rule-of-3-state.json', {
|
|
107
|
+
count: 3, lastHash: 'mech-test', zoomOutResolved: false, lastFailureAt: 0, zoomOutCycles: 0,
|
|
108
|
+
});
|
|
109
|
+
fs.writeFileSync(
|
|
110
|
+
path.join(sessionDir, 'zoom-out-report.md'),
|
|
111
|
+
'## Goal\nx\n## Failed Attempts\nx\n## Verified Facts\nx\n## Diagnosis\nx\n## Decision\nRESUME: new approach\n',
|
|
112
|
+
'utf8'
|
|
113
|
+
);
|
|
114
|
+
const releaseResult = runHook('rule-of-3.js', { session_id: SESSION_ID });
|
|
115
|
+
const releasedState = readState('rule-of-3-state.json');
|
|
116
|
+
check(
|
|
117
|
+
'2a-bis. Valid zoom-out report releases the breaker (exit=0, count reset)',
|
|
118
|
+
releaseResult.code === 0 &&
|
|
119
|
+
releaseResult.stdout.includes('breaker released') &&
|
|
120
|
+
releasedState.count === 0 &&
|
|
121
|
+
releasedState.zoomOutResolved === true &&
|
|
122
|
+
releasedState.zoomOutCycles === 1,
|
|
123
|
+
`Got exit=${releaseResult.code}, state=${JSON.stringify(releasedState)}`
|
|
124
|
+
);
|
|
125
|
+
|
|
126
|
+
writeState('rule-of-3-state.json', {
|
|
127
|
+
count: 3, lastHash: 'mech-test', zoomOutResolved: false,
|
|
128
|
+
lastFailureAt: Date.now() + 60000, zoomOutCycles: 1,
|
|
129
|
+
});
|
|
130
|
+
const hardLockResult = runHook('rule-of-3.js', { session_id: SESSION_ID });
|
|
131
|
+
check(
|
|
132
|
+
'2a-ter. Second trip on same signature hard-locks (exit=2, repeat trip)',
|
|
133
|
+
hardLockResult.code === 2 && hardLockResult.stderr.includes('repeat trip - hard lock'),
|
|
134
|
+
`Got exit=${hardLockResult.code}, stderr="${hardLockResult.stderr.slice(0, 200)}"`
|
|
135
|
+
);
|
|
136
|
+
|
|
137
|
+
// ---------------------------------------------------------------------------
|
|
138
|
+
// 2b. Boundary guard blocks an oversized Read
|
|
139
|
+
// ---------------------------------------------------------------------------
|
|
140
|
+
console.log('\n[2b] Boundary guard...');
|
|
141
|
+
|
|
142
|
+
const bigFile = tempFile('.mechanism-test-big.tmp');
|
|
143
|
+
fs.writeFileSync(bigFile, 'x'.repeat(600 * 1024));
|
|
144
|
+
const boundaryResult = runHook('boundary-guard.js', { tool_name: 'Read', tool_input: { file_path: bigFile } });
|
|
145
|
+
check(
|
|
146
|
+
'2b. Blocks a 600KB Read without limit/offset (exit=2, BLOCKED)',
|
|
147
|
+
boundaryResult.code === 2 && boundaryResult.stderr.includes('[Boundary Guard] BLOCKED'),
|
|
148
|
+
`Got exit=${boundaryResult.code}, stderr="${boundaryResult.stderr.slice(0, 200)}"`
|
|
149
|
+
);
|
|
150
|
+
fs.unlinkSync(bigFile);
|
|
151
|
+
|
|
152
|
+
// ---------------------------------------------------------------------------
|
|
153
|
+
// 2c. State persistence (WAL) records a failure, then clears it on success
|
|
154
|
+
// ---------------------------------------------------------------------------
|
|
155
|
+
console.log('\n[2c] State persistence (WAL)...');
|
|
156
|
+
|
|
157
|
+
runHook('state-persist.js', {
|
|
158
|
+
session_id: SESSION_ID,
|
|
159
|
+
tool_name: 'Bash',
|
|
160
|
+
tool_response: { stdout: '', stderr: 'npm ERR! mech-test failure' },
|
|
161
|
+
});
|
|
162
|
+
const failedHandoff = readState('handoff-state.json');
|
|
163
|
+
check(
|
|
164
|
+
'2c. Failing Bash call is recorded as status=failed',
|
|
165
|
+
failedHandoff.status === 'failed' && failedHandoff.errorSummary.includes('mech-test failure'),
|
|
166
|
+
`Got ${JSON.stringify(failedHandoff)}`
|
|
167
|
+
);
|
|
168
|
+
|
|
169
|
+
runHook('state-persist.js', {
|
|
170
|
+
session_id: SESSION_ID,
|
|
171
|
+
tool_name: 'Bash',
|
|
172
|
+
tool_response: { stdout: 'ok', exitCode: 0 },
|
|
173
|
+
});
|
|
174
|
+
const resolvedHandoff = readState('handoff-state.json');
|
|
175
|
+
check(
|
|
176
|
+
'2c. A subsequent successful command clears status back to idle',
|
|
177
|
+
resolvedHandoff.status === 'idle' && !!resolvedHandoff.lastResolved,
|
|
178
|
+
`Got ${JSON.stringify(resolvedHandoff)}`
|
|
179
|
+
);
|
|
180
|
+
|
|
181
|
+
// ---------------------------------------------------------------------------
|
|
182
|
+
// 2d. Fact-audit reminder reaches the agent via tier-router.js stdin
|
|
183
|
+
// ---------------------------------------------------------------------------
|
|
184
|
+
console.log('\n[2d] Fact-audit reminder...');
|
|
185
|
+
|
|
186
|
+
const tierRouterPath = path.join(root, 'harness-everything', 'scripts', 'tier-router.js');
|
|
187
|
+
const factAuditOut = execSync(`node "${tierRouterPath}"`, {
|
|
188
|
+
encoding: 'utf8',
|
|
189
|
+
input: JSON.stringify({ prompt: 'what exit code does this hook use by default and is it documented' }),
|
|
190
|
+
});
|
|
191
|
+
check(
|
|
192
|
+
'2d. tier-router.js reads {"prompt":...} from stdin and emits FACT-AUDIT REMINDER',
|
|
193
|
+
factAuditOut.includes('FACT-AUDIT REMINDER'),
|
|
194
|
+
`Output was:\n${factAuditOut.slice(0, 300)}`
|
|
195
|
+
);
|
|
196
|
+
|
|
197
|
+
// ---------------------------------------------------------------------------
|
|
198
|
+
// 2e. Subagent scope guard catches an out-of-scope file change
|
|
199
|
+
// ---------------------------------------------------------------------------
|
|
200
|
+
console.log('\n[2e] Subagent scope guard...');
|
|
201
|
+
|
|
202
|
+
runHook('subagent-scope-guard.js', {
|
|
203
|
+
session_id: SESSION_ID, tool_name: 'Task', hook_event_name: 'PreToolUse', tool_input: {},
|
|
204
|
+
});
|
|
205
|
+
const scopeFile = tempFile('.mechanism-test-scope.tmp');
|
|
206
|
+
fs.writeFileSync(scopeFile, 'unexpected change');
|
|
207
|
+
const scopeResult = runHook('subagent-scope-guard.js', {
|
|
208
|
+
session_id: SESSION_ID, tool_name: 'Task', hook_event_name: 'PostToolUse', tool_input: {},
|
|
209
|
+
});
|
|
210
|
+
check(
|
|
211
|
+
'2e. Flags the newly-created file after a Task burst (exit=2, lists filename)',
|
|
212
|
+
scopeResult.code === 2 &&
|
|
213
|
+
scopeResult.stderr.includes('Subagent Scope Guard') &&
|
|
214
|
+
scopeResult.stderr.includes(path.basename(scopeFile)),
|
|
215
|
+
`Got exit=${scopeResult.code}, stderr="${scopeResult.stderr.slice(0, 300)}"`
|
|
216
|
+
);
|
|
217
|
+
fs.unlinkSync(scopeFile);
|
|
218
|
+
|
|
219
|
+
// ---------------------------------------------------------------------------
|
|
220
|
+
// 2f. Stop gate bounces an unverified-edit stop exactly once
|
|
221
|
+
// ---------------------------------------------------------------------------
|
|
222
|
+
console.log('\n[2f] Stop gate...');
|
|
223
|
+
|
|
224
|
+
writeState('handoff-state.json', { status: 'idle', lastEditAt: Date.now(), lastVerifyAt: 0 });
|
|
225
|
+
const dirtyFile = tempFile('.mechanism-test-dirty.tmp');
|
|
226
|
+
fs.writeFileSync(dirtyFile, 'dirty');
|
|
227
|
+
const firstStop = runHook('stop-gate.js', { session_id: SESSION_ID });
|
|
228
|
+
check(
|
|
229
|
+
'2f. First stop after an unverified edit bounces (exit=2)',
|
|
230
|
+
firstStop.code === 2 && firstStop.stderr.includes('[Stop Gate]'),
|
|
231
|
+
`Got exit=${firstStop.code}, stderr="${firstStop.stderr.slice(0, 200)}"`
|
|
232
|
+
);
|
|
233
|
+
const secondStop = runHook('stop-gate.js', { session_id: SESSION_ID });
|
|
234
|
+
check(
|
|
235
|
+
'2f. Same edit batch does not bounce twice (exit=0)',
|
|
236
|
+
secondStop.code === 0,
|
|
237
|
+
`Got exit=${secondStop.code}`
|
|
238
|
+
);
|
|
239
|
+
fs.unlinkSync(dirtyFile);
|
|
240
|
+
|
|
241
|
+
// ---------------------------------------------------------------------------
|
|
242
|
+
console.log('\n=================================================');
|
|
243
|
+
if (failures > 0) {
|
|
244
|
+
console.error(` ❌ ${failures} mechanism check(s) FAILED.`);
|
|
245
|
+
console.log('=================================================');
|
|
246
|
+
process.exit(1);
|
|
247
|
+
} else {
|
|
248
|
+
console.log(' 🎉 All VERIFICATION.md §2 mechanism checks passed.');
|
|
249
|
+
console.log('=================================================');
|
|
250
|
+
process.exit(0);
|
|
251
|
+
}
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
const fs = require('fs');
|
|
3
|
+
const path = require('path');
|
|
4
|
+
const { execSync } = require('child_process');
|
|
5
|
+
|
|
6
|
+
const casesDir = path.join(__dirname, 'cases');
|
|
7
|
+
const files = fs.readdirSync(casesDir).filter(f => f.endsWith('.json'));
|
|
8
|
+
|
|
9
|
+
console.log("=== Harness Tier Verification Framework ===");
|
|
10
|
+
console.log("This framework simulates routing prompts to verify trigger rates.\n");
|
|
11
|
+
|
|
12
|
+
let passed = 0;
|
|
13
|
+
let failed = 0;
|
|
14
|
+
|
|
15
|
+
files.forEach(file => {
|
|
16
|
+
const data = JSON.parse(fs.readFileSync(path.join(casesDir, file), 'utf8'));
|
|
17
|
+
console.log(`[Testing] ${data.name}`);
|
|
18
|
+
console.log(` Prompt: "${data.prompt}"`);
|
|
19
|
+
console.log(` Expected Tier: ${data.expected_tier}`);
|
|
20
|
+
|
|
21
|
+
try {
|
|
22
|
+
const tierRouterPath = path.join(__dirname, '..', 'harness-everything', 'scripts', 'tier-router.js');
|
|
23
|
+
// Claude Code feeds UserPromptSubmit hooks JSON on stdin (a "prompt"
|
|
24
|
+
// field), not a CLI argument - match that here so this test actually
|
|
25
|
+
// exercises the real interface instead of a path tier-router.js never
|
|
26
|
+
// sees in production.
|
|
27
|
+
const output = execSync(`node "${tierRouterPath}"`, {
|
|
28
|
+
encoding: 'utf8',
|
|
29
|
+
input: JSON.stringify({ prompt: data.prompt }),
|
|
30
|
+
env: { ...process.env, HARNESS_EVAL: 'true' }
|
|
31
|
+
});
|
|
32
|
+
|
|
33
|
+
// Parse the output to find the recommended tier (accept the old
|
|
34
|
+
// "REQUIRED TIER" label too so the runner works across versions)
|
|
35
|
+
const tierMatch = output.match(/(?:RECOMMENDED|REQUIRED) TIER: (Tier \d+)/i) || output.match(/(Tier \d+)/i);
|
|
36
|
+
const actualTier = tierMatch ? tierMatch[1] : "Unknown";
|
|
37
|
+
|
|
38
|
+
if (actualTier.includes(data.expected_tier)) {
|
|
39
|
+
console.log(` -> Result: PASS (Got ${actualTier})\n`);
|
|
40
|
+
passed++;
|
|
41
|
+
} else {
|
|
42
|
+
console.log(` -> Result: FAIL (Expected ${data.expected_tier}, Got ${actualTier})\n`);
|
|
43
|
+
console.log(` Output was:\n${output}`);
|
|
44
|
+
failed++;
|
|
45
|
+
}
|
|
46
|
+
} catch (err) {
|
|
47
|
+
console.log(` -> Result: ERROR executing router`);
|
|
48
|
+
failed++;
|
|
49
|
+
}
|
|
50
|
+
});
|
|
51
|
+
|
|
52
|
+
console.log(`--- Summary: ${passed} passed, ${failed} failed ---`);
|
|
53
|
+
if (failed > 0) {
|
|
54
|
+
process.exit(1);
|
|
55
|
+
}
|
|
56
|
+
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: eval-harness
|
|
3
|
+
description: Objectively evaluates AI agent performance based on correctness, resource efficiency, and anti-loop focus.
|
|
4
|
+
author: Miya Daniel | Harness Core Team
|
|
5
|
+
version: 0.2.0
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Eval Harness (Automated Performance & Reasoning Evaluation)
|
|
9
|
+
|
|
10
|
+
## 📋 Skill Contract
|
|
11
|
+
|
|
12
|
+
| Component | Specification |
|
|
13
|
+
| :--- | :--- |
|
|
14
|
+
| **Trigger / Input** | User requests "run benchmark", "score conversation log", or "compare Vanilla vs Harness differences" — an execution log or the current conversation history. |
|
|
15
|
+
| **Expected Output** | A 4-dimension scorecard (Correctness & Factuality, Token & Step Efficiency, Anti-loop & Focus, Environment & Tool Awareness), each scored 0-10 with rationale. |
|
|
16
|
+
| **State Mutations** | Writes a generated Markdown scorecard to the project's `evals/` directory via `scripts/evaluate.js`. |
|
|
17
|
+
| **Enforcement Gate** | MUST execute `scripts/evaluate.js` with the four scores and insight text — a verbally-stated score without running the script does not count as output. |
|
|
18
|
+
|
|
19
|
+
This skill is used to objectively evaluate the performance of an AI Agent when "executing test tasks".
|
|
20
|
+
Triggered when the user requests "run benchmark", "score conversation log", or "compare Vanilla vs Harness differences".
|
|
21
|
+
|
|
22
|
+
## 1. Data Parsing `[Discover]`
|
|
23
|
+
- Read the execution logs provided by the user or directly analyze the current conversation history.
|
|
24
|
+
- Extract key metrics:
|
|
25
|
+
- **Action Count**: Includes the number of file reads, searches, and terminal executions.
|
|
26
|
+
- **Error Loop Count**: The number of times the same error message appeared consecutively triggering attempted micro-adjustments.
|
|
27
|
+
- **Token / Time Indicator**: Record token consumption or time spent if provided by the user.
|
|
28
|
+
|
|
29
|
+
## 2. Scoring Rubric `[Think]`
|
|
30
|
+
Please score based on the following four dimensions (out of 10 points each):
|
|
31
|
+
|
|
32
|
+
### A. Correctness & Factuality
|
|
33
|
+
- 10 points: Perfectly solved the problem with no side effects. Clearly knows its own knowledge boundaries, **no hallucinations or fabricated facts**.
|
|
34
|
+
- 5 points: Solved the surface problem but introduced new issues, or provided slightly outdated information.
|
|
35
|
+
- 0 points: Task failed, broke original code, or "hallucinated with straight-faced confidence".
|
|
36
|
+
|
|
37
|
+
### B. Token & Step Efficiency
|
|
38
|
+
- 10 points: Extremely streamlined, no superfluous exploration or useless tool calls (prevented over-engineering).
|
|
39
|
+
- 5 points: Reasonable steps, but with minor unnecessary file reads.
|
|
40
|
+
- 0 points: Over-engineered, writing lengthy plans or repeatedly reading irrelevant files for a simple task.
|
|
41
|
+
|
|
42
|
+
### C. Anti-loop & Focus
|
|
43
|
+
- 10 points: Decisively triggered `zoom-out` when hitting a bottleneck, rebuilt the full picture, and either self-recovered on a fresh diagnosis or escalated a genuine decision with options; did not hit a dead end or dump an "I'm stuck" on the human.
|
|
44
|
+
- 5 points: Stopped after 3~5 attempts.
|
|
45
|
+
- 0 points: Fell into a Micro-error looping cycle more than 5 times, or completely forgot the initial goal.
|
|
46
|
+
|
|
47
|
+
### D. Environment & Tool Awareness
|
|
48
|
+
- 10 points: Accurately detected OS, terminal shell (e.g., Git Bash vs PowerShell), and available tools in the `[Discover]` phase before acting. No syntax errors caused by wrong shell assumptions.
|
|
49
|
+
- 5 points: Made an initial incorrect assumption about the terminal/tools but corrected it immediately after the first failure.
|
|
50
|
+
- 0 points: Blindly threw commands (e.g., PowerShell syntax in Git Bash) and repeatedly failed without realizing the environment mismatch.
|
|
51
|
+
|
|
52
|
+
## 3. Output Report `[Summarize]`
|
|
53
|
+
- **MANDATORY**: You MUST execute the evaluation script to generate and save the report.
|
|
54
|
+
- Find the `evaluate.js` script in the `scripts/` directory of this skill and run it via terminal:
|
|
55
|
+
```bash
|
|
56
|
+
node path/to/eval-harness/scripts/evaluate.js <scoreA> <scoreB> <scoreC> <scoreD> "<Your insights here>"
|
|
57
|
+
```
|
|
58
|
+
*(Example: `node scripts/evaluate.js 10 5 10 10 "The agent successfully used zoom-out..."`)*
|
|
59
|
+
- The script will automatically generate the Markdown-formatted **Scorecard Table** and save it to the project's `evals/` directory.
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
const fs = require('fs');
|
|
3
|
+
const path = require('path');
|
|
4
|
+
|
|
5
|
+
const args = process.argv.slice(2);
|
|
6
|
+
if (args.length < 5) {
|
|
7
|
+
console.error("Usage: node evaluate.js <scoreA> <scoreB> <scoreC> <scoreD> \"<insights>\"");
|
|
8
|
+
console.error("Scores should be 0, 5, or 10.");
|
|
9
|
+
process.exit(1);
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
const [scoreA, scoreB, scoreC, scoreD, ...insightParts] = args;
|
|
13
|
+
const insights = insightParts.join(" ");
|
|
14
|
+
|
|
15
|
+
const scores = {
|
|
16
|
+
"Correctness & Factuality": parseInt(scoreA, 10),
|
|
17
|
+
"Token & Step Efficiency": parseInt(scoreB, 10),
|
|
18
|
+
"Anti-loop & Focus": parseInt(scoreC, 10),
|
|
19
|
+
"Environment & Tool Awareness": parseInt(scoreD, 10)
|
|
20
|
+
};
|
|
21
|
+
|
|
22
|
+
let total = 0;
|
|
23
|
+
let markdown = `## Harness Evaluation Report - ${new Date().toISOString()}\n\n| Dimension | Score |\n|---|---|\n`;
|
|
24
|
+
|
|
25
|
+
for (const [dim, score] of Object.entries(scores)) {
|
|
26
|
+
if (isNaN(score)) {
|
|
27
|
+
console.error(`Error: Score for ${dim} is not a valid number.`);
|
|
28
|
+
process.exit(1);
|
|
29
|
+
}
|
|
30
|
+
markdown += `| ${dim} | ${score}/10 |\n`;
|
|
31
|
+
total += score;
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
markdown += `| **Total Score** | **${total}/40** |\n\n`;
|
|
35
|
+
markdown += `### Insights\n${insights}\n\n---\n`;
|
|
36
|
+
|
|
37
|
+
function getWorkspaceRoot() {
|
|
38
|
+
let dir = path.resolve(process.cwd());
|
|
39
|
+
while (dir !== path.parse(dir).root) {
|
|
40
|
+
if (fs.existsSync(path.join(dir, '.git'))) return dir;
|
|
41
|
+
dir = path.dirname(dir);
|
|
42
|
+
}
|
|
43
|
+
return process.cwd();
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
// Store the evaluations in an evals folder in the project root
|
|
47
|
+
const evalDir = path.join(getWorkspaceRoot(), 'evals');
|
|
48
|
+
if (!fs.existsSync(evalDir)) {
|
|
49
|
+
fs.mkdirSync(evalDir, { recursive: true });
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const reportPath = path.join(evalDir, 'evaluation-reports.md');
|
|
53
|
+
try {
|
|
54
|
+
fs.appendFileSync(reportPath, markdown, 'utf8');
|
|
55
|
+
console.log(`[Success] Evaluation report generated and saved to ${reportPath}`);
|
|
56
|
+
console.log('\n--- Report Preview ---\n');
|
|
57
|
+
console.log(markdown);
|
|
58
|
+
} catch (err) {
|
|
59
|
+
console.error(`[Error] Failed to write evaluation report: ${err.message}`);
|
|
60
|
+
process.exit(1);
|
|
61
|
+
}
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: fable-discipline
|
|
3
|
+
description: Shadow guard for fable-mode to prevent context bloat and maintain physical boundaries.
|
|
4
|
+
author: Miya Daniel | Harness Core Team
|
|
5
|
+
version: 0.2.0
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Fable Discipline (Macro Task Discipline & Safety Net)
|
|
9
|
+
|
|
10
|
+
## 📋 Skill Contract
|
|
11
|
+
|
|
12
|
+
| Component | Specification |
|
|
13
|
+
| :--- | :--- |
|
|
14
|
+
| **Trigger / Input** | Runs continuously in the background for the entire duration `fable-mode` is active — not a standalone, one-shot trigger. |
|
|
15
|
+
| **Expected Output** | Periodic state compaction after each milestone; one Git commit per independently functioning logic block; a state manifest handed off between sub-agents at every handoff point. |
|
|
16
|
+
| **State Mutations** | None of its own — it constrains how `fable-mode` and `create-agent-launcher` mutate state (commit cadence, handoff manifests, read-size limits). |
|
|
17
|
+
| **Enforcement Gate** | **HALTS execution immediately** (not a retry) if build errors show a divergent trend (more fixes → more breakage), and forces `zoom-out` instead of continued patching. |
|
|
18
|
+
|
|
19
|
+
This skill acts as the shadow guard for `fable-mode`. As long as `fable-mode` is active, this skill MUST run in the background to prevent large tasks from spiraling out of control.
|
|
20
|
+
|
|
21
|
+
## 1. Anti-Context Bloat Defense
|
|
22
|
+
When executing large architectural tasks, conversation logs expand rapidly, leading to model attention loss (hallucinations or forgetting the original intent).
|
|
23
|
+
- **Strategic Compact**: After completing each core milestone, forcefully summarize the current state and decisions, discarding unnecessary past conversation details.
|
|
24
|
+
- **Avoid Broad Reads**: Prohibited from using broad Regex searches without precise conditions or reading irrelevant files over 1000 lines.
|
|
25
|
+
|
|
26
|
+
## 2. Strict Physical Boundaries
|
|
27
|
+
- **Environment Isolation**: Before modifying any core architecture, ensure you are clearly aware of the Current Working Directory (CWD).
|
|
28
|
+
- **Atomic Commits**: You are NOT allowed to modify dozens of files at once before testing. For every independently functioning logic block completed, advise the human to create a Git Commit as a safe save point.
|
|
29
|
+
|
|
30
|
+
## 3. Agent Handoff Protocol
|
|
31
|
+
When `fable-mode` spawns and switches between different sub-agents via `create-agent-launcher`, strict handoff discipline must be observed:
|
|
32
|
+
- **State Manifest**: The previous agent MUST leave a clear state record (e.g., what APIs were completed, expected inputs/outputs).
|
|
33
|
+
- **Contract Testing**: The very first step for the succeeding agent is to verify if the state manifest left by the previous agent is correct.
|
|
34
|
+
|
|
35
|
+
## 4. Ultimate Circuit Breaker: Indefinite Halt
|
|
36
|
+
- Large tasks easily generate the sunk cost fallacy of "I think it's almost fixed, let me try one more time."
|
|
37
|
+
- If the system's Build Errors exhibit a divergent trend (the more you fix, the more it breaks), **HALT EXECUTION IMMEDIATELY**.
|
|
38
|
+
- Call `zoom-out`, organize the error dependency graph for the human, and seek architectural-level refactoring advice.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Stage Contract Format
|
|
2
|
+
|
|
3
|
+
Stage contracts live in `.claude/harness-state/contracts/` and make fable-orchestrator's
|
|
4
|
+
"re-run or spot-check every stage's named check" discipline auditable instead
|
|
5
|
+
of a verbal claim. The `contract-test.js` hook (PostToolUse: Bash) watches for
|
|
6
|
+
the exact `checkCommand` running and fills in the result — you write the
|
|
7
|
+
contract, the hook resolves it.
|
|
8
|
+
|
|
9
|
+
## When to write one
|
|
10
|
+
|
|
11
|
+
Right before you (the orchestrator) run the Bash command that re-verifies a
|
|
12
|
+
stage's pass condition — one contract per stage, filed the moment you're about
|
|
13
|
+
to spot-check it. Not for every Bash call, only the ones that *are* the named
|
|
14
|
+
check.
|
|
15
|
+
|
|
16
|
+
## File
|
|
17
|
+
|
|
18
|
+
`.claude/harness-state/contracts/<stageId>.json`, created lazily. `stageId` is whatever you
|
|
19
|
+
numbered the stage in your stage map (`stage-3`, `db-migration`, etc).
|
|
20
|
+
|
|
21
|
+
## Schema
|
|
22
|
+
|
|
23
|
+
```json
|
|
24
|
+
{
|
|
25
|
+
"stageId": "stage-3",
|
|
26
|
+
"agent": "fable-worker-sonnet",
|
|
27
|
+
"task": "one-line description of what the worker was assigned",
|
|
28
|
+
"outputPath": "path to the artifact the worker produced",
|
|
29
|
+
"checkCommand": "the exact shell command you are about to run to verify it",
|
|
30
|
+
"status": "pending"
|
|
31
|
+
}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Write it with `status: "pending"` and the *exact* `checkCommand` string you're
|
|
35
|
+
about to run — the hook matches verbatim, not fuzzily. Then run that command.
|
|
36
|
+
The hook fills in `status` (`pass`/`fail`), `evidence` (tail of the command's
|
|
37
|
+
output), and `verifiedAt` for you; you don't write those fields yourself.
|
|
38
|
+
|
|
39
|
+
## Reading contracts back
|
|
40
|
+
|
|
41
|
+
Before delivering, `Read` (or `Grep status` across) `.claude/harness-state/contracts/*.json`.
|
|
42
|
+
Any file still `"status": "pending"` means the named check was declared but
|
|
43
|
+
never actually run — that stage's output hasn't been verified, whatever the
|
|
44
|
+
worker's report claimed. Resolve it before building further on that stage.
|
|
45
|
+
|
|
46
|
+
## Not a blocking gate
|
|
47
|
+
|
|
48
|
+
The hook can only react after a Bash call finishes — it can't stop a bad stage
|
|
49
|
+
from being built on in real time. It surfaces a failed check loudly (fed back
|
|
50
|
+
to you immediately after the command runs), but the actual discipline is still
|
|
51
|
+
yours: don't build on a stage whose contract is `pending` or `fail`.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: fable-mode
|
|
3
|
+
description: Macro task planning and execution engine for complex requirements and low-level architecture changes.
|
|
4
|
+
author: Miya Daniel | Harness Core Team
|
|
5
|
+
version: 0.2.0
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Fable Mode (Macro Task Planning & Execution Engine)
|
|
9
|
+
|
|
10
|
+
## 📋 Skill Contract
|
|
11
|
+
|
|
12
|
+
| Component | Specification |
|
|
13
|
+
| :--- | :--- |
|
|
14
|
+
| **Trigger / Input** | Tier 3 Task classification. Input: Massive architectural requirement or module creation. |
|
|
15
|
+
| **Expected Output** | 1. Documented architectural plan. 2. A sequence of subagent handoffs (`create-agent-launcher`). 3. System integration verified by terminal scripts. |
|
|
16
|
+
| **State Mutations** | 1. Generates `todo-driven-workflow` checklist (via CLI) tracking subagent milestones. 2. `subagent-scope-state.json` updates during handoffs. |
|
|
17
|
+
| **Enforcement Gate** | You MUST use `node harness-everything/scripts/todo-cli.js` to track milestones. You MUST run system tests (`verify-gate.js` or equivalent) after EACH subagent returns before moving to the next. |
|
|
18
|
+
|
|
19
|
+
This skill is automatically triggered and loaded by the `harness-everything` router when a task is judged as a **Tier 3 (Macro Task)**.
|
|
20
|
+
It applies to building new feature modules from scratch, massive low-level architecture changes, or extremely vague and complex user requirements.
|
|
21
|
+
|
|
22
|
+
## Core Concept: Plan Before Acting
|
|
23
|
+
|
|
24
|
+
In Fable Mode, you are no longer a simple "typist", but an "Architect" and "Project Manager".
|
|
25
|
+
You MUST strictly comply with and pair this with `fable-discipline` to control risks.
|
|
26
|
+
|
|
27
|
+
## Sub-Skills (bundled with this skill)
|
|
28
|
+
- `execution-guardrails/` — always-on operational rules (verify-before-flag, warning batching, find-and-replace safety). These apply on EVERY tier and every model, not just inside Fable Mode; this skill's self-critique and file edits inherit them.
|
|
29
|
+
- `fable-haiku/` — opt-in variant that routes the staged loop to a cheap Haiku worker agent for bulk mechanical work. Only use when the user explicitly asks for it.
|
|
30
|
+
|
|
31
|
+
## Execution Phases:
|
|
32
|
+
|
|
33
|
+
### 1. Discovery & Planning
|
|
34
|
+
- **Assess Current State**: Comprehensively review the file architecture related to the requirement and understand dependencies.
|
|
35
|
+
- **Deconstruct Task**: Break down the massive macro task into independently verifiable sub-tasks.
|
|
36
|
+
- **Produce Plan**: Write a clear implementation plan (can be stored in chat or a temporary file) and confirm it with the Human Partner.
|
|
37
|
+
- **Materialize the Plan as a Checklist**: The confirmed plan MUST be initialized as the `todo-driven-workflow` checklist — every sub-task, sub-agent handoff, and milestone check is tracked there, not in prose. This is the harness's base execution loop.
|
|
38
|
+
|
|
39
|
+
### 2. Sub-agent Orchestration
|
|
40
|
+
- If the task involves different professional domains (e.g., modifying Postgres schema and writing React UI), it is **PROHIBITED** for a single model context to handle everything from start to finish, as this leads to attention scatter and Token waste.
|
|
41
|
+
- **Mandatory Call**: Invoke the `create-agent-launcher` skill.
|
|
42
|
+
- Based on the deconstructed sub-tasks, configure the corresponding Sub-agents (e.g., spawn a Database Expert to handle the Schema, then hand off to a Frontend Expert for the UI).
|
|
43
|
+
|
|
44
|
+
### 3. Monitoring & Integration
|
|
45
|
+
- As the chief commander, Fable Mode is responsible for monitoring the progress of each Sub-agent.
|
|
46
|
+
- Ensure the outputs of various sub-agents can integrate perfectly.
|
|
47
|
+
- Upon completing each Milestone, a global test or check MUST be executed.
|
|
48
|
+
|
|
49
|
+
## Handoff & Defense
|
|
50
|
+
- Fable Mode consumes massive system resources and Context Window.
|
|
51
|
+
- Once the core module scaffolding is built and the task downgrades to Tier 2 feature completion, you MUST **proactively exit Fable Mode** and hand off to the `tdd` mode for subsequent detailed implementation.
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: fable-orchestrator
|
|
3
|
+
description: Staged-execution orchestrator for large, multi-part, or multi-session tasks. Use when fable-mode discipline must run with enforced delegation — it writes the stage map, delegates ALL artifact production to fable-worker-sonnet / fable-worker-haiku, verifies every stage with a failable check, and sends high-stakes deliverables to fable-verifier for a cold re-check. It has no Write or Edit tool, so it cannot do the work itself.
|
|
4
|
+
tools: Read, Grep, Glob, Bash, Task, TodoWrite
|
|
5
|
+
model: opus
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
You are the fable orchestrator. You coordinate; you do not produce. You have no
|
|
9
|
+
Write or Edit tool by design — every artifact must come from a worker agent. Your
|
|
10
|
+
Bash access is for read-only inspection and running verification commands (tests,
|
|
11
|
+
greps, diffs) ONLY. Never create or modify a file through Bash redirection,
|
|
12
|
+
heredocs, tee, sed -i, or any other side channel — that defeats the reason Write
|
|
13
|
+
was removed. If you catch yourself about to produce content, stop and delegate.
|
|
14
|
+
|
|
15
|
+
## Core loop
|
|
16
|
+
|
|
17
|
+
**1. Stage map (before touching anything).** Write the full stage plan first.
|
|
18
|
+
Number stages; give each a brief expected output. Each stage produces one
|
|
19
|
+
verifiable artifact; if a stage produces nothing checkable, merge it with the
|
|
20
|
+
next. Update the map when new information invalidates it — living document, not
|
|
21
|
+
contract. Replan budget: at most two full replans per run; a third means the task
|
|
22
|
+
is ambiguous at the requirements level — return the ambiguity to the caller
|
|
23
|
+
instead of burning stages. Scope rule: deliver the task as specified; new scope
|
|
24
|
+
discovered mid-run is surfaced as a recommendation at delivery, not silently
|
|
25
|
+
built.
|
|
26
|
+
|
|
27
|
+
**2. Delegate by name.** Every artifact-producing stage goes to a named agent via
|
|
28
|
+
the Task tool:
|
|
29
|
+
- `fable-worker-sonnet` — stage work needing real reasoning (research synthesis,
|
|
30
|
+
nontrivial code, analysis).
|
|
31
|
+
- `fable-worker-haiku` — bulk mechanical work (file processing, format
|
|
32
|
+
conversion, boilerplate, scraping structured data).
|
|
33
|
+
- `fable-verifier` — cold verification of a finished deliverable; brief it with
|
|
34
|
+
ONLY the spec and the artifact path, never your reasoning.
|
|
35
|
+
|
|
36
|
+
Brief each worker with: its specific task, the exact output path, relevant
|
|
37
|
+
context from prior stages, and the pass condition its artifact must satisfy.
|
|
38
|
+
Spawn independent stages concurrently; cap concurrent workers at four. Workers do
|
|
39
|
+
not spawn workers.
|
|
40
|
+
|
|
41
|
+
**3. Verify with a check that can fail — external artifacts only.** Each stage
|
|
42
|
+
defines a pass condition an external artifact satisfies: a test that runs, a file
|
|
43
|
+
that provably exists in the expected shape, a source actually fetched and read,
|
|
44
|
+
an output diffed against the spec. "I reviewed it and it looks right" is not a
|
|
45
|
+
check. Every check names the exact command, file, or comparison. Re-run or
|
|
46
|
+
spot-check each worker's named check yourself (Bash, read-only) before building
|
|
47
|
+
on its output. If a fix at stage N invalidates a prior stage's output, re-run
|
|
48
|
+
that stage's check before continuing.
|
|
49
|
+
|
|
50
|
+
Before running that spot-check command, write a stage contract per
|
|
51
|
+
`fable-mode/CONTRACT-FORMAT.md` — this turns "I re-ran it" into an artifact
|
|
52
|
+
`contract-test.js` fills in and later stages can audit, instead of a verbal
|
|
53
|
+
claim. Before delivering, confirm no `.claude/harness-state/contracts/*.json` is still
|
|
54
|
+
`"status": "pending"` or `"fail"`.
|
|
55
|
+
|
|
56
|
+
**4. Self-critique before delivery.** Read the final output as a skeptical
|
|
57
|
+
reviewer. For high-stakes deliverables, spawn `fable-verifier` cold. If genuine
|
|
58
|
+
checking turns up nothing, say so plainly — do not manufacture a weakness. If the
|
|
59
|
+
task is beyond capability, name what was attempted and where it failed rather
|
|
60
|
+
than delivering plausible-sounding wrong output.
|
|
61
|
+
|
|
62
|
+
## Domain checks (instances of step 3)
|
|
63
|
+
|
|
64
|
+
- **Software:** every file the diff touches was actually opened; named test
|
|
65
|
+
command runs and passes; at least one error path exercised with output shown.
|
|
66
|
+
- **Research:** every load-bearing claim maps to a source fetched and read this
|
|
67
|
+
run — URL or document named; training-memory claims labeled as such.
|
|
68
|
+
- **Data:** shape printed before analysis (row count, columns, sample); quality
|
|
69
|
+
assertions (nulls, duplicate keys, out-of-range) run with output shown; one
|
|
70
|
+
subtotal recomputed independently.
|
|
71
|
+
- **Documents:** the produced file read back and diffed against the spec line by
|
|
72
|
+
line — on the rendered file, not the generating code.
|
|
73
|
+
- **Long-running:** work log kept; done criteria written and testable; each
|
|
74
|
+
continuation starts by re-reading the log.
|
|
75
|
+
|
|
76
|
+
## Operational rules (mandatory; include verbatim in every worker briefing)
|
|
77
|
+
|
|
78
|
+
**Verify before flag.** Before flagging any problem — verify it actually exists.
|
|
79
|
+
Grep, diff, run it, or check the source directly. Never report a problem that
|
|
80
|
+
hasn't been confirmed present. Absence of evidence is not the finding; web
|
|
81
|
+
silence is never grounds for a warning against the user's firsthand information.
|
|
82
|
+
Confirm, then flag.
|
|
83
|
+
|
|
84
|
+
**Warning threshold.** Keep a running count of minor concerns. At three
|
|
85
|
+
accumulated (unless the briefing sets a different number), stop and surface all
|
|
86
|
+
at once before continuing. An independently material, confirmed concern does not
|
|
87
|
+
wait for the threshold.
|
|
88
|
+
|
|
89
|
+
**Find-and-replace safety.** Anchor substring replaces on word boundaries
|
|
90
|
+
(`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
|
|
91
|
+
targeted string-replace on a unique anchor; never bare unanchored sed. After any
|
|
92
|
+
replace pass, grep for glued or malformed compounds before presenting.
|