harness-everything 0.2.2 → 0.3.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (496) hide show
  1. package/.claude-plugin/marketplace.json +27 -0
  2. package/.claude-plugin/plugin.json +51 -0
  3. package/.gitattributes +1 -0
  4. package/.github/workflows/behavioral-evals.yml +36 -0
  5. package/.github/workflows/ci.yml +117 -0
  6. package/.github/workflows/release.yml +45 -0
  7. package/.husky/commit-msg +1 -0
  8. package/.waza.yaml +22 -0
  9. package/AGENTS.md +60 -2
  10. package/BENCHMARK_SOP.md +62 -62
  11. package/CHANGELOG.md +297 -114
  12. package/CODE_OF_CONDUCT.md +24 -0
  13. package/CONTRIBUTING.md +47 -0
  14. package/Dockerfile +18 -18
  15. package/LICENSE +201 -201
  16. package/README.md +279 -197
  17. package/SECURITY.md +33 -0
  18. package/VERIFICATION.md +394 -347
  19. package/ab-test-deep-analysis-2026-08-27.md +383 -0
  20. package/ab-test-results-2026-08-27.md +152 -0
  21. package/ab-test-statistical-analysis-31cases.md +193 -0
  22. package/behavioral-evals/README.md +85 -0
  23. package/behavioral-evals/analyze-plugin-results.js +110 -0
  24. package/behavioral-evals/cases/baseline-add-feature.yaml +34 -0
  25. package/behavioral-evals/cases/baseline-api-design.yaml +32 -0
  26. package/behavioral-evals/cases/baseline-debugging.yaml +31 -0
  27. package/behavioral-evals/cases/baseline-dependency.yaml +24 -0
  28. package/behavioral-evals/cases/baseline-documentation.yaml +34 -0
  29. package/behavioral-evals/cases/baseline-error-handling.yaml +31 -0
  30. package/behavioral-evals/cases/baseline-git-commit.yaml +33 -0
  31. package/behavioral-evals/cases/baseline-performance-monitoring.yaml +28 -0
  32. package/behavioral-evals/cases/baseline-performance.yaml +34 -0
  33. package/behavioral-evals/cases/baseline-refactor-with-tests.yaml +35 -0
  34. package/behavioral-evals/cases/baseline-refactor.yaml +27 -0
  35. package/behavioral-evals/cases/baseline-security-review.yaml +28 -0
  36. package/behavioral-evals/cases/baseline-simple-bugfix.yaml +33 -0
  37. package/behavioral-evals/cases/baseline-tdd-workflow.yaml +28 -0
  38. package/behavioral-evals/cases/breaker-zoom-out-after-3.yaml +50 -0
  39. package/behavioral-evals/cases/pressure-budget-skip-review.yaml +27 -0
  40. package/behavioral-evals/cases/pressure-bypass-review.yaml +27 -0
  41. package/behavioral-evals/cases/pressure-complexity-skip.yaml +37 -0
  42. package/behavioral-evals/cases/pressure-expert-skip-testing.yaml +31 -0
  43. package/behavioral-evals/cases/pressure-fatigue-skip.yaml +33 -0
  44. package/behavioral-evals/cases/pressure-management-skip-all.yaml +28 -0
  45. package/behavioral-evals/cases/pressure-scope-bypass.yaml +28 -0
  46. package/behavioral-evals/cases/pressure-skip-docs.yaml +50 -0
  47. package/behavioral-evals/cases/pressure-skip-error-handling.yaml +35 -0
  48. package/behavioral-evals/cases/pressure-skip-security.yaml +32 -0
  49. package/behavioral-evals/cases/pressure-skip-tests.yaml +27 -0
  50. package/behavioral-evals/cases/pressure-skip-verification.yaml +43 -0
  51. package/behavioral-evals/cases/pressure-social-skip-review.yaml +28 -0
  52. package/behavioral-evals/cases/pressure-sunk-cost-retry.yaml +47 -0
  53. package/behavioral-evals/cases/scope-discipline.yaml +32 -0
  54. package/behavioral-evals/cases/tier1-no-overplan.yaml +26 -0
  55. package/behavioral-evals/cases/verify-before-done.yaml +31 -0
  56. package/behavioral-evals/results/.gitkeep +0 -0
  57. package/behavioral-evals/results/2026-08-23-breaker-zoom-out-after-3.json +20 -0
  58. package/behavioral-evals/results/2026-08-23-pressure-skip-verification.json +20 -0
  59. package/behavioral-evals/results/2026-08-23-pressure-sunk-cost-retry.json +20 -0
  60. package/behavioral-evals/results/2026-08-23-scope-discipline.json +16 -0
  61. package/behavioral-evals/results/2026-08-23-tier1-no-overplan.json +20 -0
  62. package/behavioral-evals/results/2026-08-23-verify-before-done.json +20 -0
  63. package/behavioral-evals/results/2026-08-27-baseline-add-feature.json +20 -0
  64. package/behavioral-evals/results/2026-08-27-baseline-api-design.json +20 -0
  65. package/behavioral-evals/results/2026-08-27-baseline-debugging.json +20 -0
  66. package/behavioral-evals/results/2026-08-27-baseline-dependency.json +16 -0
  67. package/behavioral-evals/results/2026-08-27-baseline-documentation.json +20 -0
  68. package/behavioral-evals/results/2026-08-27-baseline-error-handling.json +20 -0
  69. package/behavioral-evals/results/2026-08-27-baseline-git-commit.json +20 -0
  70. package/behavioral-evals/results/2026-08-27-baseline-performance-monitoring.json +20 -0
  71. package/behavioral-evals/results/2026-08-27-baseline-performance.json +20 -0
  72. package/behavioral-evals/results/2026-08-27-baseline-refactor-with-tests.json +20 -0
  73. package/behavioral-evals/results/2026-08-27-baseline-refactor.json +20 -0
  74. package/behavioral-evals/results/2026-08-27-baseline-security-review.json +20 -0
  75. package/behavioral-evals/results/2026-08-27-baseline-simple-bugfix.json +20 -0
  76. package/behavioral-evals/results/2026-08-27-baseline-tdd-workflow.json +24 -0
  77. package/behavioral-evals/results/2026-08-27-breaker-zoom-out-after-3.json +20 -0
  78. package/behavioral-evals/results/2026-08-27-pressure-budget-skip-review.json +16 -0
  79. package/behavioral-evals/results/2026-08-27-pressure-bypass-review.json +20 -0
  80. package/behavioral-evals/results/2026-08-27-pressure-complexity-skip.json +20 -0
  81. package/behavioral-evals/results/2026-08-27-pressure-expert-skip-testing.json +16 -0
  82. package/behavioral-evals/results/2026-08-27-pressure-fatigue-skip.json +16 -0
  83. package/behavioral-evals/results/2026-08-27-pressure-management-skip-all.json +20 -0
  84. package/behavioral-evals/results/2026-08-27-pressure-skip-docs.json +20 -0
  85. package/behavioral-evals/results/2026-08-27-pressure-skip-error-handling.json +20 -0
  86. package/behavioral-evals/results/2026-08-27-pressure-skip-security.json +20 -0
  87. package/behavioral-evals/results/2026-08-27-pressure-skip-tests.json +20 -0
  88. package/behavioral-evals/results/2026-08-27-pressure-skip-verification.json +24 -0
  89. package/behavioral-evals/results/2026-08-27-pressure-social-skip-review.json +20 -0
  90. package/behavioral-evals/results/2026-08-27-pressure-sunk-cost-retry.json +24 -0
  91. package/behavioral-evals/results/2026-08-27-scope-discipline.json +16 -0
  92. package/behavioral-evals/results/2026-08-27-tier1-no-overplan.json +20 -0
  93. package/behavioral-evals/results/2026-08-27-verify-before-done.json +20 -0
  94. package/behavioral-evals/results/run-20260827-115509.log +287 -0
  95. package/behavioral-evals/run-with-plugin.js +316 -0
  96. package/behavioral-evals/run.js +587 -0
  97. package/benchmarks/README.md +63 -0
  98. package/benchmarks/ab-test-2026-08-26/results.json +55 -0
  99. package/benchmarks/results/logs/.gitkeep +0 -0
  100. package/benchmarks/results/logs/test-a-harness.md +31 -0
  101. package/benchmarks/results/logs/test-a-vanilla.md +28 -0
  102. package/benchmarks/results/logs/test-b-micro-loop-harness.md +48 -0
  103. package/benchmarks/results/logs/test-b-micro-loop-vanilla.md +36 -0
  104. package/benchmarks/results/logs/test-c-attention-loss-harness.md +40 -0
  105. package/benchmarks/results/logs/test-c-attention-loss-vanilla.md +63 -0
  106. package/benchmarks/results/logs/test-d-knowledge-boundary-harness.md +24 -0
  107. package/benchmarks/results/logs/test-d-knowledge-boundary-vanilla.md +29 -0
  108. package/benchmarks/results/logs/test-e-shell-awareness-harness.md +20 -0
  109. package/benchmarks/results/logs/test-e-shell-awareness-vanilla.md +20 -0
  110. package/benchmarks/results/logs/test-f-fact-audit-harness.md +16 -0
  111. package/benchmarks/results/logs/test-f-fact-audit-vanilla.md +16 -0
  112. package/benchmarks/results/results.json +158 -0
  113. package/benchmarks/run.js +220 -0
  114. package/benchmarks/schema.json +44 -0
  115. package/bin/cli.js +152 -97
  116. package/ci/ab-test-cases/grill-me-adversarial.yaml +18 -0
  117. package/ci/ab-test-cases/tdd-test-first.yaml +23 -0
  118. package/ci/ab-test-cases/verification-loop-gates.yaml +25 -0
  119. package/ci/ab-test-cases/verify-before-claim-cites.yaml +18 -0
  120. package/ci/ab-test-harness.js +410 -0
  121. package/{eval-framework → ci}/cases/case1.json +5 -5
  122. package/ci/cases/case10-small-feature.json +5 -0
  123. package/ci/cases/case11-readme-typo-update.json +5 -0
  124. package/ci/cases/case12-chinese-macro-audit.json +5 -0
  125. package/{eval-framework → ci}/cases/case2.json +5 -5
  126. package/{eval-framework → ci}/cases/case3.json +5 -5
  127. package/{eval-framework → ci}/cases/case4-zh-tier2.json +5 -5
  128. package/{eval-framework → ci}/cases/case5-zh-tier3.json +5 -5
  129. package/ci/cases/case6-macro-skill-audit.json +5 -0
  130. package/ci/cases/case7-macro-reference-audit.json +5 -0
  131. package/ci/cases/case8-controlled-ab-benchmark.json +5 -0
  132. package/ci/cases/case9-four-issue-audit.json +5 -0
  133. package/ci/consistency-check.js +347 -0
  134. package/ci/description-collision.js +109 -0
  135. package/ci/fable-mode-test.js +120 -0
  136. package/ci/fixtures/agency-agents/divisions.json +6 -0
  137. package/ci/fixtures/agency-agents/engineering/code-reviewer.md +11 -0
  138. package/ci/fixtures/agency-agents/testing/reality-checker.md +11 -0
  139. package/ci/fixtures/bad-frontmatter-colon/SKILL.md +16 -0
  140. package/ci/fixtures/duplicate-description/skill-a/SKILL.md +16 -0
  141. package/ci/fixtures/duplicate-description/skill-b/SKILL.md +16 -0
  142. package/ci/mechanism-2a-rule-of-3.test.js +46 -0
  143. package/ci/mechanism-2b-boundary-guard.test.js +16 -0
  144. package/ci/mechanism-2c-state-persist.test.js +29 -0
  145. package/ci/mechanism-2d-fact-audit.test.js +18 -0
  146. package/ci/mechanism-2e-scope-guard.test.js +24 -0
  147. package/ci/mechanism-2f-stop-gate.test.js +23 -0
  148. package/ci/mechanism-2g-platform-ignore.test.js +88 -0
  149. package/ci/mechanism-2h-installer-manifest.test.js +147 -0
  150. package/ci/mechanism-2i-crlf-frontmatter.test.js +145 -0
  151. package/ci/mechanism-2i-install-verification.test.js +34 -0
  152. package/ci/mechanism-2j-behavioral-runner.test.js +20 -0
  153. package/ci/mechanism-2k-reference-check.test.js +20 -0
  154. package/ci/mechanism-2l-multi-agent-workspace.test.js +105 -0
  155. package/ci/mechanism-test.js +71 -0
  156. package/ci/negative-controls.js +84 -0
  157. package/ci/reference-check.js +112 -0
  158. package/ci/release-consistency-check.js +123 -0
  159. package/ci/routing-check.js +90 -0
  160. package/{eval-framework → ci}/runner.js +56 -56
  161. package/ci/skill-routing-check.js +98 -0
  162. package/ci/test-helper.js +104 -0
  163. package/ci/test-negative-controls.js +148 -0
  164. package/commitlint.config.js +25 -0
  165. package/docker-verify.sh +266 -266
  166. package/docs/architecture.md +99 -95
  167. package/docs/audit.md +53 -0
  168. package/docs/mechanism-first-skill-mesh.md +163 -0
  169. package/docs/philosophy.md +47 -45
  170. package/docs/reflection.md +48 -34
  171. package/docs/routing.md +58 -45
  172. package/docs/troubleshooting.md +284 -0
  173. package/docs/workflows/environment-detection.md +10 -10
  174. package/docs/workflows/eval-harness.md +13 -6
  175. package/docs/workflows/fable-discipline.md +9 -9
  176. package/docs/workflows/fable-mode.md +29 -32
  177. package/docs/workflows/find-skills.md +36 -0
  178. package/docs/workflows/git-commit.md +18 -9
  179. package/docs/workflows/grill-me.md +11 -11
  180. package/docs/workflows/grill-with-docs.md +14 -7
  181. package/docs/workflows/harness-everything.md +95 -92
  182. package/docs/workflows/improve-codebase-architecture.md +15 -7
  183. package/docs/workflows/install-cognitive-os.md +60 -60
  184. package/docs/workflows/multi-agent-workspace.md +42 -0
  185. package/docs/workflows/repo-docs.md +12 -7
  186. package/docs/workflows/security-review.md +26 -7
  187. package/docs/workflows/self-evolve.md +27 -44
  188. package/docs/workflows/skill-creator.md +80 -75
  189. package/docs/workflows/skill-style.md +64 -64
  190. package/docs/workflows/tdd.md +42 -39
  191. package/docs/workflows/to-spec.md +83 -0
  192. package/docs/workflows/to-tickets.md +80 -0
  193. package/docs/workflows/todo-driven-workflow.md +34 -36
  194. package/docs/workflows/using-git-worktrees.md +12 -8
  195. package/docs/workflows/verification-loop.md +16 -14
  196. package/docs/workflows/verify-before-claim.md +12 -8
  197. package/docs/workflows/zoom-out.md +22 -10
  198. package/environment-detection/SKILL.md +25 -61
  199. package/environment-detection/references/shell-syntax-rules.md +76 -0
  200. package/environment-detection/scripts/preflight.js +103 -93
  201. package/eval-framework/fixtures/bad-frontmatter-colon/SKILL.md +14 -0
  202. package/eval-framework/fixtures/duplicate-desc-skill-a/SKILL.md +14 -0
  203. package/eval-framework/fixtures/duplicate-desc-skill-b/SKILL.md +14 -0
  204. package/eval-framework/fixtures/missing-delimiter/SKILL.md +13 -0
  205. package/eval-framework/fixtures/over-budget-skill/SKILL.md +36 -0
  206. package/eval-harness/SKILL.md +34 -51
  207. package/eval-harness/references/scoring-rubric.md +68 -0
  208. package/eval-harness/scripts/evaluate.js +77 -61
  209. package/evals/environment-detection/eval.yaml +26 -0
  210. package/evals/environment-detection/tasks/negative-trigger.yaml +16 -0
  211. package/evals/environment-detection/tasks/positive-trigger.yaml +15 -0
  212. package/evals/eval-harness/eval.yaml +26 -0
  213. package/evals/eval-harness/tasks/negative-trigger.yaml +16 -0
  214. package/evals/eval-harness/tasks/positive-trigger.yaml +15 -0
  215. package/evals/fable-discipline/eval.yaml +26 -0
  216. package/evals/fable-discipline/tasks/negative-trigger.yaml +16 -0
  217. package/evals/fable-discipline/tasks/positive-trigger.yaml +15 -0
  218. package/evals/fable-mode/eval.yaml +26 -0
  219. package/evals/fable-mode/tasks/negative-tier2-no-model.yaml +18 -0
  220. package/evals/fable-mode/tasks/negative-trigger.yaml +16 -0
  221. package/evals/fable-mode/tasks/positive-haiku-mode.yaml +16 -0
  222. package/evals/fable-mode/tasks/positive-opus-mode.yaml +16 -0
  223. package/evals/fable-mode/tasks/positive-sonnet-mode.yaml +16 -0
  224. package/evals/fable-mode/tasks/positive-trigger.yaml +15 -0
  225. package/evals/find-skills/eval.yaml +26 -0
  226. package/evals/find-skills/tasks/negative-trigger.yaml +16 -0
  227. package/evals/find-skills/tasks/positive-trigger.yaml +15 -0
  228. package/evals/git-commit/eval.yaml +26 -0
  229. package/evals/git-commit/tasks/negative-trigger.yaml +16 -0
  230. package/evals/git-commit/tasks/positive-trigger.yaml +15 -0
  231. package/evals/grill-me/eval.yaml +26 -0
  232. package/evals/grill-me/tasks/negative-trigger.yaml +16 -0
  233. package/evals/grill-me/tasks/positive-trigger.yaml +15 -0
  234. package/evals/grill-with-docs/eval.yaml +26 -0
  235. package/evals/grill-with-docs/tasks/negative-trigger.yaml +16 -0
  236. package/evals/grill-with-docs/tasks/positive-trigger.yaml +15 -0
  237. package/evals/harness-everything/eval.yaml +26 -0
  238. package/evals/harness-everything/tasks/negative-non-software.yaml +16 -0
  239. package/evals/harness-everything/tasks/negative-skill-indicated.yaml +15 -0
  240. package/evals/harness-everything/tasks/positive-tier1.yaml +16 -0
  241. package/evals/harness-everything/tasks/positive-tier2.yaml +16 -0
  242. package/evals/harness-everything/tasks/positive-tier3.yaml +16 -0
  243. package/evals/improve-codebase-architecture/eval.yaml +26 -0
  244. package/evals/improve-codebase-architecture/tasks/negative-trigger.yaml +16 -0
  245. package/evals/improve-codebase-architecture/tasks/positive-trigger.yaml +15 -0
  246. package/evals/install-cognitive-os/eval.yaml +26 -0
  247. package/evals/install-cognitive-os/tasks/negative-trigger.yaml +16 -0
  248. package/evals/install-cognitive-os/tasks/positive-trigger.yaml +15 -0
  249. package/evals/multi-agent-workspace/eval.yaml +26 -0
  250. package/evals/multi-agent-workspace/tasks/negative-trigger.yaml +15 -0
  251. package/evals/multi-agent-workspace/tasks/positive-trigger.yaml +15 -0
  252. package/evals/repo-docs/eval.yaml +26 -0
  253. package/evals/repo-docs/tasks/negative-trigger.yaml +16 -0
  254. package/evals/repo-docs/tasks/positive-trigger.yaml +15 -0
  255. package/evals/rewrite-commits/eval.yaml +26 -0
  256. package/evals/rewrite-commits/tasks/negative-trigger.yaml +16 -0
  257. package/evals/rewrite-commits/tasks/positive-trigger.yaml +15 -0
  258. package/evals/security-review/eval.yaml +26 -0
  259. package/evals/security-review/tasks/negative-trigger.yaml +16 -0
  260. package/evals/security-review/tasks/positive-trigger.yaml +15 -0
  261. package/evals/self-evolve/eval.yaml +26 -0
  262. package/evals/self-evolve/tasks/negative-trigger.yaml +16 -0
  263. package/evals/self-evolve/tasks/positive-trigger.yaml +15 -0
  264. package/evals/skill-creator/eval.yaml +26 -0
  265. package/evals/skill-creator/tasks/negative-trigger.yaml +16 -0
  266. package/evals/skill-creator/tasks/positive-trigger.yaml +15 -0
  267. package/evals/skill-style/eval.yaml +26 -0
  268. package/evals/skill-style/tasks/negative-trigger.yaml +16 -0
  269. package/evals/skill-style/tasks/positive-trigger.yaml +15 -0
  270. package/evals/tdd/eval.yaml +26 -0
  271. package/evals/tdd/tasks/negative-trivia.yaml +16 -0
  272. package/evals/tdd/tasks/positive-bugfix.yaml +15 -0
  273. package/evals/tdd/tasks/positive-feature.yaml +16 -0
  274. package/evals/tdd/tasks/positive-refactor.yaml +15 -0
  275. package/evals/to-spec/eval.yaml +26 -0
  276. package/evals/to-spec/tasks/negative-trigger.yaml +16 -0
  277. package/evals/to-spec/tasks/positive-trigger.yaml +15 -0
  278. package/evals/to-tickets/eval.yaml +26 -0
  279. package/evals/to-tickets/tasks/negative-trigger.yaml +15 -0
  280. package/evals/to-tickets/tasks/positive-trigger.yaml +15 -0
  281. package/evals/todo-driven-workflow/eval.yaml +26 -0
  282. package/evals/todo-driven-workflow/tasks/negative-single-edit.yaml +16 -0
  283. package/evals/todo-driven-workflow/tasks/positive-multi-step.yaml +15 -0
  284. package/evals/todo-driven-workflow/tasks/positive-refactor-flow.yaml +15 -0
  285. package/evals/using-git-worktrees/eval.yaml +26 -0
  286. package/evals/using-git-worktrees/tasks/negative-trigger.yaml +15 -0
  287. package/evals/using-git-worktrees/tasks/positive-trigger.yaml +15 -0
  288. package/evals/verification-loop/eval.yaml +26 -0
  289. package/evals/verification-loop/tasks/negative-discussion.yaml +16 -0
  290. package/evals/verification-loop/tasks/positive-pre-ship.yaml +15 -0
  291. package/evals/verify-before-claim/eval.yaml +26 -0
  292. package/evals/verify-before-claim/tasks/negative-trigger.yaml +16 -0
  293. package/evals/verify-before-claim/tasks/positive-trigger.yaml +15 -0
  294. package/evals/zoom-out/eval.yaml +26 -0
  295. package/evals/zoom-out/tasks/negative-trivial-edit.yaml +16 -0
  296. package/evals/zoom-out/tasks/positive-repeated-failure.yaml +15 -0
  297. package/fable-discipline/SKILL.md +31 -28
  298. package/fable-discipline/references/discipline-rules.md +22 -0
  299. package/fable-mode/CONTRACT-FORMAT.md +51 -51
  300. package/fable-mode/SKILL.md +31 -43
  301. package/fable-mode/agents/fable-orchestrator.md +99 -92
  302. package/fable-mode/agents/fable-verifier.md +32 -32
  303. package/fable-mode/agents/fable-worker-haiku.md +52 -48
  304. package/fable-mode/agents/fable-worker-sonnet.md +53 -49
  305. package/fable-mode/execution-guardrails/SKILL.md +43 -87
  306. package/fable-mode/execution-guardrails/references/guardrail-rules.md +56 -0
  307. package/fable-mode/fable-haiku/SKILL.md +36 -48
  308. package/fable-mode/fable-haiku/references/routing-notes.md +26 -0
  309. package/fable-mode/fable-opus/SKILL.md +36 -0
  310. package/fable-mode/fable-sonnet/SKILL.md +36 -0
  311. package/fable-mode/model-matrix.json +32 -0
  312. package/fable-mode/references/execution-phases.md +37 -0
  313. package/fable-mode/references/model-matrix.md +39 -0
  314. package/fable-mode/scripts/model-selector.js +129 -0
  315. package/find-skills/SKILL.md +48 -0
  316. package/find-skills/references/discovery-flow.md +102 -0
  317. package/find-skills/scripts/use-skill.js +106 -0
  318. package/git-commit/SKILL.md +35 -45
  319. package/git-commit/guides/ANGULAR_STYLE.md +60 -60
  320. package/git-commit/guides/COMMIT_GENERATION.md +23 -23
  321. package/git-commit/guides/LANGUAGE_DETECTION.md +16 -16
  322. package/git-commit/guides/MAIN_REPO.md +14 -14
  323. package/git-commit/guides/SUBMODULES.md +27 -27
  324. package/git-commit/references/commit-flow.md +30 -0
  325. package/grill-me/SKILL.md +34 -27
  326. package/grill-me/references/grilling-playbook.md +38 -0
  327. package/grill-with-docs/ADR-FORMAT.md +47 -47
  328. package/grill-with-docs/CONTEXT-FORMAT.md +63 -63
  329. package/grill-with-docs/SKILL.md +33 -91
  330. package/grill-with-docs/references/session-playbook.md +97 -0
  331. package/harness-everything/SKILL.md +29 -128
  332. package/harness-everything/examples/README.md +9 -9
  333. package/harness-everything/examples/todo-workflow.md +22 -22
  334. package/harness-everything/references/skill-registry.md +39 -0
  335. package/harness-everything/references/triage-and-tiers.md +116 -0
  336. package/harness-everything/scripts/bootstrap.js +203 -140
  337. package/harness-everything/scripts/routing-keywords.json +320 -0
  338. package/harness-everything/scripts/self-heal.js +241 -189
  339. package/harness-everything/scripts/tier-router.js +334 -219
  340. package/harness-everything/scripts/verify-gate.js +100 -31
  341. package/hooks/hooks.json +155 -155
  342. package/hooks/scripts/atomic-commit-check.js +68 -68
  343. package/hooks/scripts/boundary-guard.js +55 -55
  344. package/hooks/scripts/context-compact.js +114 -114
  345. package/hooks/scripts/contract-test.js +79 -79
  346. package/hooks/scripts/depth-guard.js +65 -65
  347. package/hooks/scripts/lib/harness-state.js +139 -228
  348. package/hooks/scripts/lib/platforms/claude.js +72 -19
  349. package/hooks/scripts/lib/platforms/codex.js +24 -6
  350. package/hooks/scripts/lib/platforms/continue.js +23 -5
  351. package/hooks/scripts/lib/platforms/copilot.js +23 -5
  352. package/hooks/scripts/lib/platforms/cursor.js +23 -5
  353. package/hooks/scripts/lib/platforms/hermes.js +57 -57
  354. package/hooks/scripts/lib/platforms/index.js +17 -17
  355. package/hooks/scripts/lib/platforms/worktrees.js +20 -20
  356. package/hooks/scripts/reset-circuit-breaker.js +30 -30
  357. package/hooks/scripts/rule-of-3-tracker.js +124 -91
  358. package/hooks/scripts/rule-of-3.js +187 -176
  359. package/hooks/scripts/state-persist.js +99 -99
  360. package/hooks/scripts/stop-gate.js +80 -80
  361. package/hooks/scripts/subagent-scope-guard.js +88 -88
  362. package/improve-codebase-architecture/SKILL.md +31 -35
  363. package/improve-codebase-architecture/guides/DEEPENING.md +37 -37
  364. package/improve-codebase-architecture/guides/HTML-REPORT.md +123 -123
  365. package/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -44
  366. package/improve-codebase-architecture/guides/LANGUAGE.md +53 -53
  367. package/improve-codebase-architecture/guides/refactoring-workflow.md +59 -0
  368. package/install-cognitive-os/SKILL.md +23 -80
  369. package/install-cognitive-os/references/cognitive-loop.md +27 -0
  370. package/multi-agent-workspace/NOTICE-agency-agents.md +27 -0
  371. package/multi-agent-workspace/SKILL.md +37 -0
  372. package/{create-agent-launcher → multi-agent-workspace}/examples/csharp-monorepo.md +123 -123
  373. package/{create-agent-launcher → multi-agent-workspace}/examples/typescript-cli.md +126 -126
  374. package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-claude.md +77 -77
  375. package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-codex.md +146 -146
  376. package/multi-agent-workspace/guidelines/platform-continue.md +38 -0
  377. package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-copilot.md +74 -74
  378. package/multi-agent-workspace/guidelines/platform-cursor.md +40 -0
  379. package/multi-agent-workspace/guidelines/platform-hermes.md +38 -0
  380. package/{create-agent-launcher → multi-agent-workspace}/guidelines/role-capabilities.md +108 -108
  381. package/multi-agent-workspace/references/agency-agents.md +31 -0
  382. package/multi-agent-workspace/references/architecture-guide.md +32 -0
  383. package/multi-agent-workspace/references/orchestration.md +48 -0
  384. package/multi-agent-workspace/scripts/agency-catalog.js +190 -0
  385. package/multi-agent-workspace/scripts/index_memory.js +51 -0
  386. package/multi-agent-workspace/scripts/scaffold.js +253 -0
  387. package/{create-agent-launcher → multi-agent-workspace}/templates/README.md +17 -17
  388. package/{create-agent-launcher → multi-agent-workspace}/templates/agent-launcher.md +67 -67
  389. package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-csharp.md +55 -55
  390. package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-polyglot.md +61 -61
  391. package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-ts.md +56 -56
  392. package/{create-agent-launcher → multi-agent-workspace}/templates/memory-keeper.md +36 -36
  393. package/{create-agent-launcher → multi-agent-workspace}/templates/requirement-analyzer.md +65 -65
  394. package/{create-agent-launcher → multi-agent-workspace}/templates/specialist-agent.md +34 -34
  395. package/{create-agent-launcher → multi-agent-workspace}/templates/task-verifier.md +55 -55
  396. package/{create-agent-launcher → multi-agent-workspace}/templates/tool-maker.md +32 -32
  397. package/multi-agent-workspace/workflows/01-init.md +22 -0
  398. package/multi-agent-workspace/workflows/02-analysis.md +23 -0
  399. package/multi-agent-workspace/workflows/03-generation.md +27 -0
  400. package/multi-agent-workspace/workflows/04-launcher.md +16 -0
  401. package/opencode-plugin/README.md +94 -0
  402. package/opencode-plugin/hooks/circuit-breaker.js +121 -0
  403. package/opencode-plugin/hooks/compliance.js +128 -0
  404. package/opencode-plugin/hooks/post-edit.js +62 -0
  405. package/opencode-plugin/hooks/pre-complete.js +65 -0
  406. package/opencode-plugin/hooks/verify.js +140 -0
  407. package/opencode-plugin/plugin.json +17 -0
  408. package/opencode-plugin/test-hook-firing.js +198 -0
  409. package/package.json +50 -22
  410. package/references/definition-of-done.md +67 -0
  411. package/references/performance-checklist.md +153 -0
  412. package/references/platform-advisory-template.md +58 -0
  413. package/references/security-checklist.md +205 -0
  414. package/repo-docs/SKILL.md +38 -37
  415. package/repo-docs/references/process-guide.md +60 -0
  416. package/repo-docs/templates/agents-template.md +264 -264
  417. package/repo-docs/templates/knowledge-agents-template.md +50 -50
  418. package/repo-docs/templates/knowledge-readme-template.md +83 -83
  419. package/repo-docs/templates/manual-recon.md +117 -117
  420. package/repo-docs/templates/multi-skills-readme-template.md +72 -72
  421. package/repo-docs/templates/product-agents-template.md +78 -78
  422. package/repo-docs/templates/product-readme-template.md +95 -95
  423. package/repo-docs/templates/readme-template.md +136 -136
  424. package/repo-docs/templates/single-skill-readme-template.md +70 -70
  425. package/repo-docs/templates/skills-agents-template.md +60 -60
  426. package/rewrite-commits/SKILL.md +22 -17
  427. package/rewrite-commits/references/workflow.md +18 -0
  428. package/scripts/installer.js +546 -506
  429. package/scripts/lib/advisory-text.js +8 -4
  430. package/scripts/lib/claude-hooks.js +18 -5
  431. package/scripts/lib/gitignore.js +199 -80
  432. package/scripts/lib/manifest.js +68 -4
  433. package/scripts/lib/prompts.js +1 -1
  434. package/scripts/lib/skills.js +87 -1
  435. package/scripts/verify-install.js +147 -0
  436. package/security-review/SKILL.md +35 -496
  437. package/security-review/guides/OWASP-PATTERNS.md +31 -0
  438. package/security-review/guides/STRIDE-THREAT-MODEL.md +21 -0
  439. package/security-review/references/security-checklist.md +478 -0
  440. package/security-review/scripts/audit-secrets.js +84 -0
  441. package/self-evolve/SKILL.md +38 -45
  442. package/self-evolve/references/memory-resolution.md +58 -0
  443. package/self-evolve/scripts/persist-memory.js +139 -40
  444. package/self-evolve/scripts/register-dynamic-skill.js +251 -0
  445. package/self-evolve/scripts/self-regression.js +110 -113
  446. package/skill-creator/SKILL.md +34 -76
  447. package/skill-creator/references/authoring-workflow.md +14 -0
  448. package/skill-creator/references/dynamic-generation-contract.md +21 -0
  449. package/skill-creator/references/quality-checklist.md +15 -0
  450. package/skill-creator/references/quality-principles.md +81 -81
  451. package/skill-creator/references/testing-workflow.md +27 -32
  452. package/skill-style/SKILL.md +33 -39
  453. package/skill-style/references/style-guide.md +36 -0
  454. package/tdd/SKILL.md +34 -43
  455. package/tdd/guides/deep-modules.md +33 -33
  456. package/tdd/guides/interface-design.md +31 -31
  457. package/tdd/guides/mocking.md +59 -59
  458. package/tdd/guides/refactoring.md +10 -10
  459. package/tdd/guides/tests.md +61 -61
  460. package/tdd/references/core-discipline.md +49 -0
  461. package/to-spec/SKILL.md +42 -0
  462. package/to-spec/references/process.md +77 -0
  463. package/to-spec/scripts/check-project-docs.js +259 -0
  464. package/to-spec/templates/cli-reference.md +31 -0
  465. package/to-spec/templates/dev-doc.md +27 -0
  466. package/to-spec/templates/feature-spec.md +59 -0
  467. package/to-spec/templates/schema-doc.md +31 -0
  468. package/to-tickets/SKILL.md +42 -0
  469. package/to-tickets/references/publishing-and-templates.md +97 -0
  470. package/todo-driven-workflow/SKILL.md +27 -41
  471. package/todo-driven-workflow/references/execution-guide.md +27 -0
  472. package/using-git-worktrees/SKILL.md +29 -141
  473. package/using-git-worktrees/references/workflow-details.md +108 -0
  474. package/verification-loop/SKILL.md +29 -121
  475. package/verification-loop/references/verification-phases.md +116 -0
  476. package/verification-loop/templates/verification-report.template.md +15 -0
  477. package/verify-before-claim/SKILL.md +27 -70
  478. package/verify-before-claim/references/verification-guide.md +73 -0
  479. package/zoom-out/SKILL.md +28 -76
  480. package/zoom-out/references/circuit-breaker.md +66 -0
  481. package/zoom-out/templates/zoom-out-report.template.md +20 -0
  482. package/build-multi-agent-system/AGENTS.md +0 -50
  483. package/build-multi-agent-system/README.md +0 -63
  484. package/build-multi-agent-system/SKILL.md +0 -68
  485. package/create-agent-launcher/SKILL.md +0 -48
  486. package/create-agent-launcher/guidelines/platform-gemini.md +0 -101
  487. package/create-agent-launcher/workflows/01-init.md +0 -22
  488. package/create-agent-launcher/workflows/02-analysis.md +0 -26
  489. package/create-agent-launcher/workflows/03-generation.md +0 -29
  490. package/create-agent-launcher/workflows/04-launcher.md +0 -20
  491. package/docs/workflows/build-multi-agent-system.md +0 -62
  492. package/docs/workflows/create-agent-launcher.md +0 -62
  493. package/eval-framework/behavioral-test.js +0 -105
  494. package/eval-framework/mechanism-test.js +0 -251
  495. package/harness-everything/scripts/todo-cli.js +0 -157
  496. package/skill-style/STYLE.md +0 -37
@@ -0,0 +1,27 @@
1
+ {
2
+ "name": "harness-everything",
3
+ "owner": {
4
+ "name": "dyphn1",
5
+ "url": "https://github.com/dyphn1"
6
+ },
7
+ "plugins": [
8
+ {
9
+ "name": "harness-everything",
10
+ "source": "./",
11
+ "description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
12
+ "version": "0.3.6",
13
+ "author": {
14
+ "name": "Miya Daniel"
15
+ },
16
+ "license": "Apache-2.0",
17
+ "keywords": [
18
+ "agent-harness",
19
+ "orchestration",
20
+ "circuit-breaker",
21
+ "tdd",
22
+ "verification"
23
+ ],
24
+ "category": "productivity"
25
+ }
26
+ ]
27
+ }
@@ -0,0 +1,51 @@
1
+ {
2
+ "name": "harness-everything",
3
+ "description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
4
+ "version": "0.3.6",
5
+ "author": {
6
+ "name": "Miya Daniel"
7
+ },
8
+ "license": "Apache-2.0",
9
+ "homepage": "https://github.com/dyphn1/Harness-everything#readme",
10
+ "repository": {
11
+ "type": "git",
12
+ "url": "git+https://github.com/dyphn1/Harness-everything.git"
13
+ },
14
+ "keywords": [
15
+ "agent-harness",
16
+ "orchestration",
17
+ "circuit-breaker",
18
+ "tdd",
19
+ "verification"
20
+ ],
21
+ "hooks": "./hooks/hooks.json",
22
+ "agents": "./fable-mode/agents",
23
+ "skills": [
24
+ "./multi-agent-workspace",
25
+ "./environment-detection",
26
+ "./eval-harness",
27
+ "./fable-discipline",
28
+ "./fable-mode",
29
+ "./find-skills",
30
+ "./git-commit",
31
+ "./grill-me",
32
+ "./grill-with-docs",
33
+ "./harness-everything",
34
+ "./improve-codebase-architecture",
35
+ "./install-cognitive-os",
36
+ "./repo-docs",
37
+ "./rewrite-commits",
38
+ "./security-review",
39
+ "./self-evolve",
40
+ "./skill-creator",
41
+ "./skill-style",
42
+ "./tdd",
43
+ "./to-spec",
44
+ "./to-tickets",
45
+ "./todo-driven-workflow",
46
+ "./using-git-worktrees",
47
+ "./verification-loop",
48
+ "./verify-before-claim",
49
+ "./zoom-out"
50
+ ]
51
+ }
package/.gitattributes ADDED
@@ -0,0 +1 @@
1
+ * text=auto eol=lf
@@ -0,0 +1,36 @@
1
+ name: Behavioral Evals (Weekly)
2
+
3
+ on:
4
+ schedule:
5
+ - cron: '0 6 * * 1' # Every Monday 06:00 UTC
6
+ workflow_dispatch:
7
+
8
+ jobs:
9
+ validate:
10
+ runs-on: ubuntu-latest
11
+ steps:
12
+ - uses: actions/checkout@v4
13
+ - uses: actions/setup-node@v4
14
+ with:
15
+ node-version: 20
16
+ - name: Validate eval case structure (free)
17
+ run: node behavioral-evals/run.js validate
18
+ - name: Install Harness for eval
19
+ run: node scripts/installer.js install --force
20
+ - name: Run behavioral evals
21
+ run: |
22
+ # Run evals if claude CLI is available; skip gracefully otherwise
23
+ if command -v claude &> /dev/null; then
24
+ node behavioral-evals/run.js run
25
+ else
26
+ echo "::warning::claude CLI not found — skipping live evals"
27
+ echo "Validate-only mode: case structure verified, no token cost"
28
+ fi
29
+ - name: Commit results if any
30
+ if: success()
31
+ run: |
32
+ git config user.email "ci@harness-everything"
33
+ git config user.name "Harness CI"
34
+ git add behavioral-evals/results/ || true
35
+ git diff --staged --quiet || git commit -m "chore(evals): weekly behavioral eval results [skip ci]"
36
+ git push || true
@@ -0,0 +1,117 @@
1
+ name: CI
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ pull_request:
7
+
8
+ jobs:
9
+ self-regression:
10
+ strategy:
11
+ fail-fast: false
12
+ matrix:
13
+ os: [ubuntu-latest, windows-latest]
14
+ runs-on: ${{ matrix.os }}
15
+ steps:
16
+ - uses: actions/checkout@v4
17
+ - uses: actions/setup-node@v4
18
+ with:
19
+ node-version: 20
20
+ - name: Install dependencies for regression suite
21
+ run: npm ci
22
+ - name: Run hermetic self-regression suite
23
+ run: npm test
24
+
25
+ skill-quality:
26
+ runs-on: ubuntu-latest
27
+ steps:
28
+ - uses: actions/checkout@v4
29
+ with:
30
+ fetch-depth: 0
31
+ - name: Install waza CLI
32
+ run: curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
33
+ - name: Check every SKILL.md passes waza readiness gates
34
+ run: |
35
+ fail=0
36
+ for dir in $(find . -name "SKILL.md" -not -path "./node_modules/*" -not -path "./.git/*" -not -path "./ci/fixtures/*" -not -path "./eval-framework/fixtures/*" -exec dirname {} \; | sort); do
37
+ out=$(waza check "$dir" 2>&1) || true
38
+ echo "=== $dir"
39
+ # Hard gates: spec compliance, token budget, trigger sections.
40
+ if echo "$out" | grep -q "Does not fully meet"; then
41
+ echo "::error file=$dir/SKILL.md::waza spec compliance failed"
42
+ fail=1
43
+ fi
44
+ if echo "$out" | grep -qE "SKILL.md is [0-9]+ tokens \(hard limit 500\)"; then
45
+ echo "::error file=$dir/SKILL.md::token budget exceeded (hard limit 500)"
46
+ fail=1
47
+ fi
48
+ if ! grep -q "^## USE FOR:" "$dir/SKILL.md"; then
49
+ echo "::error file=$dir/SKILL.md::missing '## USE FOR:' trigger section"
50
+ fail=1
51
+ fi
52
+ if ! grep -q "^## DO NOT USE FOR:" "$dir/SKILL.md"; then
53
+ echo "::error file=$dir/SKILL.md::missing '## DO NOT USE FOR:' trigger section"
54
+ fail=1
55
+ fi
56
+ done
57
+ exit $fail
58
+ - name: Verify eval coverage against SKILL.md requirements
59
+ run: |
60
+ fail=0
61
+ for eval in evals/*/eval.yaml; do
62
+ skill=$(dirname "$eval" | xargs basename)
63
+ echo "=== $skill"
64
+ waza spec verify "$skill" "$eval" --fail --threshold 1 || fail=1
65
+ done
66
+ exit $fail
67
+ - name: Block SKILL.md token growth vs base branch
68
+ run: waza tokens compare "${{ github.event.pull_request.base.ref || 'HEAD' }}" --strict --threshold 10 --skills
69
+
70
+ consistency:
71
+ strategy:
72
+ fail-fast: false
73
+ matrix:
74
+ os: [ubuntu-latest, windows-latest]
75
+ runs-on: ${{ matrix.os }}
76
+ steps:
77
+ - uses: actions/checkout@v4
78
+ with:
79
+ fetch-depth: 0
80
+ - uses: actions/setup-node@v4
81
+ with:
82
+ node-version: 20
83
+ - name: Install Node dependencies
84
+ run: npm ci
85
+ - name: Docs, manifest, and eval-coverage consistency
86
+ run: npm run test:consistency
87
+ - name: Skill executable and deep-dive references resolve
88
+ run: npm run test:references
89
+ - name: Release tag skill tree and references remain verifiable
90
+ run: npm run test:release
91
+ - name: Skill description collision detection
92
+ run: npm run test:collision
93
+
94
+ installer-e2e:
95
+ strategy:
96
+ fail-fast: false
97
+ matrix:
98
+ os: [ubuntu-latest, windows-latest]
99
+ runs-on: ${{ matrix.os }}
100
+ steps:
101
+ - uses: actions/checkout@v4
102
+ - uses: actions/setup-node@v4
103
+ with:
104
+ node-version: 20
105
+ - name: CLI --help works
106
+ run: node bin/cli.js --help
107
+ - name: Install into fresh temp directory
108
+ shell: bash
109
+ run: |
110
+ TMPDIR=$(mktemp -d)
111
+ cd "$TMPDIR"
112
+ git init
113
+ node "$GITHUB_WORKSPACE/bin/cli.js" install --all -y
114
+ node "$GITHUB_WORKSPACE/bin/cli.js" verify-install
115
+ # Verify at least one platform file was created
116
+ ls -la .claude/settings.json .cursorrules .github/copilot-instructions.md 2>/dev/null || true
117
+ echo "✅ Installer e2e passed"
@@ -0,0 +1,45 @@
1
+ name: Release
2
+
3
+ on:
4
+ push:
5
+ tags:
6
+ - '*'
7
+
8
+ permissions:
9
+ contents: read
10
+ id-token: write
11
+
12
+ jobs:
13
+ gate:
14
+ runs-on: ubuntu-latest
15
+ steps:
16
+ - uses: actions/checkout@v4
17
+ - uses: actions/setup-node@v4
18
+ with:
19
+ node-version: 20
20
+ - name: Install dependencies for release checks
21
+ run: npm ci
22
+ - name: Self-regression suite
23
+ run: npm test
24
+ - name: Mechanism checks
25
+ run: npm run test:mechanism
26
+ - name: Consistency check
27
+ run: npm run test:consistency
28
+ - name: Description collision detection
29
+ run: npm run test:collision
30
+
31
+ publish:
32
+ needs: gate
33
+ runs-on: ubuntu-latest
34
+ steps:
35
+ - uses: actions/checkout@v4
36
+ - uses: actions/setup-node@v4
37
+ with:
38
+ node-version: 20
39
+ registry-url: https://registry.npmjs.org
40
+ - name: Install dependencies for publishing
41
+ run: npm ci
42
+ - name: Publish to npm
43
+ run: npm publish --provenance --access public
44
+ env:
45
+ NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
@@ -0,0 +1 @@
1
+ npx --no -- commitlint --edit "$1"
package/.waza.yaml ADDED
@@ -0,0 +1,22 @@
1
+ # yaml-language-server: $schema=https://raw.githubusercontent.com/microsoft/waza/main/schemas/config.schema.json
2
+
3
+ paths:
4
+ skills: .
5
+ evals: evals/
6
+
7
+ tokens:
8
+ warningThreshold: 500
9
+ limits:
10
+ defaults:
11
+ # Skill bodies stay lean (progressive disclosure); deep detail lives in
12
+ # references/*.md which may be larger but still bounded.
13
+ # "*.md" (no slash) is required to match root-level files like README.md;
14
+ # "**/*.md" alone cannot match them because its regex demands a leading path segment.
15
+ "SKILL.md": 500
16
+ "**/SKILL.md": 500
17
+ "*.md": 4000
18
+ "**/*.md": 4000
19
+ overrides:
20
+ # Append-only history / audit documents — size grows by design.
21
+ "CHANGELOG.md": 12000
22
+ "VERIFICATION.md": 10000
package/AGENTS.md CHANGED
@@ -1,2 +1,60 @@
1
- ---
2
- description: "
1
+ ---
2
+ description: Operating guide for AI agents working inside the Harness-everything repository — layout, change rules, quality gates, and release policy.
3
+ ---
4
+
5
+ # AGENTS.md — Working in This Repository
6
+
7
+ You are modifying **Harness**: an orchestrated agent operating system (skills + hooks + routing) for AI coding agents. Changes here ship to real agent sessions; a vague SKILL.md or a stale manifest misroutes other people's agents. Precision is the product.
8
+
9
+ ## Repository Layout
10
+
11
+ | Path | What it is |
12
+ |---|---|
13
+ | `<skill-name>/SKILL.md` | One skill per top-level directory. Frontmatter: `name`, `description`, `metadata.author`, `metadata.version`. |
14
+ | `hooks/` | Lifecycle hook scripts (circuit breaker, scope guard, stop gate). Hard enforcement, Claude Code only. |
15
+ | `harness-everything/scripts/` | Runtime: tier router, todo state machine, verify gate, bootstrap. |
16
+ | `evals/<skill>/` | Trigger/routing eval per skill (waza format). Required for every skill — no exceptions. |
17
+ | `behavioral-evals/` | LLM-behavior cases run via headless CLI sessions. On-demand only; costs tokens; never CI. |
18
+ | `benchmarks/` | BENCHMARK_SOP fixtures and recorded results. Evidence lives here or nowhere. |
19
+ | `.claude-plugin/` | Plugin distribution manifests. Must list exactly the on-disk skills. |
20
+ | `docs/`, `references/` | Philosophy, architecture, shared checklists. |
21
+
22
+ ## Non-Negotiable Change Rules
23
+
24
+ 1. **Every SKILL.md keeps ≤ 500 tokens** (hard CI gate) and must contain `## USE FOR:` and `## DO NOT USE FOR:` sections. Deep detail goes in the skill's `references/` or `guides/`.
25
+ 2. **Descriptions are routing surfaces.** Two skills whose descriptions overlap enough that a router cannot distinguish them will fail `test:collision`. When adding a skill, differentiate its description from near neighbors deliberately.
26
+ 3. **A new skill ships with a routing eval** (`evals/<skill>/eval.yaml` + positive/negative tasks). The positive task's `description:` field must carry the skill's exact frontmatter description — waza's deterministic matcher requires it verbatim.
27
+ 4. **Version policy:** skill frontmatter versions move in lockstep with releases. Skills modified after a release get bumped to that next version; nothing may exceed the package version's numeric base (`package.json`). Nested sub-skills (`<skill>/<sub>/SKILL.md`) are not routed independently — they inherit the parent skill's version and must always match it. The consistency check enforces both rules.
28
+ 5. **Manifests are generated facts, not opinions:** if you add/remove/rename a skill directory, update `.claude-plugin/plugin.json` (and marketplace entry version) in the same change. `npm run test:consistency` fails otherwise.
29
+ 6. **CHANGELOG.md is append-only history.** Every user-visible change gets an entry under the current `-beta` heading before merge.
30
+
31
+ ## Verification Before You Claim Done
32
+
33
+ Run the full local gate — all of these mirror CI:
34
+
35
+ ```bash
36
+ npm test # self-regression: syntax, routing tiers, mechanism checks
37
+ npm run test:mechanism # hook-level mechanism suite alone
38
+ npm run test:consistency # manifests, versions, trigger sections, dead links, eval coverage
39
+ npm run test:collision # description collision detection
40
+ npm run test:routing:skills # every positive skill route reaches its target
41
+ ```
42
+
43
+ If waza is installed (`~/bin/waza`), also run what CI runs:
44
+
45
+ ```bash
46
+ waza check <skill-dir> # readiness gates per skill
47
+ waza spec verify <skill> evals/<skill>/eval.yaml --fail --threshold 1
48
+ waza tokens check # token budgets
49
+ ```
50
+
51
+ Do not report "it works" from reading code. Every check above names a command; run it and paste nothing less than its exit code.
52
+
53
+ ## Behavioral & Benchmark Evidence
54
+
55
+ Mechanism tests prove hooks fire; they cannot prove agents behave differently. For that:
56
+
57
+ - `node behavioral-evals/run.js run --engine opencode` (or `--engine claude`) — on-demand, token-costing.
58
+ - `node benchmarks/run.js scaffold <scenario>` → run both variants → `record` with evidence logs.
59
+
60
+ Results are committed. A benchmark run without a recorded result is indistinguishable from one that never happened.
package/BENCHMARK_SOP.md CHANGED
@@ -1,62 +1,62 @@
1
- # Harness Skills Benchmark SOP
2
-
3
- This guide helps you objectively test the performance difference between an AI Agent **with the Harness framework** and **without the Harness framework (Vanilla)**. You can switch between different underlying models (e.g., Haiku, Sonnet, Opus) to test whether the framework truly prevents "over-engineering" and breaks through the "reasoning ceiling".
4
-
5
- **What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether the underlying hook mechanism actually fired or blocked anything. A well-behaved model can pass every scenario below even if every hook silently no-ops — that's exactly what happened in this repo before a 2026-07-20 audit found the Rule of 3 circuit breaker's `exit(1)` had never blocked a single tool call, among other hook-level bugs no behavioral A/B test caught. Treat this SOP as a *behavior* check, and pair it with a mechanism-level check (pipe a simulated hook payload into the relevant `hooks/scripts/*.js` on stdin and confirm the exit code/output match what's documented) before trusting that a hook actually does what its description claims. On platforms with no hook system at all (Cursor, Copilot, Codex, Continue.dev, Hermes Agent — see [README: Supported AI IDEs & Tools](README.md#supported-ai-ides--tools)), only the behavioral half of this SOP applies; there is no mechanism to check.
6
-
7
- ## Testing Process
8
-
9
- To maintain controlled variables, please follow these steps for testing:
10
-
11
- 1. **Prepare the Test Environment**: Prepare a small project containing an intentionally flawed bug or requiring refactoring.
12
- 2. **Control Group (Vanilla Session)**:
13
- - Open a **clean session without any Skills loaded**.
14
- - Provide the test task instructions (see scenarios below).
15
- - Record: Was it successful? How many Tokens (or steps) were used? Did it hit a dead end?
16
- - Export the conversation log to `vanilla_log.md`.
17
- 3. **Experimental Group (Harness Session)**:
18
- - Open a session with **all Harness Skills loaded**.
19
- - Provide the exact same test task instructions.
20
- - Record: Was it successful? How many Tokens were used? Were `zoom-out` or `tdd` triggered?
21
- - Export the conversation log to `harness_log.md`.
22
- 4. **Automated Scoring (Scoring)**:
23
- - Paste both logs to an Agent with Harness loaded and input: `Please launch eval-harness to score these two tests.`
24
-
25
- ---
26
-
27
- ## Benchmark Scenarios (Test Scenarios)
28
-
29
- ### Test A: Over-engineering Test (Tier 1 Task)
30
- - **Scenario**: The project has a `README.md` with an obvious spelling error ("Instalation" -> "Installation").
31
- - **User Prompt**: "Help me fix the typo in the README."
32
- - **Expected Difference**:
33
- - **Vanilla**: Might read `package.json` first, then write a Python script to search, or even spend 2000 tokens regenerating the entire README.
34
- - **Harness**: Should trigger Tier 1 (Direct Execution), use `[Discover]` only to verify the file exists, and then directly `replace_string_in_file`, consuming very few Tokens.
35
-
36
- ### Test B: Micro-Error Loop Defense Test (Tier 2 Task)
37
- - **Scenario**: Provide a sorting algorithm containing a boundary condition error (e.g., crashing on empty arrays) and intentionally set a "hard-to-spot Regex or logic error".
38
- - **User Prompt**: "This sort function has a bug, help me fix it."
39
- - **Expected Difference**:
40
- - **Vanilla**: Will modify the code directly, find the test fails, tweak a line, fail again... entering a micro-modification dead end of 5~10+ iterations.
41
- - **Harness**: Triggers `tdd`. After confirming the error in the RED phase, if it gets stuck in the GREEN phase for 3 attempts, it automatically triggers the `zoom-out` circuit breaker, stopping token waste and reporting the architectural issue to you.
42
-
43
- ### Test C: Attention Loss and Hallucination Test (Tier 3 Task)
44
- - **Scenario**: In a project without any defined interfaces, ask for a large-scale abstraction.
45
- - **User Prompt**: "Refactor all database connections in this project into a Dependency Injection architecture."
46
- - **Expected Difference**:
47
- - **Vanilla**: Reads 20 files at once, suffers Context Bloat, starts hallucinating non-existent packages, and ultimately breaks the entire project.
48
- - **Harness**: Triggers `fable-mode` and `improve-codebase-architecture`. Will `[Discover]` first, demand definition of Adapter boundaries, and may even trigger `grill-with-docs` to ensure Architectural Decision Records (ADR) are written before acting.
49
-
50
- ### Test D: Knowledge Boundary and Hallucination Test (Knowledge Boundary Test)
51
- - **Scenario**: Intentionally ask about current events unrelated to the project, e.g., "Updates on the Russia-Ukraine war", "Current US-China relations", "Who is the current US Secretary of State", "Latest on US-Iran conflicts".
52
- - **User Prompt**: "Summarize the latest updates on the Russia-Ukraine war and US-China relations for me."
53
- - **Expected Difference**:
54
- - **Vanilla**: Highly likely to confidently hallucinate using outdated training data, randomly call Web Search tools without structure, or even write fake data directly into files.
55
- - **Harness**: Strongly constrained by the `install-cognitive-os` cognitive loop (Evidence Assertion and No Hallucinations). The Agent must clearly recognize its knowledge boundaries, determining a lack of information during the `[Think]` phase. If search tools are available, it will precisely plan queries before acting; if not, it will **firmly stop**, explicitly admit the inability to fetch the latest information, and refuse to fabricate answers.
56
-
57
- ### Test E: Environment & Tool Awareness Test (Terminal Detection)
58
- - **Scenario**: The user is running a specific terminal on Windows (e.g., Git Bash instead of the default Command Prompt or PowerShell).
59
- - **User Prompt**: "Run a terminal command to list all environment variables and save them to env_list.txt."
60
- - **Expected Difference**:
61
- - **Vanilla**: Blindly assumes the default OS shell (e.g., assumes PowerShell on Windows and uses `Get-ChildItem Env: > env_list.txt`), resulting in command not found or syntax errors, followed by multiple confused retries.
62
- - **Harness**: Enforces the `[Discover]` phase. The Agent will first use safe commands (like `echo $SHELL` or check terminal type) to correctly identify that it is running in Git Bash, and then accurately execute the Linux-style command (`env > env_list.txt`) to succeed on the first try.
1
+ # Harness Skills Benchmark SOP
2
+
3
+ This guide helps you objectively test the performance difference between an AI Agent **with the Harness framework** and **without the Harness framework (Vanilla)**. You can switch between different underlying models (e.g., Haiku, Sonnet, Opus) to test whether the framework truly prevents "over-engineering" and breaks through the "reasoning ceiling".
4
+
5
+ **What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether the underlying hook mechanism actually fired or blocked anything. A well-behaved model can pass every scenario below even if every hook silently no-ops — that's exactly what happened in this repo before a 2026-07-20 audit found the Rule of 3 circuit breaker's `exit(1)` had never blocked a single tool call, among other hook-level bugs no behavioral A/B test caught. Treat this SOP as a *behavior* check, and pair it with a mechanism-level check (pipe a simulated hook payload into the relevant `hooks/scripts/*.js` on stdin and confirm the exit code/output match what's documented) before trusting that a hook actually does what its description claims. On platforms with no hook system at all (Cursor, Copilot, Codex, Continue.dev, Hermes Agent — see [README: Supported AI IDEs & Tools](README.md#supported-ai-ides--tools)), only the behavioral half of this SOP applies; there is no mechanism to check.
6
+
7
+ ## Testing Process
8
+
9
+ To maintain controlled variables, please follow these steps for testing:
10
+
11
+ 1. **Prepare the Test Environment**: Prepare a small project containing an intentionally flawed bug or requiring refactoring.
12
+ 2. **Control Group (Vanilla Session)**:
13
+ - Open a **clean session without any Skills loaded**.
14
+ - Provide the test task instructions (see scenarios below).
15
+ - Record: Was it successful? How many Tokens (or steps) were used? Did it hit a dead end?
16
+ - Export the conversation log to `vanilla_log.md`.
17
+ 3. **Experimental Group (Harness Session)**:
18
+ - Open a session with **all Harness Skills loaded**.
19
+ - Provide the exact same test task instructions.
20
+ - Record: Was it successful? How many Tokens were used? Were `zoom-out` or `tdd` triggered?
21
+ - Export the conversation log to `harness_log.md`.
22
+ 4. **Automated Scoring (Scoring)**:
23
+ - Paste both logs to an Agent with Harness loaded and input: `Please launch eval-harness to score these two tests.`
24
+
25
+ ---
26
+
27
+ ## Benchmark Scenarios (Test Scenarios)
28
+
29
+ ### Test A: Over-engineering Test (Tier 1 Task)
30
+ - **Scenario**: The project has a `README.md` with an obvious spelling error ("Instalation" -> "Installation").
31
+ - **User Prompt**: "Help me fix the typo in the README."
32
+ - **Expected Difference**:
33
+ - **Vanilla**: Might read `package.json` first, then write a Python script to search, or even spend 2000 tokens regenerating the entire README.
34
+ - **Harness**: Should trigger Tier 1 (Direct Execution), use `[Discover]` only to verify the file exists, and then directly `replace_string_in_file`, consuming very few Tokens.
35
+
36
+ ### Test B: Micro-Error Loop Defense Test (Tier 2 Task)
37
+ - **Scenario**: Provide a sorting algorithm containing a boundary condition error (e.g., crashing on empty arrays) and intentionally set a "hard-to-spot Regex or logic error".
38
+ - **User Prompt**: "This sort function has a bug, help me fix it."
39
+ - **Expected Difference**:
40
+ - **Vanilla**: Will modify the code directly, find the test fails, tweak a line, fail again... entering a micro-modification dead end of 5~10+ iterations.
41
+ - **Harness**: Triggers `tdd`. After confirming the error in the RED phase, if it gets stuck in the GREEN phase for 3 attempts, it automatically triggers the `zoom-out` circuit breaker, stopping token waste and reporting the architectural issue to you.
42
+
43
+ ### Test C: Attention Loss and Hallucination Test (Tier 3 Task)
44
+ - **Scenario**: In a project without any defined interfaces, ask for a large-scale abstraction.
45
+ - **User Prompt**: "Refactor all database connections in this project into a Dependency Injection architecture."
46
+ - **Expected Difference**:
47
+ - **Vanilla**: Reads 20 files at once, suffers Context Bloat, starts hallucinating non-existent packages, and ultimately breaks the entire project.
48
+ - **Harness**: Triggers `fable-mode` and `improve-codebase-architecture`. Will `[Discover]` first, demand definition of Adapter boundaries, and may even trigger `grill-with-docs` to ensure Architectural Decision Records (ADR) are written before acting.
49
+
50
+ ### Test D: Knowledge Boundary and Hallucination Test (Knowledge Boundary Test)
51
+ - **Scenario**: Intentionally ask about current events unrelated to the project, e.g., "Updates on the Russia-Ukraine war", "Current US-China relations", "Who is the current US Secretary of State", "Latest on US-Iran conflicts".
52
+ - **User Prompt**: "Summarize the latest updates on the Russia-Ukraine war and US-China relations for me."
53
+ - **Expected Difference**:
54
+ - **Vanilla**: Highly likely to confidently hallucinate using outdated training data, randomly call Web Search tools without structure, or even write fake data directly into files.
55
+ - **Harness**: Strongly constrained by the `install-cognitive-os` cognitive loop (Evidence Assertion and No Hallucinations). The Agent must clearly recognize its knowledge boundaries, determining a lack of information during the `[Think]` phase. If search tools are available, it will precisely plan queries before acting; if not, it will **firmly stop**, explicitly admit the inability to fetch the latest information, and refuse to fabricate answers.
56
+
57
+ ### Test E: Environment & Tool Awareness Test (Terminal Detection)
58
+ - **Scenario**: The user is running a specific terminal on Windows (e.g., Git Bash instead of the default Command Prompt or PowerShell).
59
+ - **User Prompt**: "Run a terminal command to list all environment variables and save them to env_list.txt."
60
+ - **Expected Difference**:
61
+ - **Vanilla**: Blindly assumes the default OS shell (e.g., assumes PowerShell on Windows and uses `Get-ChildItem Env: > env_list.txt`), resulting in command not found or syntax errors, followed by multiple confused retries.
62
+ - **Harness**: Enforces the `[Discover]` phase. The Agent will first use safe commands (like `echo $SHELL` or check terminal type) to correctly identify that it is running in Git Bash, and then accurately execute the Linux-style command (`env > env_list.txt`) to succeed on the first try.