harness-everything 0.2.1 → 0.3.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (494) hide show
  1. package/.claude-plugin/marketplace.json +27 -0
  2. package/.claude-plugin/plugin.json +51 -0
  3. package/.gitattributes +1 -0
  4. package/.github/workflows/behavioral-evals.yml +36 -0
  5. package/.github/workflows/ci.yml +117 -0
  6. package/.github/workflows/release.yml +45 -0
  7. package/.husky/commit-msg +1 -0
  8. package/.waza.yaml +22 -0
  9. package/AGENTS.md +60 -2
  10. package/BENCHMARK_SOP.md +62 -62
  11. package/CHANGELOG.md +297 -114
  12. package/CODE_OF_CONDUCT.md +24 -0
  13. package/CONTRIBUTING.md +47 -0
  14. package/Dockerfile +18 -18
  15. package/LICENSE +201 -201
  16. package/README.md +279 -197
  17. package/SECURITY.md +33 -0
  18. package/VERIFICATION.md +394 -347
  19. package/ab-test-deep-analysis-2026-08-27.md +383 -0
  20. package/ab-test-results-2026-08-27.md +152 -0
  21. package/ab-test-statistical-analysis-31cases.md +193 -0
  22. package/behavioral-evals/README.md +85 -0
  23. package/behavioral-evals/analyze-plugin-results.js +110 -0
  24. package/behavioral-evals/cases/baseline-add-feature.yaml +34 -0
  25. package/behavioral-evals/cases/baseline-api-design.yaml +32 -0
  26. package/behavioral-evals/cases/baseline-debugging.yaml +31 -0
  27. package/behavioral-evals/cases/baseline-dependency.yaml +24 -0
  28. package/behavioral-evals/cases/baseline-documentation.yaml +34 -0
  29. package/behavioral-evals/cases/baseline-error-handling.yaml +31 -0
  30. package/behavioral-evals/cases/baseline-git-commit.yaml +33 -0
  31. package/behavioral-evals/cases/baseline-performance-monitoring.yaml +28 -0
  32. package/behavioral-evals/cases/baseline-performance.yaml +34 -0
  33. package/behavioral-evals/cases/baseline-refactor-with-tests.yaml +35 -0
  34. package/behavioral-evals/cases/baseline-refactor.yaml +27 -0
  35. package/behavioral-evals/cases/baseline-security-review.yaml +28 -0
  36. package/behavioral-evals/cases/baseline-simple-bugfix.yaml +33 -0
  37. package/behavioral-evals/cases/baseline-tdd-workflow.yaml +28 -0
  38. package/behavioral-evals/cases/breaker-zoom-out-after-3.yaml +50 -0
  39. package/behavioral-evals/cases/pressure-budget-skip-review.yaml +27 -0
  40. package/behavioral-evals/cases/pressure-bypass-review.yaml +27 -0
  41. package/behavioral-evals/cases/pressure-complexity-skip.yaml +37 -0
  42. package/behavioral-evals/cases/pressure-expert-skip-testing.yaml +31 -0
  43. package/behavioral-evals/cases/pressure-fatigue-skip.yaml +33 -0
  44. package/behavioral-evals/cases/pressure-management-skip-all.yaml +28 -0
  45. package/behavioral-evals/cases/pressure-scope-bypass.yaml +28 -0
  46. package/behavioral-evals/cases/pressure-skip-docs.yaml +50 -0
  47. package/behavioral-evals/cases/pressure-skip-error-handling.yaml +35 -0
  48. package/behavioral-evals/cases/pressure-skip-security.yaml +32 -0
  49. package/behavioral-evals/cases/pressure-skip-tests.yaml +27 -0
  50. package/behavioral-evals/cases/pressure-skip-verification.yaml +43 -0
  51. package/behavioral-evals/cases/pressure-social-skip-review.yaml +28 -0
  52. package/behavioral-evals/cases/pressure-sunk-cost-retry.yaml +47 -0
  53. package/behavioral-evals/cases/scope-discipline.yaml +32 -0
  54. package/behavioral-evals/cases/tier1-no-overplan.yaml +26 -0
  55. package/behavioral-evals/cases/verify-before-done.yaml +31 -0
  56. package/behavioral-evals/results/.gitkeep +0 -0
  57. package/behavioral-evals/results/2026-08-23-breaker-zoom-out-after-3.json +20 -0
  58. package/behavioral-evals/results/2026-08-23-pressure-skip-verification.json +20 -0
  59. package/behavioral-evals/results/2026-08-23-pressure-sunk-cost-retry.json +20 -0
  60. package/behavioral-evals/results/2026-08-23-scope-discipline.json +16 -0
  61. package/behavioral-evals/results/2026-08-23-tier1-no-overplan.json +20 -0
  62. package/behavioral-evals/results/2026-08-23-verify-before-done.json +20 -0
  63. package/behavioral-evals/results/2026-08-27-baseline-add-feature.json +20 -0
  64. package/behavioral-evals/results/2026-08-27-baseline-api-design.json +20 -0
  65. package/behavioral-evals/results/2026-08-27-baseline-debugging.json +20 -0
  66. package/behavioral-evals/results/2026-08-27-baseline-dependency.json +16 -0
  67. package/behavioral-evals/results/2026-08-27-baseline-documentation.json +20 -0
  68. package/behavioral-evals/results/2026-08-27-baseline-error-handling.json +20 -0
  69. package/behavioral-evals/results/2026-08-27-baseline-git-commit.json +20 -0
  70. package/behavioral-evals/results/2026-08-27-baseline-performance-monitoring.json +20 -0
  71. package/behavioral-evals/results/2026-08-27-baseline-performance.json +20 -0
  72. package/behavioral-evals/results/2026-08-27-baseline-refactor-with-tests.json +20 -0
  73. package/behavioral-evals/results/2026-08-27-baseline-refactor.json +20 -0
  74. package/behavioral-evals/results/2026-08-27-baseline-security-review.json +20 -0
  75. package/behavioral-evals/results/2026-08-27-baseline-simple-bugfix.json +20 -0
  76. package/behavioral-evals/results/2026-08-27-baseline-tdd-workflow.json +24 -0
  77. package/behavioral-evals/results/2026-08-27-breaker-zoom-out-after-3.json +20 -0
  78. package/behavioral-evals/results/2026-08-27-pressure-budget-skip-review.json +16 -0
  79. package/behavioral-evals/results/2026-08-27-pressure-bypass-review.json +20 -0
  80. package/behavioral-evals/results/2026-08-27-pressure-complexity-skip.json +20 -0
  81. package/behavioral-evals/results/2026-08-27-pressure-expert-skip-testing.json +16 -0
  82. package/behavioral-evals/results/2026-08-27-pressure-fatigue-skip.json +16 -0
  83. package/behavioral-evals/results/2026-08-27-pressure-management-skip-all.json +20 -0
  84. package/behavioral-evals/results/2026-08-27-pressure-skip-docs.json +20 -0
  85. package/behavioral-evals/results/2026-08-27-pressure-skip-error-handling.json +20 -0
  86. package/behavioral-evals/results/2026-08-27-pressure-skip-security.json +20 -0
  87. package/behavioral-evals/results/2026-08-27-pressure-skip-tests.json +20 -0
  88. package/behavioral-evals/results/2026-08-27-pressure-skip-verification.json +24 -0
  89. package/behavioral-evals/results/2026-08-27-pressure-social-skip-review.json +20 -0
  90. package/behavioral-evals/results/2026-08-27-pressure-sunk-cost-retry.json +24 -0
  91. package/behavioral-evals/results/2026-08-27-scope-discipline.json +16 -0
  92. package/behavioral-evals/results/2026-08-27-tier1-no-overplan.json +20 -0
  93. package/behavioral-evals/results/2026-08-27-verify-before-done.json +20 -0
  94. package/behavioral-evals/results/run-20260827-115509.log +287 -0
  95. package/behavioral-evals/run-with-plugin.js +316 -0
  96. package/behavioral-evals/run.js +587 -0
  97. package/benchmarks/README.md +63 -0
  98. package/benchmarks/ab-test-2026-08-26/results.json +55 -0
  99. package/benchmarks/results/logs/.gitkeep +0 -0
  100. package/benchmarks/results/logs/test-a-harness.md +31 -0
  101. package/benchmarks/results/logs/test-a-vanilla.md +28 -0
  102. package/benchmarks/results/logs/test-b-micro-loop-harness.md +48 -0
  103. package/benchmarks/results/logs/test-b-micro-loop-vanilla.md +36 -0
  104. package/benchmarks/results/logs/test-c-attention-loss-harness.md +40 -0
  105. package/benchmarks/results/logs/test-c-attention-loss-vanilla.md +63 -0
  106. package/benchmarks/results/logs/test-d-knowledge-boundary-harness.md +24 -0
  107. package/benchmarks/results/logs/test-d-knowledge-boundary-vanilla.md +29 -0
  108. package/benchmarks/results/logs/test-e-shell-awareness-harness.md +20 -0
  109. package/benchmarks/results/logs/test-e-shell-awareness-vanilla.md +20 -0
  110. package/benchmarks/results/logs/test-f-fact-audit-harness.md +16 -0
  111. package/benchmarks/results/logs/test-f-fact-audit-vanilla.md +16 -0
  112. package/benchmarks/results/results.json +158 -0
  113. package/benchmarks/run.js +220 -0
  114. package/benchmarks/schema.json +44 -0
  115. package/bin/cli.js +152 -97
  116. package/ci/ab-test-cases/grill-me-adversarial.yaml +18 -0
  117. package/ci/ab-test-cases/tdd-test-first.yaml +23 -0
  118. package/ci/ab-test-cases/verification-loop-gates.yaml +25 -0
  119. package/ci/ab-test-cases/verify-before-claim-cites.yaml +18 -0
  120. package/ci/ab-test-harness.js +410 -0
  121. package/{eval-framework → ci}/cases/case1.json +5 -5
  122. package/ci/cases/case10-small-feature.json +5 -0
  123. package/ci/cases/case11-readme-typo-update.json +5 -0
  124. package/ci/cases/case12-chinese-macro-audit.json +5 -0
  125. package/{eval-framework → ci}/cases/case2.json +5 -5
  126. package/{eval-framework → ci}/cases/case3.json +5 -5
  127. package/{eval-framework → ci}/cases/case4-zh-tier2.json +5 -5
  128. package/{eval-framework → ci}/cases/case5-zh-tier3.json +5 -5
  129. package/ci/cases/case6-macro-skill-audit.json +5 -0
  130. package/ci/cases/case7-macro-reference-audit.json +5 -0
  131. package/ci/cases/case8-controlled-ab-benchmark.json +5 -0
  132. package/ci/cases/case9-four-issue-audit.json +5 -0
  133. package/ci/consistency-check.js +347 -0
  134. package/ci/description-collision.js +109 -0
  135. package/ci/fable-mode-test.js +120 -0
  136. package/ci/fixtures/agency-agents/divisions.json +6 -0
  137. package/ci/fixtures/agency-agents/engineering/code-reviewer.md +11 -0
  138. package/ci/fixtures/agency-agents/testing/reality-checker.md +11 -0
  139. package/ci/fixtures/bad-frontmatter-colon/SKILL.md +16 -0
  140. package/ci/fixtures/duplicate-description/skill-a/SKILL.md +16 -0
  141. package/ci/fixtures/duplicate-description/skill-b/SKILL.md +16 -0
  142. package/ci/mechanism-2a-rule-of-3.test.js +46 -0
  143. package/ci/mechanism-2b-boundary-guard.test.js +16 -0
  144. package/ci/mechanism-2c-state-persist.test.js +29 -0
  145. package/ci/mechanism-2d-fact-audit.test.js +18 -0
  146. package/ci/mechanism-2e-scope-guard.test.js +24 -0
  147. package/ci/mechanism-2f-stop-gate.test.js +23 -0
  148. package/ci/mechanism-2g-platform-ignore.test.js +88 -0
  149. package/ci/mechanism-2h-installer-manifest.test.js +147 -0
  150. package/ci/mechanism-2i-crlf-frontmatter.test.js +145 -0
  151. package/ci/mechanism-2i-install-verification.test.js +34 -0
  152. package/ci/mechanism-2j-behavioral-runner.test.js +20 -0
  153. package/ci/mechanism-2k-reference-check.test.js +20 -0
  154. package/ci/mechanism-2l-multi-agent-workspace.test.js +105 -0
  155. package/ci/mechanism-test.js +71 -0
  156. package/ci/negative-controls.js +84 -0
  157. package/ci/reference-check.js +112 -0
  158. package/ci/release-consistency-check.js +123 -0
  159. package/ci/routing-check.js +90 -0
  160. package/{eval-framework → ci}/runner.js +56 -56
  161. package/ci/skill-routing-check.js +98 -0
  162. package/ci/test-helper.js +104 -0
  163. package/ci/test-negative-controls.js +148 -0
  164. package/commitlint.config.js +25 -0
  165. package/docker-verify.sh +266 -266
  166. package/docs/architecture.md +99 -95
  167. package/docs/audit.md +53 -0
  168. package/docs/mechanism-first-skill-mesh.md +163 -0
  169. package/docs/philosophy.md +47 -45
  170. package/docs/reflection.md +48 -34
  171. package/docs/routing.md +58 -45
  172. package/docs/troubleshooting.md +284 -0
  173. package/docs/workflows/environment-detection.md +63 -63
  174. package/docs/workflows/eval-harness.md +64 -57
  175. package/docs/workflows/fable-discipline.md +59 -59
  176. package/docs/workflows/fable-mode.md +12 -15
  177. package/docs/workflows/find-skills.md +36 -0
  178. package/docs/workflows/git-commit.md +71 -62
  179. package/docs/workflows/grill-me.md +62 -62
  180. package/docs/workflows/grill-with-docs.md +69 -62
  181. package/docs/workflows/harness-everything.md +95 -92
  182. package/docs/workflows/improve-codebase-architecture.md +68 -60
  183. package/docs/workflows/install-cognitive-os.md +60 -60
  184. package/docs/workflows/multi-agent-workspace.md +42 -0
  185. package/docs/workflows/repo-docs.md +64 -59
  186. package/docs/workflows/rewrite-commits.md +60 -60
  187. package/docs/workflows/security-review.md +77 -58
  188. package/docs/workflows/self-evolve.md +42 -59
  189. package/docs/workflows/skill-creator.md +80 -75
  190. package/docs/workflows/skill-style.md +64 -64
  191. package/docs/workflows/tdd.md +7 -4
  192. package/docs/workflows/to-spec.md +83 -0
  193. package/docs/workflows/to-tickets.md +80 -0
  194. package/docs/workflows/todo-driven-workflow.md +28 -30
  195. package/docs/workflows/using-git-worktrees.md +67 -63
  196. package/docs/workflows/verification-loop.md +71 -69
  197. package/docs/workflows/verify-before-claim.md +64 -60
  198. package/docs/workflows/zoom-out.md +74 -62
  199. package/environment-detection/SKILL.md +41 -77
  200. package/environment-detection/references/shell-syntax-rules.md +76 -0
  201. package/environment-detection/scripts/preflight.js +103 -93
  202. package/eval-framework/fixtures/bad-frontmatter-colon/SKILL.md +14 -0
  203. package/eval-framework/fixtures/duplicate-desc-skill-a/SKILL.md +14 -0
  204. package/eval-framework/fixtures/duplicate-desc-skill-b/SKILL.md +14 -0
  205. package/eval-framework/fixtures/missing-delimiter/SKILL.md +13 -0
  206. package/eval-framework/fixtures/over-budget-skill/SKILL.md +36 -0
  207. package/eval-harness/SKILL.md +42 -59
  208. package/eval-harness/references/scoring-rubric.md +68 -0
  209. package/eval-harness/scripts/evaluate.js +77 -61
  210. package/evals/environment-detection/eval.yaml +26 -0
  211. package/evals/environment-detection/tasks/negative-trigger.yaml +16 -0
  212. package/evals/environment-detection/tasks/positive-trigger.yaml +15 -0
  213. package/evals/eval-harness/eval.yaml +26 -0
  214. package/evals/eval-harness/tasks/negative-trigger.yaml +16 -0
  215. package/evals/eval-harness/tasks/positive-trigger.yaml +15 -0
  216. package/evals/fable-discipline/eval.yaml +26 -0
  217. package/evals/fable-discipline/tasks/negative-trigger.yaml +16 -0
  218. package/evals/fable-discipline/tasks/positive-trigger.yaml +15 -0
  219. package/evals/fable-mode/eval.yaml +26 -0
  220. package/evals/fable-mode/tasks/negative-tier2-no-model.yaml +18 -0
  221. package/evals/fable-mode/tasks/negative-trigger.yaml +16 -0
  222. package/evals/fable-mode/tasks/positive-haiku-mode.yaml +16 -0
  223. package/evals/fable-mode/tasks/positive-opus-mode.yaml +16 -0
  224. package/evals/fable-mode/tasks/positive-sonnet-mode.yaml +16 -0
  225. package/evals/fable-mode/tasks/positive-trigger.yaml +15 -0
  226. package/evals/find-skills/eval.yaml +26 -0
  227. package/evals/find-skills/tasks/negative-trigger.yaml +16 -0
  228. package/evals/find-skills/tasks/positive-trigger.yaml +15 -0
  229. package/evals/git-commit/eval.yaml +26 -0
  230. package/evals/git-commit/tasks/negative-trigger.yaml +16 -0
  231. package/evals/git-commit/tasks/positive-trigger.yaml +15 -0
  232. package/evals/grill-me/eval.yaml +26 -0
  233. package/evals/grill-me/tasks/negative-trigger.yaml +16 -0
  234. package/evals/grill-me/tasks/positive-trigger.yaml +15 -0
  235. package/evals/grill-with-docs/eval.yaml +26 -0
  236. package/evals/grill-with-docs/tasks/negative-trigger.yaml +16 -0
  237. package/evals/grill-with-docs/tasks/positive-trigger.yaml +15 -0
  238. package/evals/harness-everything/eval.yaml +26 -0
  239. package/evals/harness-everything/tasks/negative-non-software.yaml +16 -0
  240. package/evals/harness-everything/tasks/negative-skill-indicated.yaml +15 -0
  241. package/evals/harness-everything/tasks/positive-tier1.yaml +16 -0
  242. package/evals/harness-everything/tasks/positive-tier2.yaml +16 -0
  243. package/evals/harness-everything/tasks/positive-tier3.yaml +16 -0
  244. package/evals/improve-codebase-architecture/eval.yaml +26 -0
  245. package/evals/improve-codebase-architecture/tasks/negative-trigger.yaml +16 -0
  246. package/evals/improve-codebase-architecture/tasks/positive-trigger.yaml +15 -0
  247. package/evals/install-cognitive-os/eval.yaml +26 -0
  248. package/evals/install-cognitive-os/tasks/negative-trigger.yaml +16 -0
  249. package/evals/install-cognitive-os/tasks/positive-trigger.yaml +15 -0
  250. package/evals/multi-agent-workspace/eval.yaml +26 -0
  251. package/evals/multi-agent-workspace/tasks/negative-trigger.yaml +15 -0
  252. package/evals/multi-agent-workspace/tasks/positive-trigger.yaml +15 -0
  253. package/evals/repo-docs/eval.yaml +26 -0
  254. package/evals/repo-docs/tasks/negative-trigger.yaml +16 -0
  255. package/evals/repo-docs/tasks/positive-trigger.yaml +15 -0
  256. package/evals/rewrite-commits/eval.yaml +26 -0
  257. package/evals/rewrite-commits/tasks/negative-trigger.yaml +16 -0
  258. package/evals/rewrite-commits/tasks/positive-trigger.yaml +15 -0
  259. package/evals/security-review/eval.yaml +26 -0
  260. package/evals/security-review/tasks/negative-trigger.yaml +16 -0
  261. package/evals/security-review/tasks/positive-trigger.yaml +15 -0
  262. package/evals/self-evolve/eval.yaml +26 -0
  263. package/evals/self-evolve/tasks/negative-trigger.yaml +16 -0
  264. package/evals/self-evolve/tasks/positive-trigger.yaml +15 -0
  265. package/evals/skill-creator/eval.yaml +26 -0
  266. package/evals/skill-creator/tasks/negative-trigger.yaml +16 -0
  267. package/evals/skill-creator/tasks/positive-trigger.yaml +15 -0
  268. package/evals/skill-style/eval.yaml +26 -0
  269. package/evals/skill-style/tasks/negative-trigger.yaml +16 -0
  270. package/evals/skill-style/tasks/positive-trigger.yaml +15 -0
  271. package/evals/tdd/eval.yaml +26 -0
  272. package/evals/tdd/tasks/negative-trivia.yaml +16 -0
  273. package/evals/tdd/tasks/positive-bugfix.yaml +15 -0
  274. package/evals/tdd/tasks/positive-feature.yaml +16 -0
  275. package/evals/tdd/tasks/positive-refactor.yaml +15 -0
  276. package/evals/to-spec/eval.yaml +26 -0
  277. package/evals/to-spec/tasks/negative-trigger.yaml +16 -0
  278. package/evals/to-spec/tasks/positive-trigger.yaml +15 -0
  279. package/evals/to-tickets/eval.yaml +26 -0
  280. package/evals/to-tickets/tasks/negative-trigger.yaml +15 -0
  281. package/evals/to-tickets/tasks/positive-trigger.yaml +15 -0
  282. package/evals/todo-driven-workflow/eval.yaml +26 -0
  283. package/evals/todo-driven-workflow/tasks/negative-single-edit.yaml +16 -0
  284. package/evals/todo-driven-workflow/tasks/positive-multi-step.yaml +15 -0
  285. package/evals/todo-driven-workflow/tasks/positive-refactor-flow.yaml +15 -0
  286. package/evals/using-git-worktrees/eval.yaml +26 -0
  287. package/evals/using-git-worktrees/tasks/negative-trigger.yaml +15 -0
  288. package/evals/using-git-worktrees/tasks/positive-trigger.yaml +15 -0
  289. package/evals/verification-loop/eval.yaml +26 -0
  290. package/evals/verification-loop/tasks/negative-discussion.yaml +16 -0
  291. package/evals/verification-loop/tasks/positive-pre-ship.yaml +15 -0
  292. package/evals/verify-before-claim/eval.yaml +26 -0
  293. package/evals/verify-before-claim/tasks/negative-trigger.yaml +16 -0
  294. package/evals/verify-before-claim/tasks/positive-trigger.yaml +15 -0
  295. package/evals/zoom-out/eval.yaml +26 -0
  296. package/evals/zoom-out/tasks/negative-trivial-edit.yaml +16 -0
  297. package/evals/zoom-out/tasks/positive-repeated-failure.yaml +15 -0
  298. package/fable-discipline/SKILL.md +41 -38
  299. package/fable-discipline/references/discipline-rules.md +22 -0
  300. package/fable-mode/CONTRACT-FORMAT.md +51 -51
  301. package/fable-mode/SKILL.md +39 -51
  302. package/fable-mode/agents/fable-orchestrator.md +99 -92
  303. package/fable-mode/agents/fable-verifier.md +32 -32
  304. package/fable-mode/agents/fable-worker-haiku.md +52 -48
  305. package/fable-mode/agents/fable-worker-sonnet.md +53 -49
  306. package/fable-mode/execution-guardrails/SKILL.md +43 -87
  307. package/fable-mode/execution-guardrails/references/guardrail-rules.md +56 -0
  308. package/fable-mode/fable-haiku/SKILL.md +36 -48
  309. package/fable-mode/fable-haiku/references/routing-notes.md +26 -0
  310. package/fable-mode/fable-opus/SKILL.md +36 -0
  311. package/fable-mode/fable-sonnet/SKILL.md +36 -0
  312. package/fable-mode/model-matrix.json +32 -0
  313. package/fable-mode/references/execution-phases.md +37 -0
  314. package/fable-mode/references/model-matrix.md +39 -0
  315. package/fable-mode/scripts/model-selector.js +129 -0
  316. package/find-skills/SKILL.md +48 -0
  317. package/find-skills/references/discovery-flow.md +102 -0
  318. package/find-skills/scripts/use-skill.js +106 -0
  319. package/git-commit/SKILL.md +45 -55
  320. package/git-commit/guides/ANGULAR_STYLE.md +60 -60
  321. package/git-commit/guides/COMMIT_GENERATION.md +23 -23
  322. package/git-commit/guides/LANGUAGE_DETECTION.md +16 -16
  323. package/git-commit/guides/MAIN_REPO.md +14 -14
  324. package/git-commit/guides/SUBMODULES.md +27 -27
  325. package/git-commit/references/commit-flow.md +30 -0
  326. package/grill-me/SKILL.md +43 -36
  327. package/grill-me/references/grilling-playbook.md +38 -0
  328. package/grill-with-docs/ADR-FORMAT.md +47 -47
  329. package/grill-with-docs/CONTEXT-FORMAT.md +63 -63
  330. package/grill-with-docs/SKILL.md +41 -99
  331. package/grill-with-docs/references/session-playbook.md +97 -0
  332. package/harness-everything/SKILL.md +42 -141
  333. package/harness-everything/references/skill-registry.md +39 -0
  334. package/harness-everything/references/triage-and-tiers.md +116 -0
  335. package/harness-everything/scripts/bootstrap.js +203 -140
  336. package/harness-everything/scripts/routing-keywords.json +320 -0
  337. package/harness-everything/scripts/self-heal.js +241 -189
  338. package/harness-everything/scripts/tier-router.js +334 -219
  339. package/harness-everything/scripts/verify-gate.js +87 -18
  340. package/hooks/hooks.json +155 -155
  341. package/hooks/scripts/boundary-guard.js +55 -55
  342. package/hooks/scripts/contract-test.js +79 -79
  343. package/hooks/scripts/depth-guard.js +65 -65
  344. package/hooks/scripts/lib/harness-state.js +139 -228
  345. package/hooks/scripts/lib/platforms/claude.js +185 -42
  346. package/hooks/scripts/lib/platforms/codex.js +114 -46
  347. package/hooks/scripts/lib/platforms/continue.js +91 -41
  348. package/hooks/scripts/lib/platforms/copilot.js +109 -41
  349. package/hooks/scripts/lib/platforms/cursor.js +89 -41
  350. package/hooks/scripts/lib/platforms/hermes.js +57 -32
  351. package/hooks/scripts/lib/platforms/index.js +17 -17
  352. package/hooks/scripts/lib/platforms/worktrees.js +20 -20
  353. package/hooks/scripts/reset-circuit-breaker.js +30 -30
  354. package/hooks/scripts/rule-of-3-tracker.js +124 -91
  355. package/hooks/scripts/rule-of-3.js +14 -3
  356. package/hooks/scripts/state-persist.js +99 -99
  357. package/hooks/scripts/stop-gate.js +80 -80
  358. package/hooks/scripts/subagent-scope-guard.js +88 -88
  359. package/improve-codebase-architecture/SKILL.md +40 -44
  360. package/improve-codebase-architecture/guides/DEEPENING.md +37 -37
  361. package/improve-codebase-architecture/guides/HTML-REPORT.md +123 -123
  362. package/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -44
  363. package/improve-codebase-architecture/guides/LANGUAGE.md +53 -53
  364. package/improve-codebase-architecture/guides/refactoring-workflow.md +59 -0
  365. package/install-cognitive-os/SKILL.md +38 -95
  366. package/install-cognitive-os/references/cognitive-loop.md +27 -0
  367. package/multi-agent-workspace/NOTICE-agency-agents.md +27 -0
  368. package/multi-agent-workspace/SKILL.md +37 -0
  369. package/{create-agent-launcher → multi-agent-workspace}/examples/csharp-monorepo.md +123 -123
  370. package/{create-agent-launcher → multi-agent-workspace}/examples/typescript-cli.md +126 -126
  371. package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-claude.md +77 -77
  372. package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-codex.md +146 -146
  373. package/multi-agent-workspace/guidelines/platform-continue.md +38 -0
  374. package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-copilot.md +74 -74
  375. package/multi-agent-workspace/guidelines/platform-cursor.md +40 -0
  376. package/multi-agent-workspace/guidelines/platform-hermes.md +38 -0
  377. package/{create-agent-launcher → multi-agent-workspace}/guidelines/role-capabilities.md +108 -108
  378. package/multi-agent-workspace/references/agency-agents.md +31 -0
  379. package/multi-agent-workspace/references/architecture-guide.md +32 -0
  380. package/multi-agent-workspace/references/orchestration.md +48 -0
  381. package/multi-agent-workspace/scripts/agency-catalog.js +190 -0
  382. package/multi-agent-workspace/scripts/index_memory.js +51 -0
  383. package/multi-agent-workspace/scripts/scaffold.js +253 -0
  384. package/{create-agent-launcher → multi-agent-workspace}/templates/README.md +17 -17
  385. package/{create-agent-launcher → multi-agent-workspace}/templates/agent-launcher.md +67 -67
  386. package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-csharp.md +55 -55
  387. package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-polyglot.md +61 -61
  388. package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-ts.md +56 -56
  389. package/{create-agent-launcher → multi-agent-workspace}/templates/memory-keeper.md +36 -36
  390. package/{create-agent-launcher → multi-agent-workspace}/templates/requirement-analyzer.md +65 -65
  391. package/{create-agent-launcher → multi-agent-workspace}/templates/specialist-agent.md +34 -34
  392. package/{create-agent-launcher → multi-agent-workspace}/templates/task-verifier.md +55 -55
  393. package/{create-agent-launcher → multi-agent-workspace}/templates/tool-maker.md +32 -32
  394. package/multi-agent-workspace/workflows/01-init.md +22 -0
  395. package/multi-agent-workspace/workflows/02-analysis.md +23 -0
  396. package/multi-agent-workspace/workflows/03-generation.md +27 -0
  397. package/multi-agent-workspace/workflows/04-launcher.md +16 -0
  398. package/opencode-plugin/README.md +94 -0
  399. package/opencode-plugin/hooks/circuit-breaker.js +121 -0
  400. package/opencode-plugin/hooks/compliance.js +128 -0
  401. package/opencode-plugin/hooks/post-edit.js +62 -0
  402. package/opencode-plugin/hooks/pre-complete.js +65 -0
  403. package/opencode-plugin/hooks/verify.js +140 -0
  404. package/opencode-plugin/plugin.json +17 -0
  405. package/opencode-plugin/test-hook-firing.js +198 -0
  406. package/package.json +50 -22
  407. package/references/definition-of-done.md +67 -0
  408. package/references/performance-checklist.md +153 -0
  409. package/references/platform-advisory-template.md +58 -0
  410. package/references/security-checklist.md +205 -0
  411. package/repo-docs/SKILL.md +46 -45
  412. package/repo-docs/references/process-guide.md +60 -0
  413. package/repo-docs/templates/agents-template.md +264 -264
  414. package/repo-docs/templates/knowledge-agents-template.md +50 -50
  415. package/repo-docs/templates/knowledge-readme-template.md +83 -83
  416. package/repo-docs/templates/manual-recon.md +117 -117
  417. package/repo-docs/templates/multi-skills-readme-template.md +72 -72
  418. package/repo-docs/templates/product-agents-template.md +78 -78
  419. package/repo-docs/templates/product-readme-template.md +95 -95
  420. package/repo-docs/templates/readme-template.md +136 -136
  421. package/repo-docs/templates/single-skill-readme-template.md +70 -70
  422. package/repo-docs/templates/skills-agents-template.md +60 -60
  423. package/rewrite-commits/SKILL.md +37 -32
  424. package/rewrite-commits/references/workflow.md +18 -0
  425. package/scripts/installer.js +546 -649
  426. package/scripts/lib/advisory-text.js +167 -163
  427. package/scripts/lib/claude-hooks.js +84 -71
  428. package/scripts/lib/gitignore.js +199 -80
  429. package/scripts/lib/manifest.js +146 -82
  430. package/scripts/lib/prompts.js +241 -241
  431. package/scripts/lib/skills.js +260 -172
  432. package/scripts/lib/workspace.js +28 -28
  433. package/scripts/verify-install.js +147 -0
  434. package/security-review/SKILL.md +50 -511
  435. package/security-review/guides/OWASP-PATTERNS.md +31 -0
  436. package/security-review/guides/STRIDE-THREAT-MODEL.md +21 -0
  437. package/security-review/references/security-checklist.md +478 -0
  438. package/security-review/scripts/audit-secrets.js +84 -0
  439. package/self-evolve/SKILL.md +47 -54
  440. package/self-evolve/references/memory-resolution.md +58 -0
  441. package/self-evolve/scripts/persist-memory.js +139 -40
  442. package/self-evolve/scripts/register-dynamic-skill.js +251 -0
  443. package/self-evolve/scripts/self-regression.js +110 -113
  444. package/skill-creator/SKILL.md +49 -91
  445. package/skill-creator/references/authoring-workflow.md +14 -0
  446. package/skill-creator/references/dynamic-generation-contract.md +21 -0
  447. package/skill-creator/references/quality-checklist.md +15 -0
  448. package/skill-creator/references/quality-principles.md +81 -81
  449. package/skill-creator/references/testing-workflow.md +20 -25
  450. package/skill-style/SKILL.md +42 -48
  451. package/skill-style/references/style-guide.md +36 -0
  452. package/tdd/SKILL.md +44 -53
  453. package/tdd/guides/deep-modules.md +33 -33
  454. package/tdd/guides/interface-design.md +31 -31
  455. package/tdd/guides/mocking.md +59 -59
  456. package/tdd/guides/refactoring.md +10 -10
  457. package/tdd/guides/tests.md +61 -61
  458. package/tdd/references/core-discipline.md +49 -0
  459. package/to-spec/SKILL.md +42 -0
  460. package/to-spec/references/process.md +77 -0
  461. package/to-spec/scripts/check-project-docs.js +259 -0
  462. package/to-spec/templates/cli-reference.md +31 -0
  463. package/to-spec/templates/dev-doc.md +27 -0
  464. package/to-spec/templates/feature-spec.md +59 -0
  465. package/to-spec/templates/schema-doc.md +31 -0
  466. package/to-tickets/SKILL.md +42 -0
  467. package/to-tickets/references/publishing-and-templates.md +97 -0
  468. package/todo-driven-workflow/SKILL.md +40 -54
  469. package/todo-driven-workflow/references/execution-guide.md +27 -0
  470. package/using-git-worktrees/SKILL.md +49 -161
  471. package/using-git-worktrees/references/workflow-details.md +108 -0
  472. package/verification-loop/SKILL.md +46 -138
  473. package/verification-loop/references/verification-phases.md +116 -0
  474. package/verification-loop/templates/verification-report.template.md +15 -0
  475. package/verify-before-claim/SKILL.md +43 -86
  476. package/verify-before-claim/references/verification-guide.md +73 -0
  477. package/zoom-out/SKILL.md +42 -90
  478. package/zoom-out/references/circuit-breaker.md +66 -0
  479. package/zoom-out/templates/zoom-out-report.template.md +20 -0
  480. package/build-multi-agent-system/AGENTS.md +0 -50
  481. package/build-multi-agent-system/README.md +0 -63
  482. package/build-multi-agent-system/SKILL.md +0 -68
  483. package/create-agent-launcher/SKILL.md +0 -48
  484. package/create-agent-launcher/guidelines/platform-gemini.md +0 -101
  485. package/create-agent-launcher/workflows/01-init.md +0 -22
  486. package/create-agent-launcher/workflows/02-analysis.md +0 -26
  487. package/create-agent-launcher/workflows/03-generation.md +0 -29
  488. package/create-agent-launcher/workflows/04-launcher.md +0 -20
  489. package/docs/workflows/build-multi-agent-system.md +0 -62
  490. package/docs/workflows/create-agent-launcher.md +0 -62
  491. package/eval-framework/behavioral-test.js +0 -105
  492. package/eval-framework/mechanism-test.js +0 -251
  493. package/harness-everything/scripts/todo-cli.js +0 -157
  494. package/skill-style/STYLE.md +0 -37
@@ -1,77 +1,41 @@
1
- ---
2
- name: environment-detection
3
- description: Use at the very beginning of the session (Discover phase) to automatically detect and align with the operating system, terminal shell (e.g. Git Bash vs PowerShell vs Command Prompt), package managers, and development tools. Prevents blind command execution and repetitive tool-related errors.
4
- author: Miya Daniel | Harness Core Team
5
- version: 0.2.0
6
- ---
7
-
8
- # Environment Detection & Shell Alignment
9
-
10
- ## 📋 Skill Contract
11
-
12
- | Component | Specification |
13
- | :--- | :--- |
14
- | **Trigger / Input** | Session Start (Discover Phase), or before executing a complex terminal command sequence. |
15
- | **Expected Output** | Terminal execution of `node <this-skill-dir>/scripts/preflight.js`. Output of the detected OS and Shell. |
16
- | **State Mutations** | Context is updated with the correct path separator (`/` vs `\`) and environment variable syntax (`$VAR` vs `%VAR%`). |
17
- | **Enforcement Gate** | You MUST read the terminal output of `preflight.js`. If you use the wrong syntax (e.g. `%VAR%` in bash), the terminal will naturally Exit 1 and block your progress. |
18
-
19
- ## Core Principles
20
- - **MUST** operate using the cognitive loop: Think > Try > Summarize > Record.
21
- - **[Think]**: Before running any terminal commands, check your environment. Blind execution is the primary source of agent step waste.
22
- - **[Try]**: Run the automated preflight check script to determine exact platform properties.
23
- - **[Summarize]**: Output the detected terminal, OS, and toolchains so the session context retains these bounds.
24
- - **[Record]**: Adopt the corresponding command syntax rules for the rest of the conversation.
25
-
26
- ## Strict Workspace Boundary (僅偵測與操作當前工作區)
27
- - **Current Workspace Only**: 必須僅偵測與操作當前活動工作區目錄(即當前專案根目錄,亦為 `process.cwd()`)。
28
- - **Ignore Other Workspaces**: 即使 VS Code 內容或歷史記錄中暴露了其他工作區、暫存路徑或最近開啟的專案(例如 `d:\super.h2o.sbom`、`d:\DeveloperDocs`、`c:\Users\DanielCH.Chang\Desktop\XinputDxe_V0.0.5` 等),你也 **必須完全忽略它們**。絕對不要對這些非當前專案之路徑進行結構分析、工具檢查,更不能在其中執行任何終端機指令。
29
- - **Single-Workspace Execution**: 所有終端機指令、工具可用性檢查、路徑解析均必須嚴格限制在當前專案根目錄內。嚴格禁止跨越邊界操作鄰近或無關的資料夾。
30
-
31
- ## [State Checkpoint]
32
- - Verify your environment details *before* executing commands or writing files.
33
- - If you just started a session, run the preflight script immediately.
34
-
35
- ## [Discovery Phase: Environment Audit]
36
- 1. Run the preflight script that lives at `scripts/preflight.js` **inside this skill's own directory** — resolve the path from wherever this SKILL.md was loaded (do not guess a hard-coded location):
37
- ```bash
38
- node "<this-skill-dir>/scripts/preflight.js"
39
- ```
40
- 2. Parse the output and understand:
41
- - Operating System (Windows vs. macOS vs. Linux)
42
- - Active Shell (Git Bash vs. PowerShell vs. Command Prompt)
43
- - Available CLI Tools (node, pnpm, docker, python, etc.)
44
- 3. **Toolchain Self-Heal (工欲善其事,必先利其器)**: The harness itself is part of the environment. Audit whether this workspace's integration touchpoints (`.claude/settings.json` hooks, `.cursorrules`, `.github/copilot-instructions.md`, `AGENTS.md`) are installed, and repair any missing ones — e.g., installed via Claude Code but now opened in Copilot:
45
- ```bash
46
- node "<skills-repo-root>/harness-everything/scripts/self-heal.js"
47
- ```
48
- The script is idempotent (it delegates to the installer, which merges hooks by id and never duplicates advisory text), so running it on an already-healthy workspace is a no-op. Skip only if the user intentionally removed a touchpoint file.
49
-
50
- ## [Execution Phase: Syntax and Command Alignment]
51
- Choose the correct syntax matching your active shell:
52
-
53
- ### 1. Git Bash on Windows (User's Preferred Terminal)
54
- - **Path Slashes**: Always use forward slashes `/` (e.g., `scripts/lib/utils.js`). Never use raw backslashes `\` because they act as escape characters in Bash.
55
- - **Env Variables**: Use `$VARIABLE_NAME` syntax. Do NOT use `%VARIABLE_NAME%` or `$env:VARIABLE_NAME`.
56
- - **Command Set**: Standard Unix commands are fully supported (`ls`, `rm -rf`, `mkdir -p`, `cp`, `mv`). Do NOT run CMD native commands like `dir`, `del`, or `copy`.
57
- - **No PowerShell syntax**: Do not run PowerShell-specific scripts or syntax.
58
-
59
- ### 2. PowerShell on Windows
60
- - **Path Slashes**: Forward slashes `/` or backslashes `\\` are both acceptable.
61
- - **Env Variables**: Use `$env:VARIABLE_NAME` syntax.
62
- - **Command Set**: Use standard PowerShell commands or common aliases (e.g., `New-Item`, `Remove-Item`).
63
-
64
- ### 3. Windows Command Prompt (CMD)
65
- - **Path Slashes**: Use backslashes `\` for file paths.
66
- - **Env Variables**: Use `%VARIABLE_NAME%` syntax.
67
- - **Command Set**: Use `dir`, `del`, `copy`, `mkdir`. Do NOT use `ls`, `rm`, `cp`, `mkdir -p`.
68
-
69
- ## [Verification & Reflection Phase]
70
- 3. If any shell command fails, do **not** blindly retry the exact same command.
71
- 4. Stop and ask yourself: "Did I use the wrong syntax for this terminal? Is this a Git Bash vs. PowerShell syntax issue?"
72
- 5. If you fail 3 times on the same command, trigger the `zoom-out` circuit breaker.
73
-
74
- ## Red Flags & Common Errors
75
- - ❌ **CMD commands in Git Bash**: Attempting to run `dir` or `del` when the terminal is actually running Git Bash.
76
- - ❌ **PowerShell commands in Bash**: Attempting to run `$env:PATH` or `.ps1` files in a Bash environment.
77
- - ❌ **Path escape failures**: Sending raw Windows paths like `scripts\run.js` inside a Bash terminal, which interprets `\r` as a carriage return or escapes characters.
1
+ ---
2
+ name: environment-detection
3
+ description: Use at session start (Discover phase) to detect and align with the operating system, shell (Git Bash vs PowerShell vs CMD), package managers, and dev tools; prevents blind command execution and repetitive tool errors.
4
+ license: Apache-2.0
5
+ metadata:
6
+ author: Miya Daniel
7
+ version: 0.3.4
8
+ ---
9
+
10
+ # Environment Detection & Shell Alignment
11
+
12
+ Think > Try > Summarize > Record: detect before executing.
13
+
14
+ ## 📋 Skill Contract
15
+
16
+ | Component | Specification |
17
+ | :--- | :--- |
18
+ | **Trigger / Input** | Session start or complex terminal sequences. |
19
+ | **Expected Output** | Detected OS, shell, PATH via script or heuristics. |
20
+ | **State Mutations** | Session context adopts path/env-var syntax. |
21
+ | **Enforcement Gate** | Preflight; fall back to `<environment_info>` inspection and probing. |
22
+
23
+ Boundary: operate solely within `process.cwd()`; ignore other workspaces, history, or temp paths surfaced by the IDE — never run commands outside the current project root.
24
+
25
+ ## Workflow
26
+
27
+ 1. Preflight from this skill's own directory; parse OS, shell, available CLIs: `node "<this-skill-dir>/scripts/preflight.js"`
28
+ 2. Self-heal harness touchpoints (idempotent): `node "<skills-repo-root>/harness-everything/scripts/self-heal.js"` — audits `.claude/settings.json`, `.cursorrules`, `.github/copilot-instructions.md`, `AGENTS.md`.
29
+ 3. Adopt syntax: Git Bash → forward `/`, `$VAR`, Unix cmds (`ls`, `rm -rf`), never `dir`/`del`/PowerShell · PowerShell → `$env:VAR`, cmdlets · CMD → `\`, `%VAR%`, `dir`/`del`/`copy`, never Unix.
30
+ 4. On failure don't blindly retry — suspect wrong-shell syntax. Three failures → trigger `zoom-out`.
31
+
32
+ Deep dive: references/shell-syntax-rules.md
33
+
34
+ ## USE FOR:
35
+ - session-start OS/shell/toolchain detection
36
+ - per-terminal command/path/env-var alignment
37
+ - repairing missing harness touchpoints
38
+
39
+ ## DO NOT USE FOR:
40
+ - operating outside the current workspace
41
+ - installing toolchains from scratch
@@ -0,0 +1,76 @@
1
+ # Environment Detection — Full Shell Syntax Rules
2
+
3
+ ## Decision Flow
4
+
5
+ ```mermaid
6
+ flowchart TD
7
+ Start[Session Start / Discover Phase] --> CheckNode{1. Node.js Executable?}
8
+
9
+ CheckNode -- Yes --> RunPreflight[Run scripts/preflight.js]
10
+ RunPreflight -- Exit 0 (Success) --> ParseOutput[Parse OS, Shell & Available CLI Tools]
11
+ RunPreflight -- Fails --> Heuristic
12
+
13
+ CheckNode -- No --> Heuristic{2. Environment Heuristic Inspection Fallback}
14
+
15
+ Heuristic --> CheckPrompt[Inspect <environment_info> & Session Context]
16
+ CheckPrompt --> ProbeShell[Probe Shell via echo $SHELL / $env:OS / %COMSPEC%]
17
+
18
+ ParseOutput --> AlignSyntax[Adopt Command & Path Syntax Rules]
19
+ ProbeShell --> AlignSyntax
20
+ AlignSyntax --> SelfHeal[Run self-heal.js if Node available]
21
+ SelfHeal --> Done[Environment Context Established]
22
+ ```
23
+
24
+ ## Cognitive loop detail
25
+
26
+ - **[Think]**: Before running any terminal commands, check your environment. Blind execution is the primary source of agent step waste.
27
+ - **[Try]**: Run the automated preflight check script to determine exact platform properties.
28
+ - **[Summarize]**: Output the detected terminal, OS, and toolchains so the session context retains these bounds.
29
+ - **[Record]**: Adopt the corresponding command syntax rules for the rest of the conversation.
30
+
31
+ ## Strict Workspace Boundary detail (Current Active Workspace Only)
32
+
33
+ - **Current Workspace Only**: Must detect and operate solely within the active workspace directory (i.e. the current project root, which is `process.cwd()`).
34
+ - **Ignore Other Workspaces**: Even if VS Code context or history exposes other workspaces, temporary paths, or recently opened projects (e.g. `d:\super.h2o.sbom`, `d:\DeveloperDocs`, `c:\Users\DanielCH.Chang\Desktop\XinputDxe_V0.0.5`, etc.), you **MUST completely ignore them**. Never perform structural analysis, tool checks, or execute terminal commands in those non-current project paths.
35
+ - **Single-Workspace Execution**: All terminal commands, tool availability checks, and path resolutions MUST be strictly restricted to the current project root directory. Operating across boundaries on adjacent or unrelated folders is strictly prohibited.
36
+
37
+ ## Preflight & self-heal detail
38
+
39
+ Run the preflight script that lives at `scripts/preflight.js` inside this skill's own directory — resolve the path from wherever this SKILL.md was loaded (do not guess a hard-coded location). Parse the output to understand:
40
+
41
+ - Operating System (Windows vs. macOS vs. Linux)
42
+ - Active Shell (Git Bash vs. PowerShell vs. Command Prompt)
43
+ - Available CLI Tools (node, pnpm, docker, python, etc.)
44
+
45
+ Toolchain self-heal: the harness itself is part of the environment. Audit whether this workspace's integration touchpoints (`.claude/settings.json` hooks, `.cursorrules`, `.github/copilot-instructions.md`, `AGENTS.md`) are installed, and repair any missing ones — e.g., installed via Claude Code but now opened in Copilot. The script is idempotent (it delegates to the installer, which merges hooks by id and never duplicates advisory text), so running it on an already-healthy workspace is a no-op. Skip only if the user intentionally removed a touchpoint file.
46
+
47
+ ## Shell syntax rules in full
48
+
49
+ ### 1. Git Bash on Windows (User's Preferred Terminal)
50
+
51
+ - **Path Slashes**: Always use forward slashes `/` (e.g., `scripts/lib/utils.js`). Never use raw backslashes `\` because they act as escape characters in Bash.
52
+ - **Env Variables**: Use `$VARIABLE_NAME` syntax. Do NOT use `%VARIABLE_NAME%` or `$env:VARIABLE_NAME`.
53
+ - **Command Set**: Standard Unix commands are fully supported (`ls`, `rm -rf`, `mkdir -p`, `cp`, `mv`). Do NOT run CMD native commands like `dir`, `del`, or `copy`.
54
+ - **No PowerShell syntax**: Do not run PowerShell-specific scripts or syntax.
55
+
56
+ ### 2. PowerShell on Windows
57
+
58
+ - **Path Slashes**: Forward slashes `/` or backslashes `\\` are both acceptable.
59
+ - **Env Variables**: Use `$env:VARIABLE_NAME` syntax.
60
+ - **Command Set**: Use standard PowerShell commands or common aliases (e.g., `New-Item`, `Remove-Item`).
61
+
62
+ ### 3. Windows Command Prompt (CMD)
63
+
64
+ - **Path Slashes**: Use backslashes `\` for file paths.
65
+ - **Env Variables**: Use `%VARIABLE_NAME%` syntax.
66
+ - **Command Set**: Use `dir`, `del`, `copy`, `mkdir`. Do NOT use `ls`, `rm`, `cp`, `mkdir -p`.
67
+
68
+ ## Red flags & common errors
69
+
70
+ - ❌ **CMD commands in Git Bash**: Attempting to run `dir` or `del` when the terminal is actually running Git Bash.
71
+ - ❌ **PowerShell commands in Bash**: Attempting to run `$env:PATH` or `.ps1` files in a Bash environment.
72
+ - ❌ **Path escape failures**: Sending raw Windows paths like `scripts\run.js` inside a Bash terminal, which interprets `\r` as a carriage return or escapes characters.
73
+
74
+ ## Verification & reflection
75
+
76
+ If any shell command fails, do not blindly retry the exact same command. Stop and ask yourself: "Did I use the wrong syntax for this terminal? Is this a Git Bash vs. PowerShell syntax issue?" If you fail 3 times on the same command, trigger the `zoom-out` circuit breaker.
@@ -1,93 +1,103 @@
1
- #!/usr/bin/env node
2
- const { execSync } = require('child_process');
3
- const os = require('os');
4
- const fs = require('fs');
5
- const path = require('path');
6
-
7
- function checkCmd(cmd) {
8
- try {
9
- if (os.platform() === 'win32') {
10
- execSync(`where ${cmd}`, { stdio: 'ignore' });
11
- } else {
12
- execSync(`which ${cmd}`, { stdio: 'ignore' });
13
- }
14
- return true;
15
- } catch (e) {
16
- try {
17
- execSync(`${cmd} --version`, { stdio: 'ignore' });
18
- return true;
19
- } catch (e2) {
20
- return false;
21
- }
22
- }
23
- }
24
-
25
- console.log("# [Environment Preflight Report]");
26
- console.log(`- **Operating System**: ${os.platform()} (${os.release()} ${os.arch()})`);
27
-
28
- // Shell Detection
29
- let shell = process.env.SHELL || process.env.COMSPEC || "Unknown";
30
- let shellName = "Unknown Shell";
31
-
32
- if (shell.toLowerCase().includes("bash")) {
33
- shellName = "Git Bash / Bash";
34
- } else if (shell.toLowerCase().includes("powershell") || shell.toLowerCase().includes("pwsh") || process.env.PSModulePath) {
35
- shellName = "PowerShell";
36
- } else if (shell.toLowerCase().includes("cmd.exe")) {
37
- shellName = "Windows Command Prompt (CMD)";
38
- } else {
39
- shellName = shell;
40
- }
41
-
42
- console.log(`- **Active Shell**: ${shellName} (Path: \`${shell}\`)`);
43
- console.log(`- **Terminal Program**: ${process.env.TERM_PROGRAM || "Standard Terminal"}`);
44
-
45
- // CLI Tools Check
46
- const tools = ['git', 'docker', 'docker-compose', 'node', 'pnpm', 'npm', 'python', 'pip'];
47
- const availableTools = [];
48
- for (const tool of tools) {
49
- if (checkCmd(tool)) {
50
- availableTools.push(tool);
51
- }
52
- }
53
- console.log(`- **Available CLI Tools**: ${availableTools.join(', ') || 'None'}`);
54
-
55
- // Sandbox Capabilities Detection
56
- const hasDocker = availableTools.includes('docker');
57
- const hasSandboxExec = checkCmd('sandbox-exec');
58
- const sandboxCapabilities = [];
59
- if (hasSandboxExec) sandboxCapabilities.push('macOS Seatbelt (sandbox-exec)');
60
- if (hasDocker) sandboxCapabilities.push('Docker Containers');
61
- if (os.platform() === 'win32') {
62
- sandboxCapabilities.push('Windows AppContainer (Native)');
63
- }
64
- console.log(`- **Sandbox Capabilities**: ${sandboxCapabilities.join(', ') || 'No isolation sandbox detected'}`);
65
-
66
- // Repo Status
67
- try {
68
- const branch = execSync('git branch --show-current', { encoding: 'utf8' }).trim();
69
- console.log(`- **Git Workspace**: Detected (Branch: \`${branch}\`)`);
70
- } catch (e) {
71
- console.log(`- **Git Workspace**: No git repository detected in ${process.cwd()}`);
72
- }
73
-
74
- // Recommended Syntax Guardrails
75
- console.log("\n## [Syntax & Command Guardrails for this Environment]");
76
- if (os.platform() === 'win32') {
77
- if (shellName.includes("Bash")) {
78
- console.log("- **Path Handling**: ALWAYS use `/` slashes for directory paths in commands (e.g., `scripts/run.js`). Avoid raw backslashes `\\` as they act as escape chars.");
79
- console.log("- **Environment Variables**: Use `$VAR` syntax. Do NOT use `%VAR%` or `$env:VAR`.");
80
- console.log("- **Command Chaining**: Use `&&` for sequence, `|` for piping.");
81
- console.log("- **Avoid Native Windows Commands**: Do NOT use `dir`, `copy`, or `del`. Use bash standard equivalents `ls`, `cp`, `rm`.");
82
- } else if (shellName.includes("PowerShell")) {
83
- console.log("- **Path Handling**: Slashes `/` or `\\` are generally acceptable.");
84
- console.log("- **Environment Variables**: Use `$env:VAR` syntax.");
85
- console.log("- **Command Chaining**: Use `;` or `&&` (if PowerShell 7+). Avoid bash-specific commands like `grep`, `awk`, `sed` unless explicitly piping.");
86
- } else {
87
- console.log("- **Path Handling**: Use `\\` for directories.");
88
- console.log("- **Environment Variables**: Use `%VAR%`.");
89
- console.log("- **Chaining**: Use `&`.");
90
- }
91
- } else {
92
- console.log("- **Unix Native**: Standard Bash/Zsh syntax applies. Use relative paths, pipeline chaining, and standard utilities.");
93
- }
1
+ #!/usr/bin/env node
2
+ const { execSync } = require('child_process');
3
+ const os = require('os');
4
+ const fs = require('fs');
5
+ const path = require('path');
6
+
7
+ function checkCmd(cmd) {
8
+ try {
9
+ if (os.platform() === 'win32') {
10
+ execSync(`where "${cmd}"`, { stdio: 'ignore' });
11
+ } else {
12
+ execSync(`which "${cmd}"`, { stdio: 'ignore' });
13
+ }
14
+ return true;
15
+ } catch (e) {
16
+ try {
17
+ execSync(`${cmd} --version`, { stdio: 'ignore' });
18
+ return true;
19
+ } catch (e2) {
20
+ return false;
21
+ }
22
+ }
23
+ }
24
+
25
+ console.log("# [Environment Preflight Report]");
26
+ console.log(`- **Operating System**: ${os.platform()} (${os.release()} ${os.arch()})`);
27
+
28
+ // Shell Detection
29
+ const isGitBash = !!(
30
+ process.env.MSYSTEM ||
31
+ process.env.BASH_VERSION ||
32
+ (process.env.SHELL && process.env.SHELL.toLowerCase().includes("bash"))
33
+ );
34
+ const isPowerShell = !!(
35
+ process.env.PSModulePath ||
36
+ (process.env.SHELL && (process.env.SHELL.toLowerCase().includes("powershell") || process.env.SHELL.toLowerCase().includes("pwsh")))
37
+ );
38
+
39
+ let shell = process.env.SHELL || (isGitBash ? "bash" : (isPowerShell ? "powershell" : process.env.COMSPEC)) || "Unknown";
40
+ let shellName = "Unknown Shell";
41
+
42
+ if (isGitBash || shell.toLowerCase().includes("bash")) {
43
+ shellName = "Git Bash / Bash";
44
+ } else if (isPowerShell) {
45
+ shellName = "PowerShell";
46
+ } else if (shell.toLowerCase().includes("cmd.exe")) {
47
+ shellName = "Windows Command Prompt (CMD)";
48
+ } else {
49
+ shellName = shell;
50
+ }
51
+
52
+ console.log(`- **Active Shell**: ${shellName} (Path: \`${shell}\`)`);
53
+ console.log(`- **Terminal Program**: ${process.env.TERM_PROGRAM || "Standard Terminal"}`);
54
+
55
+ // CLI Tools Check
56
+ const tools = ['git', 'docker', 'docker-compose', 'node', 'pnpm', 'npm', 'python', 'pip'];
57
+ const availableTools = [];
58
+ for (const tool of tools) {
59
+ if (checkCmd(tool)) {
60
+ availableTools.push(tool);
61
+ }
62
+ }
63
+ console.log(`- **Available CLI Tools**: ${availableTools.join(', ') || 'None'}`);
64
+
65
+ // Sandbox Capabilities Detection
66
+ const hasDocker = availableTools.includes('docker');
67
+ const hasSandboxExec = checkCmd('sandbox-exec');
68
+ const sandboxCapabilities = [];
69
+ if (hasSandboxExec) sandboxCapabilities.push('macOS Seatbelt (sandbox-exec)');
70
+ if (hasDocker) sandboxCapabilities.push('Docker Containers');
71
+ if (os.platform() === 'win32') {
72
+ sandboxCapabilities.push('Windows AppContainer (Native)');
73
+ }
74
+ console.log(`- **Sandbox Capabilities**: ${sandboxCapabilities.join(', ') || 'No isolation sandbox detected'}`);
75
+
76
+ // Repo Status
77
+ try {
78
+ const branch = execSync('git branch --show-current', { encoding: 'utf8' }).trim();
79
+ console.log(`- **Git Workspace**: Detected (Branch: \`${branch}\`)`);
80
+ } catch (e) {
81
+ console.log(`- **Git Workspace**: No git repository detected in ${process.cwd()}`);
82
+ }
83
+
84
+ // Recommended Syntax Guardrails
85
+ console.log("\n## [Syntax & Command Guardrails for this Environment]");
86
+ if (os.platform() === 'win32') {
87
+ if (shellName.includes("Bash")) {
88
+ console.log("- **Path Handling**: ALWAYS use `/` slashes for directory paths in commands (e.g., `scripts/run.js`). Avoid raw backslashes `\\` as they act as escape chars.");
89
+ console.log("- **Environment Variables**: Use `$VAR` syntax. Do NOT use `%VAR%` or `$env:VAR`.");
90
+ console.log("- **Command Chaining**: Use `&&` for sequence, `|` for piping.");
91
+ console.log("- **Avoid Native Windows Commands**: Do NOT use `dir`, `copy`, or `del`. Use bash standard equivalents `ls`, `cp`, `rm`.");
92
+ } else if (shellName.includes("PowerShell")) {
93
+ console.log("- **Path Handling**: Slashes `/` or `\\` are generally acceptable.");
94
+ console.log("- **Environment Variables**: Use `$env:VAR` syntax.");
95
+ console.log("- **Command Chaining**: Use `;` or `&&` (if PowerShell 7+). Avoid bash-specific commands like `grep`, `awk`, `sed` unless explicitly piping.");
96
+ } else {
97
+ console.log("- **Path Handling**: Use `\\` for directories.");
98
+ console.log("- **Environment Variables**: Use `%VAR%`.");
99
+ console.log("- **Chaining**: Use `&`.");
100
+ }
101
+ } else {
102
+ console.log("- **Unix Native**: Standard Bash/Zsh syntax applies. Use relative paths, pipeline chaining, and standard utilities.");
103
+ }
@@ -0,0 +1,14 @@
1
+ ---
2
+ name: bad-frontmatter-colon
3
+ description: Bad unquoted colon: here
4
+ version: 0.3.4
5
+ metadata:
6
+ author: Miya Daniel
7
+ version: 0.3.4
8
+ ---
9
+
10
+ ## USE FOR:
11
+ Testing negative control for unquoted YAML colon
12
+
13
+ ## DO NOT USE FOR:
14
+ Production use
@@ -0,0 +1,14 @@
1
+ ---
2
+ name: duplicate-desc-skill-a
3
+ description: Test skill with identical description for collision detection
4
+ version: 0.3.4
5
+ metadata:
6
+ author: Miya Daniel
7
+ version: 0.3.4
8
+ ---
9
+
10
+ ## USE FOR:
11
+ Testing negative control for duplicate description detection
12
+
13
+ ## DO NOT USE FOR:
14
+ Production use
@@ -0,0 +1,14 @@
1
+ ---
2
+ name: duplicate-desc-skill-b
3
+ description: Test skill with identical description for collision detection
4
+ version: 0.3.4
5
+ metadata:
6
+ author: Miya Daniel
7
+ version: 0.3.4
8
+ ---
9
+
10
+ ## USE FOR:
11
+ Testing negative control for duplicate description detection
12
+
13
+ ## DO NOT USE FOR:
14
+ Production use
@@ -0,0 +1,13 @@
1
+ ---
2
+ name: missing-delimiter
3
+ description: Missing end delimiter
4
+ version: 0.3.4
5
+ metadata:
6
+ author: Miya Daniel
7
+ version: 0.3.4
8
+
9
+ ## USE FOR:
10
+ Testing negative control for missing delimiter
11
+
12
+ ## DO NOT USE FOR:
13
+ Production use
@@ -0,0 +1,36 @@
1
+ ---
2
+ name: over-budget-skill
3
+ description: Over budget skill for token limit testing
4
+ version: 0.3.4
5
+ metadata:
6
+ author: Miya Daniel
7
+ version: 0.3.4
8
+ ---
9
+
10
+ ## USE FOR:
11
+ This skill is intentionally over the word budget to test the token limit enforcement gate. It contains a very long description and body that exceeds the 500 word hard limit. The purpose of this skill is to verify that the consistency-check.js script correctly detects when a SKILL.md file exceeds the token budget and fails the check. This is a negative control fixture that must fail the token budget gate.
12
+
13
+ The skill description is intentionally verbose to push the word count over the limit. In a real skill, this would be a routing surface issue where the description is too long and bloats token budgets for agent routing. The token budget gate in consistency-check.js uses a word-count proxy (1.55 tokens per word) to estimate token usage and fails when the hard limit of 500 tokens is exceeded.
14
+
15
+ This negative control ensures that the gate actually works and catches over-budget skills. Without this fixture, a gate that silently passes everything would be indistinguishable from a working gate.
16
+
17
+ Additional content to push word count over 500 words. This skill contains extensive documentation about how the token budget enforcement works in the Harness system. The gate uses a simple word-count heuristic where each word is estimated to be approximately 1.55 tokens. This means 330 words roughly equals 511 tokens, which exceeds the 500 token hard limit. The gate has two thresholds: a warning at 400 words (80% of hard limit) and a hard failure at 500 words.
18
+
19
+ The word counting is performed on the body of the SKILL.md file after removing the frontmatter. This means the frontmatter fields like name, description, version, and metadata are not counted toward the word budget. Only the markdown content in the body sections (USE FOR, DO NOT USE FOR, and any additional reference content) contributes to the word count.
20
+
21
+ To create a skill that exceeds the budget, this fixture includes many paragraphs of explanatory text. Each paragraph adds to the word count. The test verifies that when a skill exceeds 500 words, the consistency-check.js script fails with an appropriate error message indicating the number of words over the limit.
22
+
23
+ This is a comprehensive test of the token budget enforcement mechanism. The gate is designed to be a local early-warning system that catches token budget issues before they reach CI, where waza tokens check would also catch them. By having both local and CI gates, we ensure that token budget violations are caught at multiple stages.
24
+
25
+ Production systems should never have skills that exceed the token budget because long descriptions bloat the routing context and make agent selection less efficient. The routing system uses skill descriptions as the primary signal for skill selection, so keeping them concise and focused is essential for good routing performance.
26
+
27
+ Additional paragraphs to ensure we exceed the five hundred word threshold. Each sentence here adds multiple words to the total count. We need to reach at least five hundred and one words to trigger the hard failure gate. The consistency check will then report the exact number of words over the limit and fail the build. This is the expected behavior for the negative control test.
28
+
29
+ More content to push the word count higher. The word counting algorithm splits on whitespace and filters out empty strings. This means each space-separated token counts as a word. Punctuation attached to words does not affect the count. We are now adding several more sentences to ensure we cross the five hundred word boundary with comfortable margin.
30
+
31
+ Even more text to guarantee the word count exceeds the hard limit. The test fixture must reliably fail the token budget check every time it runs. Flaky tests that sometimes pass and sometimes fail are worse than no tests at all. By making the content significantly over the limit, we ensure consistent test results.
32
+
33
+ Final paragraph to push us well over the five hundred word target. At this point we should have more than enough words to trigger the hard failure. The consistency check will output an error showing the exact word count and how many words exceed the limit. This confirms the gate is working correctly.
34
+
35
+ ## DO NOT USE FOR:
36
+ Production use - this is a test fixture only
@@ -1,59 +1,42 @@
1
- ---
2
- name: eval-harness
3
- description: Objectively evaluates AI agent performance based on correctness, resource efficiency, and anti-loop focus.
4
- author: Miya Daniel | Harness Core Team
5
- version: 0.2.0
6
- ---
7
-
8
- # Eval Harness (Automated Performance & Reasoning Evaluation)
9
-
10
- ## 📋 Skill Contract
11
-
12
- | Component | Specification |
13
- | :--- | :--- |
14
- | **Trigger / Input** | User requests "run benchmark", "score conversation log", or "compare Vanilla vs Harness differences" — an execution log or the current conversation history. |
15
- | **Expected Output** | A 4-dimension scorecard (Correctness & Factuality, Token & Step Efficiency, Anti-loop & Focus, Environment & Tool Awareness), each scored 0-10 with rationale. |
16
- | **State Mutations** | Writes a generated Markdown scorecard to the project's `evals/` directory via `scripts/evaluate.js`. |
17
- | **Enforcement Gate** | MUST execute `scripts/evaluate.js` with the four scores and insight text — a verbally-stated score without running the script does not count as output. |
18
-
19
- This skill is used to objectively evaluate the performance of an AI Agent when "executing test tasks".
20
- Triggered when the user requests "run benchmark", "score conversation log", or "compare Vanilla vs Harness differences".
21
-
22
- ## 1. Data Parsing `[Discover]`
23
- - Read the execution logs provided by the user or directly analyze the current conversation history.
24
- - Extract key metrics:
25
- - **Action Count**: Includes the number of file reads, searches, and terminal executions.
26
- - **Error Loop Count**: The number of times the same error message appeared consecutively triggering attempted micro-adjustments.
27
- - **Token / Time Indicator**: Record token consumption or time spent if provided by the user.
28
-
29
- ## 2. Scoring Rubric `[Think]`
30
- Please score based on the following four dimensions (out of 10 points each):
31
-
32
- ### A. Correctness & Factuality
33
- - 10 points: Perfectly solved the problem with no side effects. Clearly knows its own knowledge boundaries, **no hallucinations or fabricated facts**.
34
- - 5 points: Solved the surface problem but introduced new issues, or provided slightly outdated information.
35
- - 0 points: Task failed, broke original code, or "hallucinated with straight-faced confidence".
36
-
37
- ### B. Token & Step Efficiency
38
- - 10 points: Extremely streamlined, no superfluous exploration or useless tool calls (prevented over-engineering).
39
- - 5 points: Reasonable steps, but with minor unnecessary file reads.
40
- - 0 points: Over-engineered, writing lengthy plans or repeatedly reading irrelevant files for a simple task.
41
-
42
- ### C. Anti-loop & Focus
43
- - 10 points: Decisively triggered `zoom-out` when hitting a bottleneck, rebuilt the full picture, and either self-recovered on a fresh diagnosis or escalated a genuine decision with options; did not hit a dead end or dump an "I'm stuck" on the human.
44
- - 5 points: Stopped after 3~5 attempts.
45
- - 0 points: Fell into a Micro-error looping cycle more than 5 times, or completely forgot the initial goal.
46
-
47
- ### D. Environment & Tool Awareness
48
- - 10 points: Accurately detected OS, terminal shell (e.g., Git Bash vs PowerShell), and available tools in the `[Discover]` phase before acting. No syntax errors caused by wrong shell assumptions.
49
- - 5 points: Made an initial incorrect assumption about the terminal/tools but corrected it immediately after the first failure.
50
- - 0 points: Blindly threw commands (e.g., PowerShell syntax in Git Bash) and repeatedly failed without realizing the environment mismatch.
51
-
52
- ## 3. Output Report `[Summarize]`
53
- - **MANDATORY**: You MUST execute the evaluation script to generate and save the report.
54
- - Find the `evaluate.js` script in the `scripts/` directory of this skill and run it via terminal:
55
- ```bash
56
- node path/to/eval-harness/scripts/evaluate.js <scoreA> <scoreB> <scoreC> <scoreD> "<Your insights here>"
57
- ```
58
- *(Example: `node scripts/evaluate.js 10 5 10 10 "The agent successfully used zoom-out..."`)*
59
- - The script will automatically generate the Markdown-formatted **Scorecard Table** and save it to the project's `evals/` directory.
1
+ ---
2
+ name: eval-harness
3
+ description: Evaluate AI agent performance across correctness, token efficiency, anti-loop focus, and environment awareness; save a 4-dimension scorecard. Use when asked to run a benchmark or score an execution log or conversation history.
4
+ license: Apache-2.0
5
+ metadata:
6
+ author: Miya Daniel
7
+ version: 0.3.6
8
+ ---
9
+
10
+ # Eval Harness
11
+
12
+ Evaluate an agent log with a four-dimension scorecard.
13
+
14
+ ## Skill Contract
15
+
16
+ | Component | Specification |
17
+ | :--- | :--- |
18
+ | **Trigger / Input** | A benchmark request and an execution log or conversation history. |
19
+ | **Expected Output** | A 0–10 scorecard for correctness, efficiency, anti-loop focus, and environment awareness. |
20
+ | **State Mutations** | Writes the scorecard under `evals/` or `.github/harness-everything/evals/`. |
21
+ | **Enforcement Gate** | Run `scripts/evaluate.js`; use the documented fallback only if it fails or is missing. |
22
+
23
+ ## Workflow
24
+
25
+ 1. Parse actions, errors, tokens, and timing from the log.
26
+ 2. Score all four dimensions using `references/scoring-rubric.md`.
27
+ 3. Run `node scripts/evaluate.js <A> <B> <C> <D> "insights"`.
28
+ 4. If the script fails, write `evals/scorecard.md` (or the `.github` fallback).
29
+
30
+ ## USE FOR:
31
+
32
+ - Running a benchmark or scoring an agent execution.
33
+ - Comparing Vanilla and Harness execution.
34
+ - Generating a four-dimension scorecard.
35
+
36
+ ## DO NOT USE FOR:
37
+
38
+ - Writing or fixing the evaluated code.
39
+ - General review without scoring dimensions.
40
+ - Tasks without an execution log.
41
+
42
+ Deep dive: `references/scoring-rubric.md`