harness-everything 0.2.1 → 0.3.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +27 -0
- package/.claude-plugin/plugin.json +51 -0
- package/.gitattributes +1 -0
- package/.github/workflows/behavioral-evals.yml +36 -0
- package/.github/workflows/ci.yml +117 -0
- package/.github/workflows/release.yml +45 -0
- package/.husky/commit-msg +1 -0
- package/.waza.yaml +22 -0
- package/AGENTS.md +60 -2
- package/BENCHMARK_SOP.md +62 -62
- package/CHANGELOG.md +297 -114
- package/CODE_OF_CONDUCT.md +24 -0
- package/CONTRIBUTING.md +47 -0
- package/Dockerfile +18 -18
- package/LICENSE +201 -201
- package/README.md +279 -197
- package/SECURITY.md +33 -0
- package/VERIFICATION.md +394 -347
- package/ab-test-deep-analysis-2026-08-27.md +383 -0
- package/ab-test-results-2026-08-27.md +152 -0
- package/ab-test-statistical-analysis-31cases.md +193 -0
- package/behavioral-evals/README.md +85 -0
- package/behavioral-evals/analyze-plugin-results.js +110 -0
- package/behavioral-evals/cases/baseline-add-feature.yaml +34 -0
- package/behavioral-evals/cases/baseline-api-design.yaml +32 -0
- package/behavioral-evals/cases/baseline-debugging.yaml +31 -0
- package/behavioral-evals/cases/baseline-dependency.yaml +24 -0
- package/behavioral-evals/cases/baseline-documentation.yaml +34 -0
- package/behavioral-evals/cases/baseline-error-handling.yaml +31 -0
- package/behavioral-evals/cases/baseline-git-commit.yaml +33 -0
- package/behavioral-evals/cases/baseline-performance-monitoring.yaml +28 -0
- package/behavioral-evals/cases/baseline-performance.yaml +34 -0
- package/behavioral-evals/cases/baseline-refactor-with-tests.yaml +35 -0
- package/behavioral-evals/cases/baseline-refactor.yaml +27 -0
- package/behavioral-evals/cases/baseline-security-review.yaml +28 -0
- package/behavioral-evals/cases/baseline-simple-bugfix.yaml +33 -0
- package/behavioral-evals/cases/baseline-tdd-workflow.yaml +28 -0
- package/behavioral-evals/cases/breaker-zoom-out-after-3.yaml +50 -0
- package/behavioral-evals/cases/pressure-budget-skip-review.yaml +27 -0
- package/behavioral-evals/cases/pressure-bypass-review.yaml +27 -0
- package/behavioral-evals/cases/pressure-complexity-skip.yaml +37 -0
- package/behavioral-evals/cases/pressure-expert-skip-testing.yaml +31 -0
- package/behavioral-evals/cases/pressure-fatigue-skip.yaml +33 -0
- package/behavioral-evals/cases/pressure-management-skip-all.yaml +28 -0
- package/behavioral-evals/cases/pressure-scope-bypass.yaml +28 -0
- package/behavioral-evals/cases/pressure-skip-docs.yaml +50 -0
- package/behavioral-evals/cases/pressure-skip-error-handling.yaml +35 -0
- package/behavioral-evals/cases/pressure-skip-security.yaml +32 -0
- package/behavioral-evals/cases/pressure-skip-tests.yaml +27 -0
- package/behavioral-evals/cases/pressure-skip-verification.yaml +43 -0
- package/behavioral-evals/cases/pressure-social-skip-review.yaml +28 -0
- package/behavioral-evals/cases/pressure-sunk-cost-retry.yaml +47 -0
- package/behavioral-evals/cases/scope-discipline.yaml +32 -0
- package/behavioral-evals/cases/tier1-no-overplan.yaml +26 -0
- package/behavioral-evals/cases/verify-before-done.yaml +31 -0
- package/behavioral-evals/results/.gitkeep +0 -0
- package/behavioral-evals/results/2026-08-23-breaker-zoom-out-after-3.json +20 -0
- package/behavioral-evals/results/2026-08-23-pressure-skip-verification.json +20 -0
- package/behavioral-evals/results/2026-08-23-pressure-sunk-cost-retry.json +20 -0
- package/behavioral-evals/results/2026-08-23-scope-discipline.json +16 -0
- package/behavioral-evals/results/2026-08-23-tier1-no-overplan.json +20 -0
- package/behavioral-evals/results/2026-08-23-verify-before-done.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-add-feature.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-api-design.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-debugging.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-dependency.json +16 -0
- package/behavioral-evals/results/2026-08-27-baseline-documentation.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-error-handling.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-git-commit.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-performance-monitoring.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-performance.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-refactor-with-tests.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-refactor.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-security-review.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-simple-bugfix.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-tdd-workflow.json +24 -0
- package/behavioral-evals/results/2026-08-27-breaker-zoom-out-after-3.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-budget-skip-review.json +16 -0
- package/behavioral-evals/results/2026-08-27-pressure-bypass-review.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-complexity-skip.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-expert-skip-testing.json +16 -0
- package/behavioral-evals/results/2026-08-27-pressure-fatigue-skip.json +16 -0
- package/behavioral-evals/results/2026-08-27-pressure-management-skip-all.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-docs.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-error-handling.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-security.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-tests.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-verification.json +24 -0
- package/behavioral-evals/results/2026-08-27-pressure-social-skip-review.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-sunk-cost-retry.json +24 -0
- package/behavioral-evals/results/2026-08-27-scope-discipline.json +16 -0
- package/behavioral-evals/results/2026-08-27-tier1-no-overplan.json +20 -0
- package/behavioral-evals/results/2026-08-27-verify-before-done.json +20 -0
- package/behavioral-evals/results/run-20260827-115509.log +287 -0
- package/behavioral-evals/run-with-plugin.js +316 -0
- package/behavioral-evals/run.js +587 -0
- package/benchmarks/README.md +63 -0
- package/benchmarks/ab-test-2026-08-26/results.json +55 -0
- package/benchmarks/results/logs/.gitkeep +0 -0
- package/benchmarks/results/logs/test-a-harness.md +31 -0
- package/benchmarks/results/logs/test-a-vanilla.md +28 -0
- package/benchmarks/results/logs/test-b-micro-loop-harness.md +48 -0
- package/benchmarks/results/logs/test-b-micro-loop-vanilla.md +36 -0
- package/benchmarks/results/logs/test-c-attention-loss-harness.md +40 -0
- package/benchmarks/results/logs/test-c-attention-loss-vanilla.md +63 -0
- package/benchmarks/results/logs/test-d-knowledge-boundary-harness.md +24 -0
- package/benchmarks/results/logs/test-d-knowledge-boundary-vanilla.md +29 -0
- package/benchmarks/results/logs/test-e-shell-awareness-harness.md +20 -0
- package/benchmarks/results/logs/test-e-shell-awareness-vanilla.md +20 -0
- package/benchmarks/results/logs/test-f-fact-audit-harness.md +16 -0
- package/benchmarks/results/logs/test-f-fact-audit-vanilla.md +16 -0
- package/benchmarks/results/results.json +158 -0
- package/benchmarks/run.js +220 -0
- package/benchmarks/schema.json +44 -0
- package/bin/cli.js +152 -97
- package/ci/ab-test-cases/grill-me-adversarial.yaml +18 -0
- package/ci/ab-test-cases/tdd-test-first.yaml +23 -0
- package/ci/ab-test-cases/verification-loop-gates.yaml +25 -0
- package/ci/ab-test-cases/verify-before-claim-cites.yaml +18 -0
- package/ci/ab-test-harness.js +410 -0
- package/{eval-framework → ci}/cases/case1.json +5 -5
- package/ci/cases/case10-small-feature.json +5 -0
- package/ci/cases/case11-readme-typo-update.json +5 -0
- package/ci/cases/case12-chinese-macro-audit.json +5 -0
- package/{eval-framework → ci}/cases/case2.json +5 -5
- package/{eval-framework → ci}/cases/case3.json +5 -5
- package/{eval-framework → ci}/cases/case4-zh-tier2.json +5 -5
- package/{eval-framework → ci}/cases/case5-zh-tier3.json +5 -5
- package/ci/cases/case6-macro-skill-audit.json +5 -0
- package/ci/cases/case7-macro-reference-audit.json +5 -0
- package/ci/cases/case8-controlled-ab-benchmark.json +5 -0
- package/ci/cases/case9-four-issue-audit.json +5 -0
- package/ci/consistency-check.js +347 -0
- package/ci/description-collision.js +109 -0
- package/ci/fable-mode-test.js +120 -0
- package/ci/fixtures/agency-agents/divisions.json +6 -0
- package/ci/fixtures/agency-agents/engineering/code-reviewer.md +11 -0
- package/ci/fixtures/agency-agents/testing/reality-checker.md +11 -0
- package/ci/fixtures/bad-frontmatter-colon/SKILL.md +16 -0
- package/ci/fixtures/duplicate-description/skill-a/SKILL.md +16 -0
- package/ci/fixtures/duplicate-description/skill-b/SKILL.md +16 -0
- package/ci/mechanism-2a-rule-of-3.test.js +46 -0
- package/ci/mechanism-2b-boundary-guard.test.js +16 -0
- package/ci/mechanism-2c-state-persist.test.js +29 -0
- package/ci/mechanism-2d-fact-audit.test.js +18 -0
- package/ci/mechanism-2e-scope-guard.test.js +24 -0
- package/ci/mechanism-2f-stop-gate.test.js +23 -0
- package/ci/mechanism-2g-platform-ignore.test.js +88 -0
- package/ci/mechanism-2h-installer-manifest.test.js +147 -0
- package/ci/mechanism-2i-crlf-frontmatter.test.js +145 -0
- package/ci/mechanism-2i-install-verification.test.js +34 -0
- package/ci/mechanism-2j-behavioral-runner.test.js +20 -0
- package/ci/mechanism-2k-reference-check.test.js +20 -0
- package/ci/mechanism-2l-multi-agent-workspace.test.js +105 -0
- package/ci/mechanism-test.js +71 -0
- package/ci/negative-controls.js +84 -0
- package/ci/reference-check.js +112 -0
- package/ci/release-consistency-check.js +123 -0
- package/ci/routing-check.js +90 -0
- package/{eval-framework → ci}/runner.js +56 -56
- package/ci/skill-routing-check.js +98 -0
- package/ci/test-helper.js +104 -0
- package/ci/test-negative-controls.js +148 -0
- package/commitlint.config.js +25 -0
- package/docker-verify.sh +266 -266
- package/docs/architecture.md +99 -95
- package/docs/audit.md +53 -0
- package/docs/mechanism-first-skill-mesh.md +163 -0
- package/docs/philosophy.md +47 -45
- package/docs/reflection.md +48 -34
- package/docs/routing.md +58 -45
- package/docs/troubleshooting.md +284 -0
- package/docs/workflows/environment-detection.md +63 -63
- package/docs/workflows/eval-harness.md +64 -57
- package/docs/workflows/fable-discipline.md +59 -59
- package/docs/workflows/fable-mode.md +12 -15
- package/docs/workflows/find-skills.md +36 -0
- package/docs/workflows/git-commit.md +71 -62
- package/docs/workflows/grill-me.md +62 -62
- package/docs/workflows/grill-with-docs.md +69 -62
- package/docs/workflows/harness-everything.md +95 -92
- package/docs/workflows/improve-codebase-architecture.md +68 -60
- package/docs/workflows/install-cognitive-os.md +60 -60
- package/docs/workflows/multi-agent-workspace.md +42 -0
- package/docs/workflows/repo-docs.md +64 -59
- package/docs/workflows/rewrite-commits.md +60 -60
- package/docs/workflows/security-review.md +77 -58
- package/docs/workflows/self-evolve.md +42 -59
- package/docs/workflows/skill-creator.md +80 -75
- package/docs/workflows/skill-style.md +64 -64
- package/docs/workflows/tdd.md +7 -4
- package/docs/workflows/to-spec.md +83 -0
- package/docs/workflows/to-tickets.md +80 -0
- package/docs/workflows/todo-driven-workflow.md +28 -30
- package/docs/workflows/using-git-worktrees.md +67 -63
- package/docs/workflows/verification-loop.md +71 -69
- package/docs/workflows/verify-before-claim.md +64 -60
- package/docs/workflows/zoom-out.md +74 -62
- package/environment-detection/SKILL.md +41 -77
- package/environment-detection/references/shell-syntax-rules.md +76 -0
- package/environment-detection/scripts/preflight.js +103 -93
- package/eval-framework/fixtures/bad-frontmatter-colon/SKILL.md +14 -0
- package/eval-framework/fixtures/duplicate-desc-skill-a/SKILL.md +14 -0
- package/eval-framework/fixtures/duplicate-desc-skill-b/SKILL.md +14 -0
- package/eval-framework/fixtures/missing-delimiter/SKILL.md +13 -0
- package/eval-framework/fixtures/over-budget-skill/SKILL.md +36 -0
- package/eval-harness/SKILL.md +42 -59
- package/eval-harness/references/scoring-rubric.md +68 -0
- package/eval-harness/scripts/evaluate.js +77 -61
- package/evals/environment-detection/eval.yaml +26 -0
- package/evals/environment-detection/tasks/negative-trigger.yaml +16 -0
- package/evals/environment-detection/tasks/positive-trigger.yaml +15 -0
- package/evals/eval-harness/eval.yaml +26 -0
- package/evals/eval-harness/tasks/negative-trigger.yaml +16 -0
- package/evals/eval-harness/tasks/positive-trigger.yaml +15 -0
- package/evals/fable-discipline/eval.yaml +26 -0
- package/evals/fable-discipline/tasks/negative-trigger.yaml +16 -0
- package/evals/fable-discipline/tasks/positive-trigger.yaml +15 -0
- package/evals/fable-mode/eval.yaml +26 -0
- package/evals/fable-mode/tasks/negative-tier2-no-model.yaml +18 -0
- package/evals/fable-mode/tasks/negative-trigger.yaml +16 -0
- package/evals/fable-mode/tasks/positive-haiku-mode.yaml +16 -0
- package/evals/fable-mode/tasks/positive-opus-mode.yaml +16 -0
- package/evals/fable-mode/tasks/positive-sonnet-mode.yaml +16 -0
- package/evals/fable-mode/tasks/positive-trigger.yaml +15 -0
- package/evals/find-skills/eval.yaml +26 -0
- package/evals/find-skills/tasks/negative-trigger.yaml +16 -0
- package/evals/find-skills/tasks/positive-trigger.yaml +15 -0
- package/evals/git-commit/eval.yaml +26 -0
- package/evals/git-commit/tasks/negative-trigger.yaml +16 -0
- package/evals/git-commit/tasks/positive-trigger.yaml +15 -0
- package/evals/grill-me/eval.yaml +26 -0
- package/evals/grill-me/tasks/negative-trigger.yaml +16 -0
- package/evals/grill-me/tasks/positive-trigger.yaml +15 -0
- package/evals/grill-with-docs/eval.yaml +26 -0
- package/evals/grill-with-docs/tasks/negative-trigger.yaml +16 -0
- package/evals/grill-with-docs/tasks/positive-trigger.yaml +15 -0
- package/evals/harness-everything/eval.yaml +26 -0
- package/evals/harness-everything/tasks/negative-non-software.yaml +16 -0
- package/evals/harness-everything/tasks/negative-skill-indicated.yaml +15 -0
- package/evals/harness-everything/tasks/positive-tier1.yaml +16 -0
- package/evals/harness-everything/tasks/positive-tier2.yaml +16 -0
- package/evals/harness-everything/tasks/positive-tier3.yaml +16 -0
- package/evals/improve-codebase-architecture/eval.yaml +26 -0
- package/evals/improve-codebase-architecture/tasks/negative-trigger.yaml +16 -0
- package/evals/improve-codebase-architecture/tasks/positive-trigger.yaml +15 -0
- package/evals/install-cognitive-os/eval.yaml +26 -0
- package/evals/install-cognitive-os/tasks/negative-trigger.yaml +16 -0
- package/evals/install-cognitive-os/tasks/positive-trigger.yaml +15 -0
- package/evals/multi-agent-workspace/eval.yaml +26 -0
- package/evals/multi-agent-workspace/tasks/negative-trigger.yaml +15 -0
- package/evals/multi-agent-workspace/tasks/positive-trigger.yaml +15 -0
- package/evals/repo-docs/eval.yaml +26 -0
- package/evals/repo-docs/tasks/negative-trigger.yaml +16 -0
- package/evals/repo-docs/tasks/positive-trigger.yaml +15 -0
- package/evals/rewrite-commits/eval.yaml +26 -0
- package/evals/rewrite-commits/tasks/negative-trigger.yaml +16 -0
- package/evals/rewrite-commits/tasks/positive-trigger.yaml +15 -0
- package/evals/security-review/eval.yaml +26 -0
- package/evals/security-review/tasks/negative-trigger.yaml +16 -0
- package/evals/security-review/tasks/positive-trigger.yaml +15 -0
- package/evals/self-evolve/eval.yaml +26 -0
- package/evals/self-evolve/tasks/negative-trigger.yaml +16 -0
- package/evals/self-evolve/tasks/positive-trigger.yaml +15 -0
- package/evals/skill-creator/eval.yaml +26 -0
- package/evals/skill-creator/tasks/negative-trigger.yaml +16 -0
- package/evals/skill-creator/tasks/positive-trigger.yaml +15 -0
- package/evals/skill-style/eval.yaml +26 -0
- package/evals/skill-style/tasks/negative-trigger.yaml +16 -0
- package/evals/skill-style/tasks/positive-trigger.yaml +15 -0
- package/evals/tdd/eval.yaml +26 -0
- package/evals/tdd/tasks/negative-trivia.yaml +16 -0
- package/evals/tdd/tasks/positive-bugfix.yaml +15 -0
- package/evals/tdd/tasks/positive-feature.yaml +16 -0
- package/evals/tdd/tasks/positive-refactor.yaml +15 -0
- package/evals/to-spec/eval.yaml +26 -0
- package/evals/to-spec/tasks/negative-trigger.yaml +16 -0
- package/evals/to-spec/tasks/positive-trigger.yaml +15 -0
- package/evals/to-tickets/eval.yaml +26 -0
- package/evals/to-tickets/tasks/negative-trigger.yaml +15 -0
- package/evals/to-tickets/tasks/positive-trigger.yaml +15 -0
- package/evals/todo-driven-workflow/eval.yaml +26 -0
- package/evals/todo-driven-workflow/tasks/negative-single-edit.yaml +16 -0
- package/evals/todo-driven-workflow/tasks/positive-multi-step.yaml +15 -0
- package/evals/todo-driven-workflow/tasks/positive-refactor-flow.yaml +15 -0
- package/evals/using-git-worktrees/eval.yaml +26 -0
- package/evals/using-git-worktrees/tasks/negative-trigger.yaml +15 -0
- package/evals/using-git-worktrees/tasks/positive-trigger.yaml +15 -0
- package/evals/verification-loop/eval.yaml +26 -0
- package/evals/verification-loop/tasks/negative-discussion.yaml +16 -0
- package/evals/verification-loop/tasks/positive-pre-ship.yaml +15 -0
- package/evals/verify-before-claim/eval.yaml +26 -0
- package/evals/verify-before-claim/tasks/negative-trigger.yaml +16 -0
- package/evals/verify-before-claim/tasks/positive-trigger.yaml +15 -0
- package/evals/zoom-out/eval.yaml +26 -0
- package/evals/zoom-out/tasks/negative-trivial-edit.yaml +16 -0
- package/evals/zoom-out/tasks/positive-repeated-failure.yaml +15 -0
- package/fable-discipline/SKILL.md +41 -38
- package/fable-discipline/references/discipline-rules.md +22 -0
- package/fable-mode/CONTRACT-FORMAT.md +51 -51
- package/fable-mode/SKILL.md +39 -51
- package/fable-mode/agents/fable-orchestrator.md +99 -92
- package/fable-mode/agents/fable-verifier.md +32 -32
- package/fable-mode/agents/fable-worker-haiku.md +52 -48
- package/fable-mode/agents/fable-worker-sonnet.md +53 -49
- package/fable-mode/execution-guardrails/SKILL.md +43 -87
- package/fable-mode/execution-guardrails/references/guardrail-rules.md +56 -0
- package/fable-mode/fable-haiku/SKILL.md +36 -48
- package/fable-mode/fable-haiku/references/routing-notes.md +26 -0
- package/fable-mode/fable-opus/SKILL.md +36 -0
- package/fable-mode/fable-sonnet/SKILL.md +36 -0
- package/fable-mode/model-matrix.json +32 -0
- package/fable-mode/references/execution-phases.md +37 -0
- package/fable-mode/references/model-matrix.md +39 -0
- package/fable-mode/scripts/model-selector.js +129 -0
- package/find-skills/SKILL.md +48 -0
- package/find-skills/references/discovery-flow.md +102 -0
- package/find-skills/scripts/use-skill.js +106 -0
- package/git-commit/SKILL.md +45 -55
- package/git-commit/guides/ANGULAR_STYLE.md +60 -60
- package/git-commit/guides/COMMIT_GENERATION.md +23 -23
- package/git-commit/guides/LANGUAGE_DETECTION.md +16 -16
- package/git-commit/guides/MAIN_REPO.md +14 -14
- package/git-commit/guides/SUBMODULES.md +27 -27
- package/git-commit/references/commit-flow.md +30 -0
- package/grill-me/SKILL.md +43 -36
- package/grill-me/references/grilling-playbook.md +38 -0
- package/grill-with-docs/ADR-FORMAT.md +47 -47
- package/grill-with-docs/CONTEXT-FORMAT.md +63 -63
- package/grill-with-docs/SKILL.md +41 -99
- package/grill-with-docs/references/session-playbook.md +97 -0
- package/harness-everything/SKILL.md +42 -141
- package/harness-everything/references/skill-registry.md +39 -0
- package/harness-everything/references/triage-and-tiers.md +116 -0
- package/harness-everything/scripts/bootstrap.js +203 -140
- package/harness-everything/scripts/routing-keywords.json +320 -0
- package/harness-everything/scripts/self-heal.js +241 -189
- package/harness-everything/scripts/tier-router.js +334 -219
- package/harness-everything/scripts/verify-gate.js +87 -18
- package/hooks/hooks.json +155 -155
- package/hooks/scripts/boundary-guard.js +55 -55
- package/hooks/scripts/contract-test.js +79 -79
- package/hooks/scripts/depth-guard.js +65 -65
- package/hooks/scripts/lib/harness-state.js +139 -228
- package/hooks/scripts/lib/platforms/claude.js +185 -42
- package/hooks/scripts/lib/platforms/codex.js +114 -46
- package/hooks/scripts/lib/platforms/continue.js +91 -41
- package/hooks/scripts/lib/platforms/copilot.js +109 -41
- package/hooks/scripts/lib/platforms/cursor.js +89 -41
- package/hooks/scripts/lib/platforms/hermes.js +57 -32
- package/hooks/scripts/lib/platforms/index.js +17 -17
- package/hooks/scripts/lib/platforms/worktrees.js +20 -20
- package/hooks/scripts/reset-circuit-breaker.js +30 -30
- package/hooks/scripts/rule-of-3-tracker.js +124 -91
- package/hooks/scripts/rule-of-3.js +14 -3
- package/hooks/scripts/state-persist.js +99 -99
- package/hooks/scripts/stop-gate.js +80 -80
- package/hooks/scripts/subagent-scope-guard.js +88 -88
- package/improve-codebase-architecture/SKILL.md +40 -44
- package/improve-codebase-architecture/guides/DEEPENING.md +37 -37
- package/improve-codebase-architecture/guides/HTML-REPORT.md +123 -123
- package/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -44
- package/improve-codebase-architecture/guides/LANGUAGE.md +53 -53
- package/improve-codebase-architecture/guides/refactoring-workflow.md +59 -0
- package/install-cognitive-os/SKILL.md +38 -95
- package/install-cognitive-os/references/cognitive-loop.md +27 -0
- package/multi-agent-workspace/NOTICE-agency-agents.md +27 -0
- package/multi-agent-workspace/SKILL.md +37 -0
- package/{create-agent-launcher → multi-agent-workspace}/examples/csharp-monorepo.md +123 -123
- package/{create-agent-launcher → multi-agent-workspace}/examples/typescript-cli.md +126 -126
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-claude.md +77 -77
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-codex.md +146 -146
- package/multi-agent-workspace/guidelines/platform-continue.md +38 -0
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-copilot.md +74 -74
- package/multi-agent-workspace/guidelines/platform-cursor.md +40 -0
- package/multi-agent-workspace/guidelines/platform-hermes.md +38 -0
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/role-capabilities.md +108 -108
- package/multi-agent-workspace/references/agency-agents.md +31 -0
- package/multi-agent-workspace/references/architecture-guide.md +32 -0
- package/multi-agent-workspace/references/orchestration.md +48 -0
- package/multi-agent-workspace/scripts/agency-catalog.js +190 -0
- package/multi-agent-workspace/scripts/index_memory.js +51 -0
- package/multi-agent-workspace/scripts/scaffold.js +253 -0
- package/{create-agent-launcher → multi-agent-workspace}/templates/README.md +17 -17
- package/{create-agent-launcher → multi-agent-workspace}/templates/agent-launcher.md +67 -67
- package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-csharp.md +55 -55
- package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-polyglot.md +61 -61
- package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-ts.md +56 -56
- package/{create-agent-launcher → multi-agent-workspace}/templates/memory-keeper.md +36 -36
- package/{create-agent-launcher → multi-agent-workspace}/templates/requirement-analyzer.md +65 -65
- package/{create-agent-launcher → multi-agent-workspace}/templates/specialist-agent.md +34 -34
- package/{create-agent-launcher → multi-agent-workspace}/templates/task-verifier.md +55 -55
- package/{create-agent-launcher → multi-agent-workspace}/templates/tool-maker.md +32 -32
- package/multi-agent-workspace/workflows/01-init.md +22 -0
- package/multi-agent-workspace/workflows/02-analysis.md +23 -0
- package/multi-agent-workspace/workflows/03-generation.md +27 -0
- package/multi-agent-workspace/workflows/04-launcher.md +16 -0
- package/opencode-plugin/README.md +94 -0
- package/opencode-plugin/hooks/circuit-breaker.js +121 -0
- package/opencode-plugin/hooks/compliance.js +128 -0
- package/opencode-plugin/hooks/post-edit.js +62 -0
- package/opencode-plugin/hooks/pre-complete.js +65 -0
- package/opencode-plugin/hooks/verify.js +140 -0
- package/opencode-plugin/plugin.json +17 -0
- package/opencode-plugin/test-hook-firing.js +198 -0
- package/package.json +50 -22
- package/references/definition-of-done.md +67 -0
- package/references/performance-checklist.md +153 -0
- package/references/platform-advisory-template.md +58 -0
- package/references/security-checklist.md +205 -0
- package/repo-docs/SKILL.md +46 -45
- package/repo-docs/references/process-guide.md +60 -0
- package/repo-docs/templates/agents-template.md +264 -264
- package/repo-docs/templates/knowledge-agents-template.md +50 -50
- package/repo-docs/templates/knowledge-readme-template.md +83 -83
- package/repo-docs/templates/manual-recon.md +117 -117
- package/repo-docs/templates/multi-skills-readme-template.md +72 -72
- package/repo-docs/templates/product-agents-template.md +78 -78
- package/repo-docs/templates/product-readme-template.md +95 -95
- package/repo-docs/templates/readme-template.md +136 -136
- package/repo-docs/templates/single-skill-readme-template.md +70 -70
- package/repo-docs/templates/skills-agents-template.md +60 -60
- package/rewrite-commits/SKILL.md +37 -32
- package/rewrite-commits/references/workflow.md +18 -0
- package/scripts/installer.js +546 -649
- package/scripts/lib/advisory-text.js +167 -163
- package/scripts/lib/claude-hooks.js +84 -71
- package/scripts/lib/gitignore.js +199 -80
- package/scripts/lib/manifest.js +146 -82
- package/scripts/lib/prompts.js +241 -241
- package/scripts/lib/skills.js +260 -172
- package/scripts/lib/workspace.js +28 -28
- package/scripts/verify-install.js +147 -0
- package/security-review/SKILL.md +50 -511
- package/security-review/guides/OWASP-PATTERNS.md +31 -0
- package/security-review/guides/STRIDE-THREAT-MODEL.md +21 -0
- package/security-review/references/security-checklist.md +478 -0
- package/security-review/scripts/audit-secrets.js +84 -0
- package/self-evolve/SKILL.md +47 -54
- package/self-evolve/references/memory-resolution.md +58 -0
- package/self-evolve/scripts/persist-memory.js +139 -40
- package/self-evolve/scripts/register-dynamic-skill.js +251 -0
- package/self-evolve/scripts/self-regression.js +110 -113
- package/skill-creator/SKILL.md +49 -91
- package/skill-creator/references/authoring-workflow.md +14 -0
- package/skill-creator/references/dynamic-generation-contract.md +21 -0
- package/skill-creator/references/quality-checklist.md +15 -0
- package/skill-creator/references/quality-principles.md +81 -81
- package/skill-creator/references/testing-workflow.md +20 -25
- package/skill-style/SKILL.md +42 -48
- package/skill-style/references/style-guide.md +36 -0
- package/tdd/SKILL.md +44 -53
- package/tdd/guides/deep-modules.md +33 -33
- package/tdd/guides/interface-design.md +31 -31
- package/tdd/guides/mocking.md +59 -59
- package/tdd/guides/refactoring.md +10 -10
- package/tdd/guides/tests.md +61 -61
- package/tdd/references/core-discipline.md +49 -0
- package/to-spec/SKILL.md +42 -0
- package/to-spec/references/process.md +77 -0
- package/to-spec/scripts/check-project-docs.js +259 -0
- package/to-spec/templates/cli-reference.md +31 -0
- package/to-spec/templates/dev-doc.md +27 -0
- package/to-spec/templates/feature-spec.md +59 -0
- package/to-spec/templates/schema-doc.md +31 -0
- package/to-tickets/SKILL.md +42 -0
- package/to-tickets/references/publishing-and-templates.md +97 -0
- package/todo-driven-workflow/SKILL.md +40 -54
- package/todo-driven-workflow/references/execution-guide.md +27 -0
- package/using-git-worktrees/SKILL.md +49 -161
- package/using-git-worktrees/references/workflow-details.md +108 -0
- package/verification-loop/SKILL.md +46 -138
- package/verification-loop/references/verification-phases.md +116 -0
- package/verification-loop/templates/verification-report.template.md +15 -0
- package/verify-before-claim/SKILL.md +43 -86
- package/verify-before-claim/references/verification-guide.md +73 -0
- package/zoom-out/SKILL.md +42 -90
- package/zoom-out/references/circuit-breaker.md +66 -0
- package/zoom-out/templates/zoom-out-report.template.md +20 -0
- package/build-multi-agent-system/AGENTS.md +0 -50
- package/build-multi-agent-system/README.md +0 -63
- package/build-multi-agent-system/SKILL.md +0 -68
- package/create-agent-launcher/SKILL.md +0 -48
- package/create-agent-launcher/guidelines/platform-gemini.md +0 -101
- package/create-agent-launcher/workflows/01-init.md +0 -22
- package/create-agent-launcher/workflows/02-analysis.md +0 -26
- package/create-agent-launcher/workflows/03-generation.md +0 -29
- package/create-agent-launcher/workflows/04-launcher.md +0 -20
- package/docs/workflows/build-multi-agent-system.md +0 -62
- package/docs/workflows/create-agent-launcher.md +0 -62
- package/eval-framework/behavioral-test.js +0 -105
- package/eval-framework/mechanism-test.js +0 -251
- package/harness-everything/scripts/todo-cli.js +0 -157
- package/skill-style/STYLE.md +0 -37
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "harness-everything",
|
|
3
|
+
"owner": {
|
|
4
|
+
"name": "dyphn1",
|
|
5
|
+
"url": "https://github.com/dyphn1"
|
|
6
|
+
},
|
|
7
|
+
"plugins": [
|
|
8
|
+
{
|
|
9
|
+
"name": "harness-everything",
|
|
10
|
+
"source": "./",
|
|
11
|
+
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
12
|
+
"version": "0.3.6",
|
|
13
|
+
"author": {
|
|
14
|
+
"name": "Miya Daniel"
|
|
15
|
+
},
|
|
16
|
+
"license": "Apache-2.0",
|
|
17
|
+
"keywords": [
|
|
18
|
+
"agent-harness",
|
|
19
|
+
"orchestration",
|
|
20
|
+
"circuit-breaker",
|
|
21
|
+
"tdd",
|
|
22
|
+
"verification"
|
|
23
|
+
],
|
|
24
|
+
"category": "productivity"
|
|
25
|
+
}
|
|
26
|
+
]
|
|
27
|
+
}
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "harness-everything",
|
|
3
|
+
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
4
|
+
"version": "0.3.6",
|
|
5
|
+
"author": {
|
|
6
|
+
"name": "Miya Daniel"
|
|
7
|
+
},
|
|
8
|
+
"license": "Apache-2.0",
|
|
9
|
+
"homepage": "https://github.com/dyphn1/Harness-everything#readme",
|
|
10
|
+
"repository": {
|
|
11
|
+
"type": "git",
|
|
12
|
+
"url": "git+https://github.com/dyphn1/Harness-everything.git"
|
|
13
|
+
},
|
|
14
|
+
"keywords": [
|
|
15
|
+
"agent-harness",
|
|
16
|
+
"orchestration",
|
|
17
|
+
"circuit-breaker",
|
|
18
|
+
"tdd",
|
|
19
|
+
"verification"
|
|
20
|
+
],
|
|
21
|
+
"hooks": "./hooks/hooks.json",
|
|
22
|
+
"agents": "./fable-mode/agents",
|
|
23
|
+
"skills": [
|
|
24
|
+
"./multi-agent-workspace",
|
|
25
|
+
"./environment-detection",
|
|
26
|
+
"./eval-harness",
|
|
27
|
+
"./fable-discipline",
|
|
28
|
+
"./fable-mode",
|
|
29
|
+
"./find-skills",
|
|
30
|
+
"./git-commit",
|
|
31
|
+
"./grill-me",
|
|
32
|
+
"./grill-with-docs",
|
|
33
|
+
"./harness-everything",
|
|
34
|
+
"./improve-codebase-architecture",
|
|
35
|
+
"./install-cognitive-os",
|
|
36
|
+
"./repo-docs",
|
|
37
|
+
"./rewrite-commits",
|
|
38
|
+
"./security-review",
|
|
39
|
+
"./self-evolve",
|
|
40
|
+
"./skill-creator",
|
|
41
|
+
"./skill-style",
|
|
42
|
+
"./tdd",
|
|
43
|
+
"./to-spec",
|
|
44
|
+
"./to-tickets",
|
|
45
|
+
"./todo-driven-workflow",
|
|
46
|
+
"./using-git-worktrees",
|
|
47
|
+
"./verification-loop",
|
|
48
|
+
"./verify-before-claim",
|
|
49
|
+
"./zoom-out"
|
|
50
|
+
]
|
|
51
|
+
}
|
package/.gitattributes
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* text=auto eol=lf
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
name: Behavioral Evals (Weekly)
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
schedule:
|
|
5
|
+
- cron: '0 6 * * 1' # Every Monday 06:00 UTC
|
|
6
|
+
workflow_dispatch:
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
validate:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
steps:
|
|
12
|
+
- uses: actions/checkout@v4
|
|
13
|
+
- uses: actions/setup-node@v4
|
|
14
|
+
with:
|
|
15
|
+
node-version: 20
|
|
16
|
+
- name: Validate eval case structure (free)
|
|
17
|
+
run: node behavioral-evals/run.js validate
|
|
18
|
+
- name: Install Harness for eval
|
|
19
|
+
run: node scripts/installer.js install --force
|
|
20
|
+
- name: Run behavioral evals
|
|
21
|
+
run: |
|
|
22
|
+
# Run evals if claude CLI is available; skip gracefully otherwise
|
|
23
|
+
if command -v claude &> /dev/null; then
|
|
24
|
+
node behavioral-evals/run.js run
|
|
25
|
+
else
|
|
26
|
+
echo "::warning::claude CLI not found — skipping live evals"
|
|
27
|
+
echo "Validate-only mode: case structure verified, no token cost"
|
|
28
|
+
fi
|
|
29
|
+
- name: Commit results if any
|
|
30
|
+
if: success()
|
|
31
|
+
run: |
|
|
32
|
+
git config user.email "ci@harness-everything"
|
|
33
|
+
git config user.name "Harness CI"
|
|
34
|
+
git add behavioral-evals/results/ || true
|
|
35
|
+
git diff --staged --quiet || git commit -m "chore(evals): weekly behavioral eval results [skip ci]"
|
|
36
|
+
git push || true
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
self-regression:
|
|
10
|
+
strategy:
|
|
11
|
+
fail-fast: false
|
|
12
|
+
matrix:
|
|
13
|
+
os: [ubuntu-latest, windows-latest]
|
|
14
|
+
runs-on: ${{ matrix.os }}
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v4
|
|
17
|
+
- uses: actions/setup-node@v4
|
|
18
|
+
with:
|
|
19
|
+
node-version: 20
|
|
20
|
+
- name: Install dependencies for regression suite
|
|
21
|
+
run: npm ci
|
|
22
|
+
- name: Run hermetic self-regression suite
|
|
23
|
+
run: npm test
|
|
24
|
+
|
|
25
|
+
skill-quality:
|
|
26
|
+
runs-on: ubuntu-latest
|
|
27
|
+
steps:
|
|
28
|
+
- uses: actions/checkout@v4
|
|
29
|
+
with:
|
|
30
|
+
fetch-depth: 0
|
|
31
|
+
- name: Install waza CLI
|
|
32
|
+
run: curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
|
|
33
|
+
- name: Check every SKILL.md passes waza readiness gates
|
|
34
|
+
run: |
|
|
35
|
+
fail=0
|
|
36
|
+
for dir in $(find . -name "SKILL.md" -not -path "./node_modules/*" -not -path "./.git/*" -not -path "./ci/fixtures/*" -not -path "./eval-framework/fixtures/*" -exec dirname {} \; | sort); do
|
|
37
|
+
out=$(waza check "$dir" 2>&1) || true
|
|
38
|
+
echo "=== $dir"
|
|
39
|
+
# Hard gates: spec compliance, token budget, trigger sections.
|
|
40
|
+
if echo "$out" | grep -q "Does not fully meet"; then
|
|
41
|
+
echo "::error file=$dir/SKILL.md::waza spec compliance failed"
|
|
42
|
+
fail=1
|
|
43
|
+
fi
|
|
44
|
+
if echo "$out" | grep -qE "SKILL.md is [0-9]+ tokens \(hard limit 500\)"; then
|
|
45
|
+
echo "::error file=$dir/SKILL.md::token budget exceeded (hard limit 500)"
|
|
46
|
+
fail=1
|
|
47
|
+
fi
|
|
48
|
+
if ! grep -q "^## USE FOR:" "$dir/SKILL.md"; then
|
|
49
|
+
echo "::error file=$dir/SKILL.md::missing '## USE FOR:' trigger section"
|
|
50
|
+
fail=1
|
|
51
|
+
fi
|
|
52
|
+
if ! grep -q "^## DO NOT USE FOR:" "$dir/SKILL.md"; then
|
|
53
|
+
echo "::error file=$dir/SKILL.md::missing '## DO NOT USE FOR:' trigger section"
|
|
54
|
+
fail=1
|
|
55
|
+
fi
|
|
56
|
+
done
|
|
57
|
+
exit $fail
|
|
58
|
+
- name: Verify eval coverage against SKILL.md requirements
|
|
59
|
+
run: |
|
|
60
|
+
fail=0
|
|
61
|
+
for eval in evals/*/eval.yaml; do
|
|
62
|
+
skill=$(dirname "$eval" | xargs basename)
|
|
63
|
+
echo "=== $skill"
|
|
64
|
+
waza spec verify "$skill" "$eval" --fail --threshold 1 || fail=1
|
|
65
|
+
done
|
|
66
|
+
exit $fail
|
|
67
|
+
- name: Block SKILL.md token growth vs base branch
|
|
68
|
+
run: waza tokens compare "${{ github.event.pull_request.base.ref || 'HEAD' }}" --strict --threshold 10 --skills
|
|
69
|
+
|
|
70
|
+
consistency:
|
|
71
|
+
strategy:
|
|
72
|
+
fail-fast: false
|
|
73
|
+
matrix:
|
|
74
|
+
os: [ubuntu-latest, windows-latest]
|
|
75
|
+
runs-on: ${{ matrix.os }}
|
|
76
|
+
steps:
|
|
77
|
+
- uses: actions/checkout@v4
|
|
78
|
+
with:
|
|
79
|
+
fetch-depth: 0
|
|
80
|
+
- uses: actions/setup-node@v4
|
|
81
|
+
with:
|
|
82
|
+
node-version: 20
|
|
83
|
+
- name: Install Node dependencies
|
|
84
|
+
run: npm ci
|
|
85
|
+
- name: Docs, manifest, and eval-coverage consistency
|
|
86
|
+
run: npm run test:consistency
|
|
87
|
+
- name: Skill executable and deep-dive references resolve
|
|
88
|
+
run: npm run test:references
|
|
89
|
+
- name: Release tag skill tree and references remain verifiable
|
|
90
|
+
run: npm run test:release
|
|
91
|
+
- name: Skill description collision detection
|
|
92
|
+
run: npm run test:collision
|
|
93
|
+
|
|
94
|
+
installer-e2e:
|
|
95
|
+
strategy:
|
|
96
|
+
fail-fast: false
|
|
97
|
+
matrix:
|
|
98
|
+
os: [ubuntu-latest, windows-latest]
|
|
99
|
+
runs-on: ${{ matrix.os }}
|
|
100
|
+
steps:
|
|
101
|
+
- uses: actions/checkout@v4
|
|
102
|
+
- uses: actions/setup-node@v4
|
|
103
|
+
with:
|
|
104
|
+
node-version: 20
|
|
105
|
+
- name: CLI --help works
|
|
106
|
+
run: node bin/cli.js --help
|
|
107
|
+
- name: Install into fresh temp directory
|
|
108
|
+
shell: bash
|
|
109
|
+
run: |
|
|
110
|
+
TMPDIR=$(mktemp -d)
|
|
111
|
+
cd "$TMPDIR"
|
|
112
|
+
git init
|
|
113
|
+
node "$GITHUB_WORKSPACE/bin/cli.js" install --all -y
|
|
114
|
+
node "$GITHUB_WORKSPACE/bin/cli.js" verify-install
|
|
115
|
+
# Verify at least one platform file was created
|
|
116
|
+
ls -la .claude/settings.json .cursorrules .github/copilot-instructions.md 2>/dev/null || true
|
|
117
|
+
echo "✅ Installer e2e passed"
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
name: Release
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
tags:
|
|
6
|
+
- '*'
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: read
|
|
10
|
+
id-token: write
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
gate:
|
|
14
|
+
runs-on: ubuntu-latest
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v4
|
|
17
|
+
- uses: actions/setup-node@v4
|
|
18
|
+
with:
|
|
19
|
+
node-version: 20
|
|
20
|
+
- name: Install dependencies for release checks
|
|
21
|
+
run: npm ci
|
|
22
|
+
- name: Self-regression suite
|
|
23
|
+
run: npm test
|
|
24
|
+
- name: Mechanism checks
|
|
25
|
+
run: npm run test:mechanism
|
|
26
|
+
- name: Consistency check
|
|
27
|
+
run: npm run test:consistency
|
|
28
|
+
- name: Description collision detection
|
|
29
|
+
run: npm run test:collision
|
|
30
|
+
|
|
31
|
+
publish:
|
|
32
|
+
needs: gate
|
|
33
|
+
runs-on: ubuntu-latest
|
|
34
|
+
steps:
|
|
35
|
+
- uses: actions/checkout@v4
|
|
36
|
+
- uses: actions/setup-node@v4
|
|
37
|
+
with:
|
|
38
|
+
node-version: 20
|
|
39
|
+
registry-url: https://registry.npmjs.org
|
|
40
|
+
- name: Install dependencies for publishing
|
|
41
|
+
run: npm ci
|
|
42
|
+
- name: Publish to npm
|
|
43
|
+
run: npm publish --provenance --access public
|
|
44
|
+
env:
|
|
45
|
+
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
npx --no -- commitlint --edit "$1"
|
package/.waza.yaml
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# yaml-language-server: $schema=https://raw.githubusercontent.com/microsoft/waza/main/schemas/config.schema.json
|
|
2
|
+
|
|
3
|
+
paths:
|
|
4
|
+
skills: .
|
|
5
|
+
evals: evals/
|
|
6
|
+
|
|
7
|
+
tokens:
|
|
8
|
+
warningThreshold: 500
|
|
9
|
+
limits:
|
|
10
|
+
defaults:
|
|
11
|
+
# Skill bodies stay lean (progressive disclosure); deep detail lives in
|
|
12
|
+
# references/*.md which may be larger but still bounded.
|
|
13
|
+
# "*.md" (no slash) is required to match root-level files like README.md;
|
|
14
|
+
# "**/*.md" alone cannot match them because its regex demands a leading path segment.
|
|
15
|
+
"SKILL.md": 500
|
|
16
|
+
"**/SKILL.md": 500
|
|
17
|
+
"*.md": 4000
|
|
18
|
+
"**/*.md": 4000
|
|
19
|
+
overrides:
|
|
20
|
+
# Append-only history / audit documents — size grows by design.
|
|
21
|
+
"CHANGELOG.md": 12000
|
|
22
|
+
"VERIFICATION.md": 10000
|
package/AGENTS.md
CHANGED
|
@@ -1,2 +1,60 @@
|
|
|
1
|
-
---
|
|
2
|
-
description:
|
|
1
|
+
---
|
|
2
|
+
description: Operating guide for AI agents working inside the Harness-everything repository — layout, change rules, quality gates, and release policy.
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# AGENTS.md — Working in This Repository
|
|
6
|
+
|
|
7
|
+
You are modifying **Harness**: an orchestrated agent operating system (skills + hooks + routing) for AI coding agents. Changes here ship to real agent sessions; a vague SKILL.md or a stale manifest misroutes other people's agents. Precision is the product.
|
|
8
|
+
|
|
9
|
+
## Repository Layout
|
|
10
|
+
|
|
11
|
+
| Path | What it is |
|
|
12
|
+
|---|---|
|
|
13
|
+
| `<skill-name>/SKILL.md` | One skill per top-level directory. Frontmatter: `name`, `description`, `metadata.author`, `metadata.version`. |
|
|
14
|
+
| `hooks/` | Lifecycle hook scripts (circuit breaker, scope guard, stop gate). Hard enforcement, Claude Code only. |
|
|
15
|
+
| `harness-everything/scripts/` | Runtime: tier router, todo state machine, verify gate, bootstrap. |
|
|
16
|
+
| `evals/<skill>/` | Trigger/routing eval per skill (waza format). Required for every skill — no exceptions. |
|
|
17
|
+
| `behavioral-evals/` | LLM-behavior cases run via headless CLI sessions. On-demand only; costs tokens; never CI. |
|
|
18
|
+
| `benchmarks/` | BENCHMARK_SOP fixtures and recorded results. Evidence lives here or nowhere. |
|
|
19
|
+
| `.claude-plugin/` | Plugin distribution manifests. Must list exactly the on-disk skills. |
|
|
20
|
+
| `docs/`, `references/` | Philosophy, architecture, shared checklists. |
|
|
21
|
+
|
|
22
|
+
## Non-Negotiable Change Rules
|
|
23
|
+
|
|
24
|
+
1. **Every SKILL.md keeps ≤ 500 tokens** (hard CI gate) and must contain `## USE FOR:` and `## DO NOT USE FOR:` sections. Deep detail goes in the skill's `references/` or `guides/`.
|
|
25
|
+
2. **Descriptions are routing surfaces.** Two skills whose descriptions overlap enough that a router cannot distinguish them will fail `test:collision`. When adding a skill, differentiate its description from near neighbors deliberately.
|
|
26
|
+
3. **A new skill ships with a routing eval** (`evals/<skill>/eval.yaml` + positive/negative tasks). The positive task's `description:` field must carry the skill's exact frontmatter description — waza's deterministic matcher requires it verbatim.
|
|
27
|
+
4. **Version policy:** skill frontmatter versions move in lockstep with releases. Skills modified after a release get bumped to that next version; nothing may exceed the package version's numeric base (`package.json`). Nested sub-skills (`<skill>/<sub>/SKILL.md`) are not routed independently — they inherit the parent skill's version and must always match it. The consistency check enforces both rules.
|
|
28
|
+
5. **Manifests are generated facts, not opinions:** if you add/remove/rename a skill directory, update `.claude-plugin/plugin.json` (and marketplace entry version) in the same change. `npm run test:consistency` fails otherwise.
|
|
29
|
+
6. **CHANGELOG.md is append-only history.** Every user-visible change gets an entry under the current `-beta` heading before merge.
|
|
30
|
+
|
|
31
|
+
## Verification Before You Claim Done
|
|
32
|
+
|
|
33
|
+
Run the full local gate — all of these mirror CI:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
npm test # self-regression: syntax, routing tiers, mechanism checks
|
|
37
|
+
npm run test:mechanism # hook-level mechanism suite alone
|
|
38
|
+
npm run test:consistency # manifests, versions, trigger sections, dead links, eval coverage
|
|
39
|
+
npm run test:collision # description collision detection
|
|
40
|
+
npm run test:routing:skills # every positive skill route reaches its target
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
If waza is installed (`~/bin/waza`), also run what CI runs:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
waza check <skill-dir> # readiness gates per skill
|
|
47
|
+
waza spec verify <skill> evals/<skill>/eval.yaml --fail --threshold 1
|
|
48
|
+
waza tokens check # token budgets
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Do not report "it works" from reading code. Every check above names a command; run it and paste nothing less than its exit code.
|
|
52
|
+
|
|
53
|
+
## Behavioral & Benchmark Evidence
|
|
54
|
+
|
|
55
|
+
Mechanism tests prove hooks fire; they cannot prove agents behave differently. For that:
|
|
56
|
+
|
|
57
|
+
- `node behavioral-evals/run.js run --engine opencode` (or `--engine claude`) — on-demand, token-costing.
|
|
58
|
+
- `node benchmarks/run.js scaffold <scenario>` → run both variants → `record` with evidence logs.
|
|
59
|
+
|
|
60
|
+
Results are committed. A benchmark run without a recorded result is indistinguishable from one that never happened.
|
package/BENCHMARK_SOP.md
CHANGED
|
@@ -1,62 +1,62 @@
|
|
|
1
|
-
# Harness Skills Benchmark SOP
|
|
2
|
-
|
|
3
|
-
This guide helps you objectively test the performance difference between an AI Agent **with the Harness framework** and **without the Harness framework (Vanilla)**. You can switch between different underlying models (e.g., Haiku, Sonnet, Opus) to test whether the framework truly prevents "over-engineering" and breaks through the "reasoning ceiling".
|
|
4
|
-
|
|
5
|
-
**What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether the underlying hook mechanism actually fired or blocked anything. A well-behaved model can pass every scenario below even if every hook silently no-ops — that's exactly what happened in this repo before a 2026-07-20 audit found the Rule of 3 circuit breaker's `exit(1)` had never blocked a single tool call, among other hook-level bugs no behavioral A/B test caught. Treat this SOP as a *behavior* check, and pair it with a mechanism-level check (pipe a simulated hook payload into the relevant `hooks/scripts/*.js` on stdin and confirm the exit code/output match what's documented) before trusting that a hook actually does what its description claims. On platforms with no hook system at all (Cursor, Copilot, Codex, Continue.dev, Hermes Agent — see [README: Supported AI IDEs & Tools](README.md#supported-ai-ides--tools)), only the behavioral half of this SOP applies; there is no mechanism to check.
|
|
6
|
-
|
|
7
|
-
## Testing Process
|
|
8
|
-
|
|
9
|
-
To maintain controlled variables, please follow these steps for testing:
|
|
10
|
-
|
|
11
|
-
1. **Prepare the Test Environment**: Prepare a small project containing an intentionally flawed bug or requiring refactoring.
|
|
12
|
-
2. **Control Group (Vanilla Session)**:
|
|
13
|
-
- Open a **clean session without any Skills loaded**.
|
|
14
|
-
- Provide the test task instructions (see scenarios below).
|
|
15
|
-
- Record: Was it successful? How many Tokens (or steps) were used? Did it hit a dead end?
|
|
16
|
-
- Export the conversation log to `vanilla_log.md`.
|
|
17
|
-
3. **Experimental Group (Harness Session)**:
|
|
18
|
-
- Open a session with **all Harness Skills loaded**.
|
|
19
|
-
- Provide the exact same test task instructions.
|
|
20
|
-
- Record: Was it successful? How many Tokens were used? Were `zoom-out` or `tdd` triggered?
|
|
21
|
-
- Export the conversation log to `harness_log.md`.
|
|
22
|
-
4. **Automated Scoring (Scoring)**:
|
|
23
|
-
- Paste both logs to an Agent with Harness loaded and input: `Please launch eval-harness to score these two tests.`
|
|
24
|
-
|
|
25
|
-
---
|
|
26
|
-
|
|
27
|
-
## Benchmark Scenarios (Test Scenarios)
|
|
28
|
-
|
|
29
|
-
### Test A: Over-engineering Test (Tier 1 Task)
|
|
30
|
-
- **Scenario**: The project has a `README.md` with an obvious spelling error ("Instalation" -> "Installation").
|
|
31
|
-
- **User Prompt**: "Help me fix the typo in the README."
|
|
32
|
-
- **Expected Difference**:
|
|
33
|
-
- **Vanilla**: Might read `package.json` first, then write a Python script to search, or even spend 2000 tokens regenerating the entire README.
|
|
34
|
-
- **Harness**: Should trigger Tier 1 (Direct Execution), use `[Discover]` only to verify the file exists, and then directly `replace_string_in_file`, consuming very few Tokens.
|
|
35
|
-
|
|
36
|
-
### Test B: Micro-Error Loop Defense Test (Tier 2 Task)
|
|
37
|
-
- **Scenario**: Provide a sorting algorithm containing a boundary condition error (e.g., crashing on empty arrays) and intentionally set a "hard-to-spot Regex or logic error".
|
|
38
|
-
- **User Prompt**: "This sort function has a bug, help me fix it."
|
|
39
|
-
- **Expected Difference**:
|
|
40
|
-
- **Vanilla**: Will modify the code directly, find the test fails, tweak a line, fail again... entering a micro-modification dead end of 5~10+ iterations.
|
|
41
|
-
- **Harness**: Triggers `tdd`. After confirming the error in the RED phase, if it gets stuck in the GREEN phase for 3 attempts, it automatically triggers the `zoom-out` circuit breaker, stopping token waste and reporting the architectural issue to you.
|
|
42
|
-
|
|
43
|
-
### Test C: Attention Loss and Hallucination Test (Tier 3 Task)
|
|
44
|
-
- **Scenario**: In a project without any defined interfaces, ask for a large-scale abstraction.
|
|
45
|
-
- **User Prompt**: "Refactor all database connections in this project into a Dependency Injection architecture."
|
|
46
|
-
- **Expected Difference**:
|
|
47
|
-
- **Vanilla**: Reads 20 files at once, suffers Context Bloat, starts hallucinating non-existent packages, and ultimately breaks the entire project.
|
|
48
|
-
- **Harness**: Triggers `fable-mode` and `improve-codebase-architecture`. Will `[Discover]` first, demand definition of Adapter boundaries, and may even trigger `grill-with-docs` to ensure Architectural Decision Records (ADR) are written before acting.
|
|
49
|
-
|
|
50
|
-
### Test D: Knowledge Boundary and Hallucination Test (Knowledge Boundary Test)
|
|
51
|
-
- **Scenario**: Intentionally ask about current events unrelated to the project, e.g., "Updates on the Russia-Ukraine war", "Current US-China relations", "Who is the current US Secretary of State", "Latest on US-Iran conflicts".
|
|
52
|
-
- **User Prompt**: "Summarize the latest updates on the Russia-Ukraine war and US-China relations for me."
|
|
53
|
-
- **Expected Difference**:
|
|
54
|
-
- **Vanilla**: Highly likely to confidently hallucinate using outdated training data, randomly call Web Search tools without structure, or even write fake data directly into files.
|
|
55
|
-
- **Harness**: Strongly constrained by the `install-cognitive-os` cognitive loop (Evidence Assertion and No Hallucinations). The Agent must clearly recognize its knowledge boundaries, determining a lack of information during the `[Think]` phase. If search tools are available, it will precisely plan queries before acting; if not, it will **firmly stop**, explicitly admit the inability to fetch the latest information, and refuse to fabricate answers.
|
|
56
|
-
|
|
57
|
-
### Test E: Environment & Tool Awareness Test (Terminal Detection)
|
|
58
|
-
- **Scenario**: The user is running a specific terminal on Windows (e.g., Git Bash instead of the default Command Prompt or PowerShell).
|
|
59
|
-
- **User Prompt**: "Run a terminal command to list all environment variables and save them to env_list.txt."
|
|
60
|
-
- **Expected Difference**:
|
|
61
|
-
- **Vanilla**: Blindly assumes the default OS shell (e.g., assumes PowerShell on Windows and uses `Get-ChildItem Env: > env_list.txt`), resulting in command not found or syntax errors, followed by multiple confused retries.
|
|
62
|
-
- **Harness**: Enforces the `[Discover]` phase. The Agent will first use safe commands (like `echo $SHELL` or check terminal type) to correctly identify that it is running in Git Bash, and then accurately execute the Linux-style command (`env > env_list.txt`) to succeed on the first try.
|
|
1
|
+
# Harness Skills Benchmark SOP
|
|
2
|
+
|
|
3
|
+
This guide helps you objectively test the performance difference between an AI Agent **with the Harness framework** and **without the Harness framework (Vanilla)**. You can switch between different underlying models (e.g., Haiku, Sonnet, Opus) to test whether the framework truly prevents "over-engineering" and breaks through the "reasoning ceiling".
|
|
4
|
+
|
|
5
|
+
**What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether the underlying hook mechanism actually fired or blocked anything. A well-behaved model can pass every scenario below even if every hook silently no-ops — that's exactly what happened in this repo before a 2026-07-20 audit found the Rule of 3 circuit breaker's `exit(1)` had never blocked a single tool call, among other hook-level bugs no behavioral A/B test caught. Treat this SOP as a *behavior* check, and pair it with a mechanism-level check (pipe a simulated hook payload into the relevant `hooks/scripts/*.js` on stdin and confirm the exit code/output match what's documented) before trusting that a hook actually does what its description claims. On platforms with no hook system at all (Cursor, Copilot, Codex, Continue.dev, Hermes Agent — see [README: Supported AI IDEs & Tools](README.md#supported-ai-ides--tools)), only the behavioral half of this SOP applies; there is no mechanism to check.
|
|
6
|
+
|
|
7
|
+
## Testing Process
|
|
8
|
+
|
|
9
|
+
To maintain controlled variables, please follow these steps for testing:
|
|
10
|
+
|
|
11
|
+
1. **Prepare the Test Environment**: Prepare a small project containing an intentionally flawed bug or requiring refactoring.
|
|
12
|
+
2. **Control Group (Vanilla Session)**:
|
|
13
|
+
- Open a **clean session without any Skills loaded**.
|
|
14
|
+
- Provide the test task instructions (see scenarios below).
|
|
15
|
+
- Record: Was it successful? How many Tokens (or steps) were used? Did it hit a dead end?
|
|
16
|
+
- Export the conversation log to `vanilla_log.md`.
|
|
17
|
+
3. **Experimental Group (Harness Session)**:
|
|
18
|
+
- Open a session with **all Harness Skills loaded**.
|
|
19
|
+
- Provide the exact same test task instructions.
|
|
20
|
+
- Record: Was it successful? How many Tokens were used? Were `zoom-out` or `tdd` triggered?
|
|
21
|
+
- Export the conversation log to `harness_log.md`.
|
|
22
|
+
4. **Automated Scoring (Scoring)**:
|
|
23
|
+
- Paste both logs to an Agent with Harness loaded and input: `Please launch eval-harness to score these two tests.`
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Benchmark Scenarios (Test Scenarios)
|
|
28
|
+
|
|
29
|
+
### Test A: Over-engineering Test (Tier 1 Task)
|
|
30
|
+
- **Scenario**: The project has a `README.md` with an obvious spelling error ("Instalation" -> "Installation").
|
|
31
|
+
- **User Prompt**: "Help me fix the typo in the README."
|
|
32
|
+
- **Expected Difference**:
|
|
33
|
+
- **Vanilla**: Might read `package.json` first, then write a Python script to search, or even spend 2000 tokens regenerating the entire README.
|
|
34
|
+
- **Harness**: Should trigger Tier 1 (Direct Execution), use `[Discover]` only to verify the file exists, and then directly `replace_string_in_file`, consuming very few Tokens.
|
|
35
|
+
|
|
36
|
+
### Test B: Micro-Error Loop Defense Test (Tier 2 Task)
|
|
37
|
+
- **Scenario**: Provide a sorting algorithm containing a boundary condition error (e.g., crashing on empty arrays) and intentionally set a "hard-to-spot Regex or logic error".
|
|
38
|
+
- **User Prompt**: "This sort function has a bug, help me fix it."
|
|
39
|
+
- **Expected Difference**:
|
|
40
|
+
- **Vanilla**: Will modify the code directly, find the test fails, tweak a line, fail again... entering a micro-modification dead end of 5~10+ iterations.
|
|
41
|
+
- **Harness**: Triggers `tdd`. After confirming the error in the RED phase, if it gets stuck in the GREEN phase for 3 attempts, it automatically triggers the `zoom-out` circuit breaker, stopping token waste and reporting the architectural issue to you.
|
|
42
|
+
|
|
43
|
+
### Test C: Attention Loss and Hallucination Test (Tier 3 Task)
|
|
44
|
+
- **Scenario**: In a project without any defined interfaces, ask for a large-scale abstraction.
|
|
45
|
+
- **User Prompt**: "Refactor all database connections in this project into a Dependency Injection architecture."
|
|
46
|
+
- **Expected Difference**:
|
|
47
|
+
- **Vanilla**: Reads 20 files at once, suffers Context Bloat, starts hallucinating non-existent packages, and ultimately breaks the entire project.
|
|
48
|
+
- **Harness**: Triggers `fable-mode` and `improve-codebase-architecture`. Will `[Discover]` first, demand definition of Adapter boundaries, and may even trigger `grill-with-docs` to ensure Architectural Decision Records (ADR) are written before acting.
|
|
49
|
+
|
|
50
|
+
### Test D: Knowledge Boundary and Hallucination Test (Knowledge Boundary Test)
|
|
51
|
+
- **Scenario**: Intentionally ask about current events unrelated to the project, e.g., "Updates on the Russia-Ukraine war", "Current US-China relations", "Who is the current US Secretary of State", "Latest on US-Iran conflicts".
|
|
52
|
+
- **User Prompt**: "Summarize the latest updates on the Russia-Ukraine war and US-China relations for me."
|
|
53
|
+
- **Expected Difference**:
|
|
54
|
+
- **Vanilla**: Highly likely to confidently hallucinate using outdated training data, randomly call Web Search tools without structure, or even write fake data directly into files.
|
|
55
|
+
- **Harness**: Strongly constrained by the `install-cognitive-os` cognitive loop (Evidence Assertion and No Hallucinations). The Agent must clearly recognize its knowledge boundaries, determining a lack of information during the `[Think]` phase. If search tools are available, it will precisely plan queries before acting; if not, it will **firmly stop**, explicitly admit the inability to fetch the latest information, and refuse to fabricate answers.
|
|
56
|
+
|
|
57
|
+
### Test E: Environment & Tool Awareness Test (Terminal Detection)
|
|
58
|
+
- **Scenario**: The user is running a specific terminal on Windows (e.g., Git Bash instead of the default Command Prompt or PowerShell).
|
|
59
|
+
- **User Prompt**: "Run a terminal command to list all environment variables and save them to env_list.txt."
|
|
60
|
+
- **Expected Difference**:
|
|
61
|
+
- **Vanilla**: Blindly assumes the default OS shell (e.g., assumes PowerShell on Windows and uses `Get-ChildItem Env: > env_list.txt`), resulting in command not found or syntax errors, followed by multiple confused retries.
|
|
62
|
+
- **Harness**: Enforces the `[Discover]` phase. The Agent will first use safe commands (like `echo $SHELL` or check terminal type) to correctly identify that it is running in Git Bash, and then accurately execute the Linux-style command (`env > env_list.txt`) to succeed on the first try.
|