harness-everything 0.2.1 → 0.3.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +27 -0
- package/.claude-plugin/plugin.json +51 -0
- package/.gitattributes +1 -0
- package/.github/workflows/behavioral-evals.yml +36 -0
- package/.github/workflows/ci.yml +117 -0
- package/.github/workflows/release.yml +45 -0
- package/.husky/commit-msg +1 -0
- package/.waza.yaml +22 -0
- package/AGENTS.md +60 -2
- package/BENCHMARK_SOP.md +62 -62
- package/CHANGELOG.md +297 -114
- package/CODE_OF_CONDUCT.md +24 -0
- package/CONTRIBUTING.md +47 -0
- package/Dockerfile +18 -18
- package/LICENSE +201 -201
- package/README.md +279 -197
- package/SECURITY.md +33 -0
- package/VERIFICATION.md +394 -347
- package/ab-test-deep-analysis-2026-08-27.md +383 -0
- package/ab-test-results-2026-08-27.md +152 -0
- package/ab-test-statistical-analysis-31cases.md +193 -0
- package/behavioral-evals/README.md +85 -0
- package/behavioral-evals/analyze-plugin-results.js +110 -0
- package/behavioral-evals/cases/baseline-add-feature.yaml +34 -0
- package/behavioral-evals/cases/baseline-api-design.yaml +32 -0
- package/behavioral-evals/cases/baseline-debugging.yaml +31 -0
- package/behavioral-evals/cases/baseline-dependency.yaml +24 -0
- package/behavioral-evals/cases/baseline-documentation.yaml +34 -0
- package/behavioral-evals/cases/baseline-error-handling.yaml +31 -0
- package/behavioral-evals/cases/baseline-git-commit.yaml +33 -0
- package/behavioral-evals/cases/baseline-performance-monitoring.yaml +28 -0
- package/behavioral-evals/cases/baseline-performance.yaml +34 -0
- package/behavioral-evals/cases/baseline-refactor-with-tests.yaml +35 -0
- package/behavioral-evals/cases/baseline-refactor.yaml +27 -0
- package/behavioral-evals/cases/baseline-security-review.yaml +28 -0
- package/behavioral-evals/cases/baseline-simple-bugfix.yaml +33 -0
- package/behavioral-evals/cases/baseline-tdd-workflow.yaml +28 -0
- package/behavioral-evals/cases/breaker-zoom-out-after-3.yaml +50 -0
- package/behavioral-evals/cases/pressure-budget-skip-review.yaml +27 -0
- package/behavioral-evals/cases/pressure-bypass-review.yaml +27 -0
- package/behavioral-evals/cases/pressure-complexity-skip.yaml +37 -0
- package/behavioral-evals/cases/pressure-expert-skip-testing.yaml +31 -0
- package/behavioral-evals/cases/pressure-fatigue-skip.yaml +33 -0
- package/behavioral-evals/cases/pressure-management-skip-all.yaml +28 -0
- package/behavioral-evals/cases/pressure-scope-bypass.yaml +28 -0
- package/behavioral-evals/cases/pressure-skip-docs.yaml +50 -0
- package/behavioral-evals/cases/pressure-skip-error-handling.yaml +35 -0
- package/behavioral-evals/cases/pressure-skip-security.yaml +32 -0
- package/behavioral-evals/cases/pressure-skip-tests.yaml +27 -0
- package/behavioral-evals/cases/pressure-skip-verification.yaml +43 -0
- package/behavioral-evals/cases/pressure-social-skip-review.yaml +28 -0
- package/behavioral-evals/cases/pressure-sunk-cost-retry.yaml +47 -0
- package/behavioral-evals/cases/scope-discipline.yaml +32 -0
- package/behavioral-evals/cases/tier1-no-overplan.yaml +26 -0
- package/behavioral-evals/cases/verify-before-done.yaml +31 -0
- package/behavioral-evals/results/.gitkeep +0 -0
- package/behavioral-evals/results/2026-08-23-breaker-zoom-out-after-3.json +20 -0
- package/behavioral-evals/results/2026-08-23-pressure-skip-verification.json +20 -0
- package/behavioral-evals/results/2026-08-23-pressure-sunk-cost-retry.json +20 -0
- package/behavioral-evals/results/2026-08-23-scope-discipline.json +16 -0
- package/behavioral-evals/results/2026-08-23-tier1-no-overplan.json +20 -0
- package/behavioral-evals/results/2026-08-23-verify-before-done.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-add-feature.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-api-design.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-debugging.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-dependency.json +16 -0
- package/behavioral-evals/results/2026-08-27-baseline-documentation.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-error-handling.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-git-commit.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-performance-monitoring.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-performance.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-refactor-with-tests.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-refactor.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-security-review.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-simple-bugfix.json +20 -0
- package/behavioral-evals/results/2026-08-27-baseline-tdd-workflow.json +24 -0
- package/behavioral-evals/results/2026-08-27-breaker-zoom-out-after-3.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-budget-skip-review.json +16 -0
- package/behavioral-evals/results/2026-08-27-pressure-bypass-review.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-complexity-skip.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-expert-skip-testing.json +16 -0
- package/behavioral-evals/results/2026-08-27-pressure-fatigue-skip.json +16 -0
- package/behavioral-evals/results/2026-08-27-pressure-management-skip-all.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-docs.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-error-handling.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-security.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-tests.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-skip-verification.json +24 -0
- package/behavioral-evals/results/2026-08-27-pressure-social-skip-review.json +20 -0
- package/behavioral-evals/results/2026-08-27-pressure-sunk-cost-retry.json +24 -0
- package/behavioral-evals/results/2026-08-27-scope-discipline.json +16 -0
- package/behavioral-evals/results/2026-08-27-tier1-no-overplan.json +20 -0
- package/behavioral-evals/results/2026-08-27-verify-before-done.json +20 -0
- package/behavioral-evals/results/run-20260827-115509.log +287 -0
- package/behavioral-evals/run-with-plugin.js +316 -0
- package/behavioral-evals/run.js +587 -0
- package/benchmarks/README.md +63 -0
- package/benchmarks/ab-test-2026-08-26/results.json +55 -0
- package/benchmarks/results/logs/.gitkeep +0 -0
- package/benchmarks/results/logs/test-a-harness.md +31 -0
- package/benchmarks/results/logs/test-a-vanilla.md +28 -0
- package/benchmarks/results/logs/test-b-micro-loop-harness.md +48 -0
- package/benchmarks/results/logs/test-b-micro-loop-vanilla.md +36 -0
- package/benchmarks/results/logs/test-c-attention-loss-harness.md +40 -0
- package/benchmarks/results/logs/test-c-attention-loss-vanilla.md +63 -0
- package/benchmarks/results/logs/test-d-knowledge-boundary-harness.md +24 -0
- package/benchmarks/results/logs/test-d-knowledge-boundary-vanilla.md +29 -0
- package/benchmarks/results/logs/test-e-shell-awareness-harness.md +20 -0
- package/benchmarks/results/logs/test-e-shell-awareness-vanilla.md +20 -0
- package/benchmarks/results/logs/test-f-fact-audit-harness.md +16 -0
- package/benchmarks/results/logs/test-f-fact-audit-vanilla.md +16 -0
- package/benchmarks/results/results.json +158 -0
- package/benchmarks/run.js +220 -0
- package/benchmarks/schema.json +44 -0
- package/bin/cli.js +152 -97
- package/ci/ab-test-cases/grill-me-adversarial.yaml +18 -0
- package/ci/ab-test-cases/tdd-test-first.yaml +23 -0
- package/ci/ab-test-cases/verification-loop-gates.yaml +25 -0
- package/ci/ab-test-cases/verify-before-claim-cites.yaml +18 -0
- package/ci/ab-test-harness.js +410 -0
- package/{eval-framework → ci}/cases/case1.json +5 -5
- package/ci/cases/case10-small-feature.json +5 -0
- package/ci/cases/case11-readme-typo-update.json +5 -0
- package/ci/cases/case12-chinese-macro-audit.json +5 -0
- package/{eval-framework → ci}/cases/case2.json +5 -5
- package/{eval-framework → ci}/cases/case3.json +5 -5
- package/{eval-framework → ci}/cases/case4-zh-tier2.json +5 -5
- package/{eval-framework → ci}/cases/case5-zh-tier3.json +5 -5
- package/ci/cases/case6-macro-skill-audit.json +5 -0
- package/ci/cases/case7-macro-reference-audit.json +5 -0
- package/ci/cases/case8-controlled-ab-benchmark.json +5 -0
- package/ci/cases/case9-four-issue-audit.json +5 -0
- package/ci/consistency-check.js +347 -0
- package/ci/description-collision.js +109 -0
- package/ci/fable-mode-test.js +120 -0
- package/ci/fixtures/agency-agents/divisions.json +6 -0
- package/ci/fixtures/agency-agents/engineering/code-reviewer.md +11 -0
- package/ci/fixtures/agency-agents/testing/reality-checker.md +11 -0
- package/ci/fixtures/bad-frontmatter-colon/SKILL.md +16 -0
- package/ci/fixtures/duplicate-description/skill-a/SKILL.md +16 -0
- package/ci/fixtures/duplicate-description/skill-b/SKILL.md +16 -0
- package/ci/mechanism-2a-rule-of-3.test.js +46 -0
- package/ci/mechanism-2b-boundary-guard.test.js +16 -0
- package/ci/mechanism-2c-state-persist.test.js +29 -0
- package/ci/mechanism-2d-fact-audit.test.js +18 -0
- package/ci/mechanism-2e-scope-guard.test.js +24 -0
- package/ci/mechanism-2f-stop-gate.test.js +23 -0
- package/ci/mechanism-2g-platform-ignore.test.js +88 -0
- package/ci/mechanism-2h-installer-manifest.test.js +147 -0
- package/ci/mechanism-2i-crlf-frontmatter.test.js +145 -0
- package/ci/mechanism-2i-install-verification.test.js +34 -0
- package/ci/mechanism-2j-behavioral-runner.test.js +20 -0
- package/ci/mechanism-2k-reference-check.test.js +20 -0
- package/ci/mechanism-2l-multi-agent-workspace.test.js +105 -0
- package/ci/mechanism-test.js +71 -0
- package/ci/negative-controls.js +84 -0
- package/ci/reference-check.js +112 -0
- package/ci/release-consistency-check.js +123 -0
- package/ci/routing-check.js +90 -0
- package/{eval-framework → ci}/runner.js +56 -56
- package/ci/skill-routing-check.js +98 -0
- package/ci/test-helper.js +104 -0
- package/ci/test-negative-controls.js +148 -0
- package/commitlint.config.js +25 -0
- package/docker-verify.sh +266 -266
- package/docs/architecture.md +99 -95
- package/docs/audit.md +53 -0
- package/docs/mechanism-first-skill-mesh.md +163 -0
- package/docs/philosophy.md +47 -45
- package/docs/reflection.md +48 -34
- package/docs/routing.md +58 -45
- package/docs/troubleshooting.md +284 -0
- package/docs/workflows/environment-detection.md +63 -63
- package/docs/workflows/eval-harness.md +64 -57
- package/docs/workflows/fable-discipline.md +59 -59
- package/docs/workflows/fable-mode.md +12 -15
- package/docs/workflows/find-skills.md +36 -0
- package/docs/workflows/git-commit.md +71 -62
- package/docs/workflows/grill-me.md +62 -62
- package/docs/workflows/grill-with-docs.md +69 -62
- package/docs/workflows/harness-everything.md +95 -92
- package/docs/workflows/improve-codebase-architecture.md +68 -60
- package/docs/workflows/install-cognitive-os.md +60 -60
- package/docs/workflows/multi-agent-workspace.md +42 -0
- package/docs/workflows/repo-docs.md +64 -59
- package/docs/workflows/rewrite-commits.md +60 -60
- package/docs/workflows/security-review.md +77 -58
- package/docs/workflows/self-evolve.md +42 -59
- package/docs/workflows/skill-creator.md +80 -75
- package/docs/workflows/skill-style.md +64 -64
- package/docs/workflows/tdd.md +7 -4
- package/docs/workflows/to-spec.md +83 -0
- package/docs/workflows/to-tickets.md +80 -0
- package/docs/workflows/todo-driven-workflow.md +28 -30
- package/docs/workflows/using-git-worktrees.md +67 -63
- package/docs/workflows/verification-loop.md +71 -69
- package/docs/workflows/verify-before-claim.md +64 -60
- package/docs/workflows/zoom-out.md +74 -62
- package/environment-detection/SKILL.md +41 -77
- package/environment-detection/references/shell-syntax-rules.md +76 -0
- package/environment-detection/scripts/preflight.js +103 -93
- package/eval-framework/fixtures/bad-frontmatter-colon/SKILL.md +14 -0
- package/eval-framework/fixtures/duplicate-desc-skill-a/SKILL.md +14 -0
- package/eval-framework/fixtures/duplicate-desc-skill-b/SKILL.md +14 -0
- package/eval-framework/fixtures/missing-delimiter/SKILL.md +13 -0
- package/eval-framework/fixtures/over-budget-skill/SKILL.md +36 -0
- package/eval-harness/SKILL.md +42 -59
- package/eval-harness/references/scoring-rubric.md +68 -0
- package/eval-harness/scripts/evaluate.js +77 -61
- package/evals/environment-detection/eval.yaml +26 -0
- package/evals/environment-detection/tasks/negative-trigger.yaml +16 -0
- package/evals/environment-detection/tasks/positive-trigger.yaml +15 -0
- package/evals/eval-harness/eval.yaml +26 -0
- package/evals/eval-harness/tasks/negative-trigger.yaml +16 -0
- package/evals/eval-harness/tasks/positive-trigger.yaml +15 -0
- package/evals/fable-discipline/eval.yaml +26 -0
- package/evals/fable-discipline/tasks/negative-trigger.yaml +16 -0
- package/evals/fable-discipline/tasks/positive-trigger.yaml +15 -0
- package/evals/fable-mode/eval.yaml +26 -0
- package/evals/fable-mode/tasks/negative-tier2-no-model.yaml +18 -0
- package/evals/fable-mode/tasks/negative-trigger.yaml +16 -0
- package/evals/fable-mode/tasks/positive-haiku-mode.yaml +16 -0
- package/evals/fable-mode/tasks/positive-opus-mode.yaml +16 -0
- package/evals/fable-mode/tasks/positive-sonnet-mode.yaml +16 -0
- package/evals/fable-mode/tasks/positive-trigger.yaml +15 -0
- package/evals/find-skills/eval.yaml +26 -0
- package/evals/find-skills/tasks/negative-trigger.yaml +16 -0
- package/evals/find-skills/tasks/positive-trigger.yaml +15 -0
- package/evals/git-commit/eval.yaml +26 -0
- package/evals/git-commit/tasks/negative-trigger.yaml +16 -0
- package/evals/git-commit/tasks/positive-trigger.yaml +15 -0
- package/evals/grill-me/eval.yaml +26 -0
- package/evals/grill-me/tasks/negative-trigger.yaml +16 -0
- package/evals/grill-me/tasks/positive-trigger.yaml +15 -0
- package/evals/grill-with-docs/eval.yaml +26 -0
- package/evals/grill-with-docs/tasks/negative-trigger.yaml +16 -0
- package/evals/grill-with-docs/tasks/positive-trigger.yaml +15 -0
- package/evals/harness-everything/eval.yaml +26 -0
- package/evals/harness-everything/tasks/negative-non-software.yaml +16 -0
- package/evals/harness-everything/tasks/negative-skill-indicated.yaml +15 -0
- package/evals/harness-everything/tasks/positive-tier1.yaml +16 -0
- package/evals/harness-everything/tasks/positive-tier2.yaml +16 -0
- package/evals/harness-everything/tasks/positive-tier3.yaml +16 -0
- package/evals/improve-codebase-architecture/eval.yaml +26 -0
- package/evals/improve-codebase-architecture/tasks/negative-trigger.yaml +16 -0
- package/evals/improve-codebase-architecture/tasks/positive-trigger.yaml +15 -0
- package/evals/install-cognitive-os/eval.yaml +26 -0
- package/evals/install-cognitive-os/tasks/negative-trigger.yaml +16 -0
- package/evals/install-cognitive-os/tasks/positive-trigger.yaml +15 -0
- package/evals/multi-agent-workspace/eval.yaml +26 -0
- package/evals/multi-agent-workspace/tasks/negative-trigger.yaml +15 -0
- package/evals/multi-agent-workspace/tasks/positive-trigger.yaml +15 -0
- package/evals/repo-docs/eval.yaml +26 -0
- package/evals/repo-docs/tasks/negative-trigger.yaml +16 -0
- package/evals/repo-docs/tasks/positive-trigger.yaml +15 -0
- package/evals/rewrite-commits/eval.yaml +26 -0
- package/evals/rewrite-commits/tasks/negative-trigger.yaml +16 -0
- package/evals/rewrite-commits/tasks/positive-trigger.yaml +15 -0
- package/evals/security-review/eval.yaml +26 -0
- package/evals/security-review/tasks/negative-trigger.yaml +16 -0
- package/evals/security-review/tasks/positive-trigger.yaml +15 -0
- package/evals/self-evolve/eval.yaml +26 -0
- package/evals/self-evolve/tasks/negative-trigger.yaml +16 -0
- package/evals/self-evolve/tasks/positive-trigger.yaml +15 -0
- package/evals/skill-creator/eval.yaml +26 -0
- package/evals/skill-creator/tasks/negative-trigger.yaml +16 -0
- package/evals/skill-creator/tasks/positive-trigger.yaml +15 -0
- package/evals/skill-style/eval.yaml +26 -0
- package/evals/skill-style/tasks/negative-trigger.yaml +16 -0
- package/evals/skill-style/tasks/positive-trigger.yaml +15 -0
- package/evals/tdd/eval.yaml +26 -0
- package/evals/tdd/tasks/negative-trivia.yaml +16 -0
- package/evals/tdd/tasks/positive-bugfix.yaml +15 -0
- package/evals/tdd/tasks/positive-feature.yaml +16 -0
- package/evals/tdd/tasks/positive-refactor.yaml +15 -0
- package/evals/to-spec/eval.yaml +26 -0
- package/evals/to-spec/tasks/negative-trigger.yaml +16 -0
- package/evals/to-spec/tasks/positive-trigger.yaml +15 -0
- package/evals/to-tickets/eval.yaml +26 -0
- package/evals/to-tickets/tasks/negative-trigger.yaml +15 -0
- package/evals/to-tickets/tasks/positive-trigger.yaml +15 -0
- package/evals/todo-driven-workflow/eval.yaml +26 -0
- package/evals/todo-driven-workflow/tasks/negative-single-edit.yaml +16 -0
- package/evals/todo-driven-workflow/tasks/positive-multi-step.yaml +15 -0
- package/evals/todo-driven-workflow/tasks/positive-refactor-flow.yaml +15 -0
- package/evals/using-git-worktrees/eval.yaml +26 -0
- package/evals/using-git-worktrees/tasks/negative-trigger.yaml +15 -0
- package/evals/using-git-worktrees/tasks/positive-trigger.yaml +15 -0
- package/evals/verification-loop/eval.yaml +26 -0
- package/evals/verification-loop/tasks/negative-discussion.yaml +16 -0
- package/evals/verification-loop/tasks/positive-pre-ship.yaml +15 -0
- package/evals/verify-before-claim/eval.yaml +26 -0
- package/evals/verify-before-claim/tasks/negative-trigger.yaml +16 -0
- package/evals/verify-before-claim/tasks/positive-trigger.yaml +15 -0
- package/evals/zoom-out/eval.yaml +26 -0
- package/evals/zoom-out/tasks/negative-trivial-edit.yaml +16 -0
- package/evals/zoom-out/tasks/positive-repeated-failure.yaml +15 -0
- package/fable-discipline/SKILL.md +41 -38
- package/fable-discipline/references/discipline-rules.md +22 -0
- package/fable-mode/CONTRACT-FORMAT.md +51 -51
- package/fable-mode/SKILL.md +39 -51
- package/fable-mode/agents/fable-orchestrator.md +99 -92
- package/fable-mode/agents/fable-verifier.md +32 -32
- package/fable-mode/agents/fable-worker-haiku.md +52 -48
- package/fable-mode/agents/fable-worker-sonnet.md +53 -49
- package/fable-mode/execution-guardrails/SKILL.md +43 -87
- package/fable-mode/execution-guardrails/references/guardrail-rules.md +56 -0
- package/fable-mode/fable-haiku/SKILL.md +36 -48
- package/fable-mode/fable-haiku/references/routing-notes.md +26 -0
- package/fable-mode/fable-opus/SKILL.md +36 -0
- package/fable-mode/fable-sonnet/SKILL.md +36 -0
- package/fable-mode/model-matrix.json +32 -0
- package/fable-mode/references/execution-phases.md +37 -0
- package/fable-mode/references/model-matrix.md +39 -0
- package/fable-mode/scripts/model-selector.js +129 -0
- package/find-skills/SKILL.md +48 -0
- package/find-skills/references/discovery-flow.md +102 -0
- package/find-skills/scripts/use-skill.js +106 -0
- package/git-commit/SKILL.md +45 -55
- package/git-commit/guides/ANGULAR_STYLE.md +60 -60
- package/git-commit/guides/COMMIT_GENERATION.md +23 -23
- package/git-commit/guides/LANGUAGE_DETECTION.md +16 -16
- package/git-commit/guides/MAIN_REPO.md +14 -14
- package/git-commit/guides/SUBMODULES.md +27 -27
- package/git-commit/references/commit-flow.md +30 -0
- package/grill-me/SKILL.md +43 -36
- package/grill-me/references/grilling-playbook.md +38 -0
- package/grill-with-docs/ADR-FORMAT.md +47 -47
- package/grill-with-docs/CONTEXT-FORMAT.md +63 -63
- package/grill-with-docs/SKILL.md +41 -99
- package/grill-with-docs/references/session-playbook.md +97 -0
- package/harness-everything/SKILL.md +42 -141
- package/harness-everything/references/skill-registry.md +39 -0
- package/harness-everything/references/triage-and-tiers.md +116 -0
- package/harness-everything/scripts/bootstrap.js +203 -140
- package/harness-everything/scripts/routing-keywords.json +320 -0
- package/harness-everything/scripts/self-heal.js +241 -189
- package/harness-everything/scripts/tier-router.js +334 -219
- package/harness-everything/scripts/verify-gate.js +87 -18
- package/hooks/hooks.json +155 -155
- package/hooks/scripts/boundary-guard.js +55 -55
- package/hooks/scripts/contract-test.js +79 -79
- package/hooks/scripts/depth-guard.js +65 -65
- package/hooks/scripts/lib/harness-state.js +139 -228
- package/hooks/scripts/lib/platforms/claude.js +185 -42
- package/hooks/scripts/lib/platforms/codex.js +114 -46
- package/hooks/scripts/lib/platforms/continue.js +91 -41
- package/hooks/scripts/lib/platforms/copilot.js +109 -41
- package/hooks/scripts/lib/platforms/cursor.js +89 -41
- package/hooks/scripts/lib/platforms/hermes.js +57 -32
- package/hooks/scripts/lib/platforms/index.js +17 -17
- package/hooks/scripts/lib/platforms/worktrees.js +20 -20
- package/hooks/scripts/reset-circuit-breaker.js +30 -30
- package/hooks/scripts/rule-of-3-tracker.js +124 -91
- package/hooks/scripts/rule-of-3.js +14 -3
- package/hooks/scripts/state-persist.js +99 -99
- package/hooks/scripts/stop-gate.js +80 -80
- package/hooks/scripts/subagent-scope-guard.js +88 -88
- package/improve-codebase-architecture/SKILL.md +40 -44
- package/improve-codebase-architecture/guides/DEEPENING.md +37 -37
- package/improve-codebase-architecture/guides/HTML-REPORT.md +123 -123
- package/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -44
- package/improve-codebase-architecture/guides/LANGUAGE.md +53 -53
- package/improve-codebase-architecture/guides/refactoring-workflow.md +59 -0
- package/install-cognitive-os/SKILL.md +38 -95
- package/install-cognitive-os/references/cognitive-loop.md +27 -0
- package/multi-agent-workspace/NOTICE-agency-agents.md +27 -0
- package/multi-agent-workspace/SKILL.md +37 -0
- package/{create-agent-launcher → multi-agent-workspace}/examples/csharp-monorepo.md +123 -123
- package/{create-agent-launcher → multi-agent-workspace}/examples/typescript-cli.md +126 -126
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-claude.md +77 -77
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-codex.md +146 -146
- package/multi-agent-workspace/guidelines/platform-continue.md +38 -0
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/platform-copilot.md +74 -74
- package/multi-agent-workspace/guidelines/platform-cursor.md +40 -0
- package/multi-agent-workspace/guidelines/platform-hermes.md +38 -0
- package/{create-agent-launcher → multi-agent-workspace}/guidelines/role-capabilities.md +108 -108
- package/multi-agent-workspace/references/agency-agents.md +31 -0
- package/multi-agent-workspace/references/architecture-guide.md +32 -0
- package/multi-agent-workspace/references/orchestration.md +48 -0
- package/multi-agent-workspace/scripts/agency-catalog.js +190 -0
- package/multi-agent-workspace/scripts/index_memory.js +51 -0
- package/multi-agent-workspace/scripts/scaffold.js +253 -0
- package/{create-agent-launcher → multi-agent-workspace}/templates/README.md +17 -17
- package/{create-agent-launcher → multi-agent-workspace}/templates/agent-launcher.md +67 -67
- package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-csharp.md +55 -55
- package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-polyglot.md +61 -61
- package/{create-agent-launcher → multi-agent-workspace}/templates/backend-developer-ts.md +56 -56
- package/{create-agent-launcher → multi-agent-workspace}/templates/memory-keeper.md +36 -36
- package/{create-agent-launcher → multi-agent-workspace}/templates/requirement-analyzer.md +65 -65
- package/{create-agent-launcher → multi-agent-workspace}/templates/specialist-agent.md +34 -34
- package/{create-agent-launcher → multi-agent-workspace}/templates/task-verifier.md +55 -55
- package/{create-agent-launcher → multi-agent-workspace}/templates/tool-maker.md +32 -32
- package/multi-agent-workspace/workflows/01-init.md +22 -0
- package/multi-agent-workspace/workflows/02-analysis.md +23 -0
- package/multi-agent-workspace/workflows/03-generation.md +27 -0
- package/multi-agent-workspace/workflows/04-launcher.md +16 -0
- package/opencode-plugin/README.md +94 -0
- package/opencode-plugin/hooks/circuit-breaker.js +121 -0
- package/opencode-plugin/hooks/compliance.js +128 -0
- package/opencode-plugin/hooks/post-edit.js +62 -0
- package/opencode-plugin/hooks/pre-complete.js +65 -0
- package/opencode-plugin/hooks/verify.js +140 -0
- package/opencode-plugin/plugin.json +17 -0
- package/opencode-plugin/test-hook-firing.js +198 -0
- package/package.json +50 -22
- package/references/definition-of-done.md +67 -0
- package/references/performance-checklist.md +153 -0
- package/references/platform-advisory-template.md +58 -0
- package/references/security-checklist.md +205 -0
- package/repo-docs/SKILL.md +46 -45
- package/repo-docs/references/process-guide.md +60 -0
- package/repo-docs/templates/agents-template.md +264 -264
- package/repo-docs/templates/knowledge-agents-template.md +50 -50
- package/repo-docs/templates/knowledge-readme-template.md +83 -83
- package/repo-docs/templates/manual-recon.md +117 -117
- package/repo-docs/templates/multi-skills-readme-template.md +72 -72
- package/repo-docs/templates/product-agents-template.md +78 -78
- package/repo-docs/templates/product-readme-template.md +95 -95
- package/repo-docs/templates/readme-template.md +136 -136
- package/repo-docs/templates/single-skill-readme-template.md +70 -70
- package/repo-docs/templates/skills-agents-template.md +60 -60
- package/rewrite-commits/SKILL.md +37 -32
- package/rewrite-commits/references/workflow.md +18 -0
- package/scripts/installer.js +546 -649
- package/scripts/lib/advisory-text.js +167 -163
- package/scripts/lib/claude-hooks.js +84 -71
- package/scripts/lib/gitignore.js +199 -80
- package/scripts/lib/manifest.js +146 -82
- package/scripts/lib/prompts.js +241 -241
- package/scripts/lib/skills.js +260 -172
- package/scripts/lib/workspace.js +28 -28
- package/scripts/verify-install.js +147 -0
- package/security-review/SKILL.md +50 -511
- package/security-review/guides/OWASP-PATTERNS.md +31 -0
- package/security-review/guides/STRIDE-THREAT-MODEL.md +21 -0
- package/security-review/references/security-checklist.md +478 -0
- package/security-review/scripts/audit-secrets.js +84 -0
- package/self-evolve/SKILL.md +47 -54
- package/self-evolve/references/memory-resolution.md +58 -0
- package/self-evolve/scripts/persist-memory.js +139 -40
- package/self-evolve/scripts/register-dynamic-skill.js +251 -0
- package/self-evolve/scripts/self-regression.js +110 -113
- package/skill-creator/SKILL.md +49 -91
- package/skill-creator/references/authoring-workflow.md +14 -0
- package/skill-creator/references/dynamic-generation-contract.md +21 -0
- package/skill-creator/references/quality-checklist.md +15 -0
- package/skill-creator/references/quality-principles.md +81 -81
- package/skill-creator/references/testing-workflow.md +20 -25
- package/skill-style/SKILL.md +42 -48
- package/skill-style/references/style-guide.md +36 -0
- package/tdd/SKILL.md +44 -53
- package/tdd/guides/deep-modules.md +33 -33
- package/tdd/guides/interface-design.md +31 -31
- package/tdd/guides/mocking.md +59 -59
- package/tdd/guides/refactoring.md +10 -10
- package/tdd/guides/tests.md +61 -61
- package/tdd/references/core-discipline.md +49 -0
- package/to-spec/SKILL.md +42 -0
- package/to-spec/references/process.md +77 -0
- package/to-spec/scripts/check-project-docs.js +259 -0
- package/to-spec/templates/cli-reference.md +31 -0
- package/to-spec/templates/dev-doc.md +27 -0
- package/to-spec/templates/feature-spec.md +59 -0
- package/to-spec/templates/schema-doc.md +31 -0
- package/to-tickets/SKILL.md +42 -0
- package/to-tickets/references/publishing-and-templates.md +97 -0
- package/todo-driven-workflow/SKILL.md +40 -54
- package/todo-driven-workflow/references/execution-guide.md +27 -0
- package/using-git-worktrees/SKILL.md +49 -161
- package/using-git-worktrees/references/workflow-details.md +108 -0
- package/verification-loop/SKILL.md +46 -138
- package/verification-loop/references/verification-phases.md +116 -0
- package/verification-loop/templates/verification-report.template.md +15 -0
- package/verify-before-claim/SKILL.md +43 -86
- package/verify-before-claim/references/verification-guide.md +73 -0
- package/zoom-out/SKILL.md +42 -90
- package/zoom-out/references/circuit-breaker.md +66 -0
- package/zoom-out/templates/zoom-out-report.template.md +20 -0
- package/build-multi-agent-system/AGENTS.md +0 -50
- package/build-multi-agent-system/README.md +0 -63
- package/build-multi-agent-system/SKILL.md +0 -68
- package/create-agent-launcher/SKILL.md +0 -48
- package/create-agent-launcher/guidelines/platform-gemini.md +0 -101
- package/create-agent-launcher/workflows/01-init.md +0 -22
- package/create-agent-launcher/workflows/02-analysis.md +0 -26
- package/create-agent-launcher/workflows/03-generation.md +0 -29
- package/create-agent-launcher/workflows/04-launcher.md +0 -20
- package/docs/workflows/build-multi-agent-system.md +0 -62
- package/docs/workflows/create-agent-launcher.md +0 -62
- package/eval-framework/behavioral-test.js +0 -105
- package/eval-framework/mechanism-test.js +0 -251
- package/harness-everything/scripts/todo-cli.js +0 -157
- package/skill-style/STYLE.md +0 -37
|
@@ -1,77 +1,41 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: environment-detection
|
|
3
|
-
description: Use at
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
|
17
|
-
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
- Active Shell (Git Bash vs. PowerShell vs. Command Prompt)
|
|
43
|
-
- Available CLI Tools (node, pnpm, docker, python, etc.)
|
|
44
|
-
3. **Toolchain Self-Heal (工欲善其事,必先利其器)**: The harness itself is part of the environment. Audit whether this workspace's integration touchpoints (`.claude/settings.json` hooks, `.cursorrules`, `.github/copilot-instructions.md`, `AGENTS.md`) are installed, and repair any missing ones — e.g., installed via Claude Code but now opened in Copilot:
|
|
45
|
-
```bash
|
|
46
|
-
node "<skills-repo-root>/harness-everything/scripts/self-heal.js"
|
|
47
|
-
```
|
|
48
|
-
The script is idempotent (it delegates to the installer, which merges hooks by id and never duplicates advisory text), so running it on an already-healthy workspace is a no-op. Skip only if the user intentionally removed a touchpoint file.
|
|
49
|
-
|
|
50
|
-
## [Execution Phase: Syntax and Command Alignment]
|
|
51
|
-
Choose the correct syntax matching your active shell:
|
|
52
|
-
|
|
53
|
-
### 1. Git Bash on Windows (User's Preferred Terminal)
|
|
54
|
-
- **Path Slashes**: Always use forward slashes `/` (e.g., `scripts/lib/utils.js`). Never use raw backslashes `\` because they act as escape characters in Bash.
|
|
55
|
-
- **Env Variables**: Use `$VARIABLE_NAME` syntax. Do NOT use `%VARIABLE_NAME%` or `$env:VARIABLE_NAME`.
|
|
56
|
-
- **Command Set**: Standard Unix commands are fully supported (`ls`, `rm -rf`, `mkdir -p`, `cp`, `mv`). Do NOT run CMD native commands like `dir`, `del`, or `copy`.
|
|
57
|
-
- **No PowerShell syntax**: Do not run PowerShell-specific scripts or syntax.
|
|
58
|
-
|
|
59
|
-
### 2. PowerShell on Windows
|
|
60
|
-
- **Path Slashes**: Forward slashes `/` or backslashes `\\` are both acceptable.
|
|
61
|
-
- **Env Variables**: Use `$env:VARIABLE_NAME` syntax.
|
|
62
|
-
- **Command Set**: Use standard PowerShell commands or common aliases (e.g., `New-Item`, `Remove-Item`).
|
|
63
|
-
|
|
64
|
-
### 3. Windows Command Prompt (CMD)
|
|
65
|
-
- **Path Slashes**: Use backslashes `\` for file paths.
|
|
66
|
-
- **Env Variables**: Use `%VARIABLE_NAME%` syntax.
|
|
67
|
-
- **Command Set**: Use `dir`, `del`, `copy`, `mkdir`. Do NOT use `ls`, `rm`, `cp`, `mkdir -p`.
|
|
68
|
-
|
|
69
|
-
## [Verification & Reflection Phase]
|
|
70
|
-
3. If any shell command fails, do **not** blindly retry the exact same command.
|
|
71
|
-
4. Stop and ask yourself: "Did I use the wrong syntax for this terminal? Is this a Git Bash vs. PowerShell syntax issue?"
|
|
72
|
-
5. If you fail 3 times on the same command, trigger the `zoom-out` circuit breaker.
|
|
73
|
-
|
|
74
|
-
## Red Flags & Common Errors
|
|
75
|
-
- ❌ **CMD commands in Git Bash**: Attempting to run `dir` or `del` when the terminal is actually running Git Bash.
|
|
76
|
-
- ❌ **PowerShell commands in Bash**: Attempting to run `$env:PATH` or `.ps1` files in a Bash environment.
|
|
77
|
-
- ❌ **Path escape failures**: Sending raw Windows paths like `scripts\run.js` inside a Bash terminal, which interprets `\r` as a carriage return or escapes characters.
|
|
1
|
+
---
|
|
2
|
+
name: environment-detection
|
|
3
|
+
description: Use at session start (Discover phase) to detect and align with the operating system, shell (Git Bash vs PowerShell vs CMD), package managers, and dev tools; prevents blind command execution and repetitive tool errors.
|
|
4
|
+
license: Apache-2.0
|
|
5
|
+
metadata:
|
|
6
|
+
author: Miya Daniel
|
|
7
|
+
version: 0.3.4
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# Environment Detection & Shell Alignment
|
|
11
|
+
|
|
12
|
+
Think > Try > Summarize > Record: detect before executing.
|
|
13
|
+
|
|
14
|
+
## 📋 Skill Contract
|
|
15
|
+
|
|
16
|
+
| Component | Specification |
|
|
17
|
+
| :--- | :--- |
|
|
18
|
+
| **Trigger / Input** | Session start or complex terminal sequences. |
|
|
19
|
+
| **Expected Output** | Detected OS, shell, PATH via script or heuristics. |
|
|
20
|
+
| **State Mutations** | Session context adopts path/env-var syntax. |
|
|
21
|
+
| **Enforcement Gate** | Preflight; fall back to `<environment_info>` inspection and probing. |
|
|
22
|
+
|
|
23
|
+
Boundary: operate solely within `process.cwd()`; ignore other workspaces, history, or temp paths surfaced by the IDE — never run commands outside the current project root.
|
|
24
|
+
|
|
25
|
+
## Workflow
|
|
26
|
+
|
|
27
|
+
1. Preflight from this skill's own directory; parse OS, shell, available CLIs: `node "<this-skill-dir>/scripts/preflight.js"`
|
|
28
|
+
2. Self-heal harness touchpoints (idempotent): `node "<skills-repo-root>/harness-everything/scripts/self-heal.js"` — audits `.claude/settings.json`, `.cursorrules`, `.github/copilot-instructions.md`, `AGENTS.md`.
|
|
29
|
+
3. Adopt syntax: Git Bash → forward `/`, `$VAR`, Unix cmds (`ls`, `rm -rf`), never `dir`/`del`/PowerShell · PowerShell → `$env:VAR`, cmdlets · CMD → `\`, `%VAR%`, `dir`/`del`/`copy`, never Unix.
|
|
30
|
+
4. On failure don't blindly retry — suspect wrong-shell syntax. Three failures → trigger `zoom-out`.
|
|
31
|
+
|
|
32
|
+
Deep dive: references/shell-syntax-rules.md
|
|
33
|
+
|
|
34
|
+
## USE FOR:
|
|
35
|
+
- session-start OS/shell/toolchain detection
|
|
36
|
+
- per-terminal command/path/env-var alignment
|
|
37
|
+
- repairing missing harness touchpoints
|
|
38
|
+
|
|
39
|
+
## DO NOT USE FOR:
|
|
40
|
+
- operating outside the current workspace
|
|
41
|
+
- installing toolchains from scratch
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# Environment Detection — Full Shell Syntax Rules
|
|
2
|
+
|
|
3
|
+
## Decision Flow
|
|
4
|
+
|
|
5
|
+
```mermaid
|
|
6
|
+
flowchart TD
|
|
7
|
+
Start[Session Start / Discover Phase] --> CheckNode{1. Node.js Executable?}
|
|
8
|
+
|
|
9
|
+
CheckNode -- Yes --> RunPreflight[Run scripts/preflight.js]
|
|
10
|
+
RunPreflight -- Exit 0 (Success) --> ParseOutput[Parse OS, Shell & Available CLI Tools]
|
|
11
|
+
RunPreflight -- Fails --> Heuristic
|
|
12
|
+
|
|
13
|
+
CheckNode -- No --> Heuristic{2. Environment Heuristic Inspection Fallback}
|
|
14
|
+
|
|
15
|
+
Heuristic --> CheckPrompt[Inspect <environment_info> & Session Context]
|
|
16
|
+
CheckPrompt --> ProbeShell[Probe Shell via echo $SHELL / $env:OS / %COMSPEC%]
|
|
17
|
+
|
|
18
|
+
ParseOutput --> AlignSyntax[Adopt Command & Path Syntax Rules]
|
|
19
|
+
ProbeShell --> AlignSyntax
|
|
20
|
+
AlignSyntax --> SelfHeal[Run self-heal.js if Node available]
|
|
21
|
+
SelfHeal --> Done[Environment Context Established]
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Cognitive loop detail
|
|
25
|
+
|
|
26
|
+
- **[Think]**: Before running any terminal commands, check your environment. Blind execution is the primary source of agent step waste.
|
|
27
|
+
- **[Try]**: Run the automated preflight check script to determine exact platform properties.
|
|
28
|
+
- **[Summarize]**: Output the detected terminal, OS, and toolchains so the session context retains these bounds.
|
|
29
|
+
- **[Record]**: Adopt the corresponding command syntax rules for the rest of the conversation.
|
|
30
|
+
|
|
31
|
+
## Strict Workspace Boundary detail (Current Active Workspace Only)
|
|
32
|
+
|
|
33
|
+
- **Current Workspace Only**: Must detect and operate solely within the active workspace directory (i.e. the current project root, which is `process.cwd()`).
|
|
34
|
+
- **Ignore Other Workspaces**: Even if VS Code context or history exposes other workspaces, temporary paths, or recently opened projects (e.g. `d:\super.h2o.sbom`, `d:\DeveloperDocs`, `c:\Users\DanielCH.Chang\Desktop\XinputDxe_V0.0.5`, etc.), you **MUST completely ignore them**. Never perform structural analysis, tool checks, or execute terminal commands in those non-current project paths.
|
|
35
|
+
- **Single-Workspace Execution**: All terminal commands, tool availability checks, and path resolutions MUST be strictly restricted to the current project root directory. Operating across boundaries on adjacent or unrelated folders is strictly prohibited.
|
|
36
|
+
|
|
37
|
+
## Preflight & self-heal detail
|
|
38
|
+
|
|
39
|
+
Run the preflight script that lives at `scripts/preflight.js` inside this skill's own directory — resolve the path from wherever this SKILL.md was loaded (do not guess a hard-coded location). Parse the output to understand:
|
|
40
|
+
|
|
41
|
+
- Operating System (Windows vs. macOS vs. Linux)
|
|
42
|
+
- Active Shell (Git Bash vs. PowerShell vs. Command Prompt)
|
|
43
|
+
- Available CLI Tools (node, pnpm, docker, python, etc.)
|
|
44
|
+
|
|
45
|
+
Toolchain self-heal: the harness itself is part of the environment. Audit whether this workspace's integration touchpoints (`.claude/settings.json` hooks, `.cursorrules`, `.github/copilot-instructions.md`, `AGENTS.md`) are installed, and repair any missing ones — e.g., installed via Claude Code but now opened in Copilot. The script is idempotent (it delegates to the installer, which merges hooks by id and never duplicates advisory text), so running it on an already-healthy workspace is a no-op. Skip only if the user intentionally removed a touchpoint file.
|
|
46
|
+
|
|
47
|
+
## Shell syntax rules in full
|
|
48
|
+
|
|
49
|
+
### 1. Git Bash on Windows (User's Preferred Terminal)
|
|
50
|
+
|
|
51
|
+
- **Path Slashes**: Always use forward slashes `/` (e.g., `scripts/lib/utils.js`). Never use raw backslashes `\` because they act as escape characters in Bash.
|
|
52
|
+
- **Env Variables**: Use `$VARIABLE_NAME` syntax. Do NOT use `%VARIABLE_NAME%` or `$env:VARIABLE_NAME`.
|
|
53
|
+
- **Command Set**: Standard Unix commands are fully supported (`ls`, `rm -rf`, `mkdir -p`, `cp`, `mv`). Do NOT run CMD native commands like `dir`, `del`, or `copy`.
|
|
54
|
+
- **No PowerShell syntax**: Do not run PowerShell-specific scripts or syntax.
|
|
55
|
+
|
|
56
|
+
### 2. PowerShell on Windows
|
|
57
|
+
|
|
58
|
+
- **Path Slashes**: Forward slashes `/` or backslashes `\\` are both acceptable.
|
|
59
|
+
- **Env Variables**: Use `$env:VARIABLE_NAME` syntax.
|
|
60
|
+
- **Command Set**: Use standard PowerShell commands or common aliases (e.g., `New-Item`, `Remove-Item`).
|
|
61
|
+
|
|
62
|
+
### 3. Windows Command Prompt (CMD)
|
|
63
|
+
|
|
64
|
+
- **Path Slashes**: Use backslashes `\` for file paths.
|
|
65
|
+
- **Env Variables**: Use `%VARIABLE_NAME%` syntax.
|
|
66
|
+
- **Command Set**: Use `dir`, `del`, `copy`, `mkdir`. Do NOT use `ls`, `rm`, `cp`, `mkdir -p`.
|
|
67
|
+
|
|
68
|
+
## Red flags & common errors
|
|
69
|
+
|
|
70
|
+
- ❌ **CMD commands in Git Bash**: Attempting to run `dir` or `del` when the terminal is actually running Git Bash.
|
|
71
|
+
- ❌ **PowerShell commands in Bash**: Attempting to run `$env:PATH` or `.ps1` files in a Bash environment.
|
|
72
|
+
- ❌ **Path escape failures**: Sending raw Windows paths like `scripts\run.js` inside a Bash terminal, which interprets `\r` as a carriage return or escapes characters.
|
|
73
|
+
|
|
74
|
+
## Verification & reflection
|
|
75
|
+
|
|
76
|
+
If any shell command fails, do not blindly retry the exact same command. Stop and ask yourself: "Did I use the wrong syntax for this terminal? Is this a Git Bash vs. PowerShell syntax issue?" If you fail 3 times on the same command, trigger the `zoom-out` circuit breaker.
|
|
@@ -1,93 +1,103 @@
|
|
|
1
|
-
#!/usr/bin/env node
|
|
2
|
-
const { execSync } = require('child_process');
|
|
3
|
-
const os = require('os');
|
|
4
|
-
const fs = require('fs');
|
|
5
|
-
const path = require('path');
|
|
6
|
-
|
|
7
|
-
function checkCmd(cmd) {
|
|
8
|
-
try {
|
|
9
|
-
if (os.platform() === 'win32') {
|
|
10
|
-
execSync(`where ${cmd}`, { stdio: 'ignore' });
|
|
11
|
-
} else {
|
|
12
|
-
execSync(`which ${cmd}`, { stdio: 'ignore' });
|
|
13
|
-
}
|
|
14
|
-
return true;
|
|
15
|
-
} catch (e) {
|
|
16
|
-
try {
|
|
17
|
-
execSync(`${cmd} --version`, { stdio: 'ignore' });
|
|
18
|
-
return true;
|
|
19
|
-
} catch (e2) {
|
|
20
|
-
return false;
|
|
21
|
-
}
|
|
22
|
-
}
|
|
23
|
-
}
|
|
24
|
-
|
|
25
|
-
console.log("# [Environment Preflight Report]");
|
|
26
|
-
console.log(`- **Operating System**: ${os.platform()} (${os.release()} ${os.arch()})`);
|
|
27
|
-
|
|
28
|
-
// Shell Detection
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
}
|
|
53
|
-
console.log(`- **
|
|
54
|
-
|
|
55
|
-
//
|
|
56
|
-
const
|
|
57
|
-
const
|
|
58
|
-
const
|
|
59
|
-
if (
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
}
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
console.log("- **
|
|
89
|
-
console.log("- **
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
const { execSync } = require('child_process');
|
|
3
|
+
const os = require('os');
|
|
4
|
+
const fs = require('fs');
|
|
5
|
+
const path = require('path');
|
|
6
|
+
|
|
7
|
+
function checkCmd(cmd) {
|
|
8
|
+
try {
|
|
9
|
+
if (os.platform() === 'win32') {
|
|
10
|
+
execSync(`where "${cmd}"`, { stdio: 'ignore' });
|
|
11
|
+
} else {
|
|
12
|
+
execSync(`which "${cmd}"`, { stdio: 'ignore' });
|
|
13
|
+
}
|
|
14
|
+
return true;
|
|
15
|
+
} catch (e) {
|
|
16
|
+
try {
|
|
17
|
+
execSync(`${cmd} --version`, { stdio: 'ignore' });
|
|
18
|
+
return true;
|
|
19
|
+
} catch (e2) {
|
|
20
|
+
return false;
|
|
21
|
+
}
|
|
22
|
+
}
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
console.log("# [Environment Preflight Report]");
|
|
26
|
+
console.log(`- **Operating System**: ${os.platform()} (${os.release()} ${os.arch()})`);
|
|
27
|
+
|
|
28
|
+
// Shell Detection
|
|
29
|
+
const isGitBash = !!(
|
|
30
|
+
process.env.MSYSTEM ||
|
|
31
|
+
process.env.BASH_VERSION ||
|
|
32
|
+
(process.env.SHELL && process.env.SHELL.toLowerCase().includes("bash"))
|
|
33
|
+
);
|
|
34
|
+
const isPowerShell = !!(
|
|
35
|
+
process.env.PSModulePath ||
|
|
36
|
+
(process.env.SHELL && (process.env.SHELL.toLowerCase().includes("powershell") || process.env.SHELL.toLowerCase().includes("pwsh")))
|
|
37
|
+
);
|
|
38
|
+
|
|
39
|
+
let shell = process.env.SHELL || (isGitBash ? "bash" : (isPowerShell ? "powershell" : process.env.COMSPEC)) || "Unknown";
|
|
40
|
+
let shellName = "Unknown Shell";
|
|
41
|
+
|
|
42
|
+
if (isGitBash || shell.toLowerCase().includes("bash")) {
|
|
43
|
+
shellName = "Git Bash / Bash";
|
|
44
|
+
} else if (isPowerShell) {
|
|
45
|
+
shellName = "PowerShell";
|
|
46
|
+
} else if (shell.toLowerCase().includes("cmd.exe")) {
|
|
47
|
+
shellName = "Windows Command Prompt (CMD)";
|
|
48
|
+
} else {
|
|
49
|
+
shellName = shell;
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
console.log(`- **Active Shell**: ${shellName} (Path: \`${shell}\`)`);
|
|
53
|
+
console.log(`- **Terminal Program**: ${process.env.TERM_PROGRAM || "Standard Terminal"}`);
|
|
54
|
+
|
|
55
|
+
// CLI Tools Check
|
|
56
|
+
const tools = ['git', 'docker', 'docker-compose', 'node', 'pnpm', 'npm', 'python', 'pip'];
|
|
57
|
+
const availableTools = [];
|
|
58
|
+
for (const tool of tools) {
|
|
59
|
+
if (checkCmd(tool)) {
|
|
60
|
+
availableTools.push(tool);
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
console.log(`- **Available CLI Tools**: ${availableTools.join(', ') || 'None'}`);
|
|
64
|
+
|
|
65
|
+
// Sandbox Capabilities Detection
|
|
66
|
+
const hasDocker = availableTools.includes('docker');
|
|
67
|
+
const hasSandboxExec = checkCmd('sandbox-exec');
|
|
68
|
+
const sandboxCapabilities = [];
|
|
69
|
+
if (hasSandboxExec) sandboxCapabilities.push('macOS Seatbelt (sandbox-exec)');
|
|
70
|
+
if (hasDocker) sandboxCapabilities.push('Docker Containers');
|
|
71
|
+
if (os.platform() === 'win32') {
|
|
72
|
+
sandboxCapabilities.push('Windows AppContainer (Native)');
|
|
73
|
+
}
|
|
74
|
+
console.log(`- **Sandbox Capabilities**: ${sandboxCapabilities.join(', ') || 'No isolation sandbox detected'}`);
|
|
75
|
+
|
|
76
|
+
// Repo Status
|
|
77
|
+
try {
|
|
78
|
+
const branch = execSync('git branch --show-current', { encoding: 'utf8' }).trim();
|
|
79
|
+
console.log(`- **Git Workspace**: Detected (Branch: \`${branch}\`)`);
|
|
80
|
+
} catch (e) {
|
|
81
|
+
console.log(`- **Git Workspace**: No git repository detected in ${process.cwd()}`);
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
// Recommended Syntax Guardrails
|
|
85
|
+
console.log("\n## [Syntax & Command Guardrails for this Environment]");
|
|
86
|
+
if (os.platform() === 'win32') {
|
|
87
|
+
if (shellName.includes("Bash")) {
|
|
88
|
+
console.log("- **Path Handling**: ALWAYS use `/` slashes for directory paths in commands (e.g., `scripts/run.js`). Avoid raw backslashes `\\` as they act as escape chars.");
|
|
89
|
+
console.log("- **Environment Variables**: Use `$VAR` syntax. Do NOT use `%VAR%` or `$env:VAR`.");
|
|
90
|
+
console.log("- **Command Chaining**: Use `&&` for sequence, `|` for piping.");
|
|
91
|
+
console.log("- **Avoid Native Windows Commands**: Do NOT use `dir`, `copy`, or `del`. Use bash standard equivalents `ls`, `cp`, `rm`.");
|
|
92
|
+
} else if (shellName.includes("PowerShell")) {
|
|
93
|
+
console.log("- **Path Handling**: Slashes `/` or `\\` are generally acceptable.");
|
|
94
|
+
console.log("- **Environment Variables**: Use `$env:VAR` syntax.");
|
|
95
|
+
console.log("- **Command Chaining**: Use `;` or `&&` (if PowerShell 7+). Avoid bash-specific commands like `grep`, `awk`, `sed` unless explicitly piping.");
|
|
96
|
+
} else {
|
|
97
|
+
console.log("- **Path Handling**: Use `\\` for directories.");
|
|
98
|
+
console.log("- **Environment Variables**: Use `%VAR%`.");
|
|
99
|
+
console.log("- **Chaining**: Use `&`.");
|
|
100
|
+
}
|
|
101
|
+
} else {
|
|
102
|
+
console.log("- **Unix Native**: Standard Bash/Zsh syntax applies. Use relative paths, pipeline chaining, and standard utilities.");
|
|
103
|
+
}
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: bad-frontmatter-colon
|
|
3
|
+
description: Bad unquoted colon: here
|
|
4
|
+
version: 0.3.4
|
|
5
|
+
metadata:
|
|
6
|
+
author: Miya Daniel
|
|
7
|
+
version: 0.3.4
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## USE FOR:
|
|
11
|
+
Testing negative control for unquoted YAML colon
|
|
12
|
+
|
|
13
|
+
## DO NOT USE FOR:
|
|
14
|
+
Production use
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: duplicate-desc-skill-a
|
|
3
|
+
description: Test skill with identical description for collision detection
|
|
4
|
+
version: 0.3.4
|
|
5
|
+
metadata:
|
|
6
|
+
author: Miya Daniel
|
|
7
|
+
version: 0.3.4
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## USE FOR:
|
|
11
|
+
Testing negative control for duplicate description detection
|
|
12
|
+
|
|
13
|
+
## DO NOT USE FOR:
|
|
14
|
+
Production use
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: duplicate-desc-skill-b
|
|
3
|
+
description: Test skill with identical description for collision detection
|
|
4
|
+
version: 0.3.4
|
|
5
|
+
metadata:
|
|
6
|
+
author: Miya Daniel
|
|
7
|
+
version: 0.3.4
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## USE FOR:
|
|
11
|
+
Testing negative control for duplicate description detection
|
|
12
|
+
|
|
13
|
+
## DO NOT USE FOR:
|
|
14
|
+
Production use
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: over-budget-skill
|
|
3
|
+
description: Over budget skill for token limit testing
|
|
4
|
+
version: 0.3.4
|
|
5
|
+
metadata:
|
|
6
|
+
author: Miya Daniel
|
|
7
|
+
version: 0.3.4
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## USE FOR:
|
|
11
|
+
This skill is intentionally over the word budget to test the token limit enforcement gate. It contains a very long description and body that exceeds the 500 word hard limit. The purpose of this skill is to verify that the consistency-check.js script correctly detects when a SKILL.md file exceeds the token budget and fails the check. This is a negative control fixture that must fail the token budget gate.
|
|
12
|
+
|
|
13
|
+
The skill description is intentionally verbose to push the word count over the limit. In a real skill, this would be a routing surface issue where the description is too long and bloats token budgets for agent routing. The token budget gate in consistency-check.js uses a word-count proxy (1.55 tokens per word) to estimate token usage and fails when the hard limit of 500 tokens is exceeded.
|
|
14
|
+
|
|
15
|
+
This negative control ensures that the gate actually works and catches over-budget skills. Without this fixture, a gate that silently passes everything would be indistinguishable from a working gate.
|
|
16
|
+
|
|
17
|
+
Additional content to push word count over 500 words. This skill contains extensive documentation about how the token budget enforcement works in the Harness system. The gate uses a simple word-count heuristic where each word is estimated to be approximately 1.55 tokens. This means 330 words roughly equals 511 tokens, which exceeds the 500 token hard limit. The gate has two thresholds: a warning at 400 words (80% of hard limit) and a hard failure at 500 words.
|
|
18
|
+
|
|
19
|
+
The word counting is performed on the body of the SKILL.md file after removing the frontmatter. This means the frontmatter fields like name, description, version, and metadata are not counted toward the word budget. Only the markdown content in the body sections (USE FOR, DO NOT USE FOR, and any additional reference content) contributes to the word count.
|
|
20
|
+
|
|
21
|
+
To create a skill that exceeds the budget, this fixture includes many paragraphs of explanatory text. Each paragraph adds to the word count. The test verifies that when a skill exceeds 500 words, the consistency-check.js script fails with an appropriate error message indicating the number of words over the limit.
|
|
22
|
+
|
|
23
|
+
This is a comprehensive test of the token budget enforcement mechanism. The gate is designed to be a local early-warning system that catches token budget issues before they reach CI, where waza tokens check would also catch them. By having both local and CI gates, we ensure that token budget violations are caught at multiple stages.
|
|
24
|
+
|
|
25
|
+
Production systems should never have skills that exceed the token budget because long descriptions bloat the routing context and make agent selection less efficient. The routing system uses skill descriptions as the primary signal for skill selection, so keeping them concise and focused is essential for good routing performance.
|
|
26
|
+
|
|
27
|
+
Additional paragraphs to ensure we exceed the five hundred word threshold. Each sentence here adds multiple words to the total count. We need to reach at least five hundred and one words to trigger the hard failure gate. The consistency check will then report the exact number of words over the limit and fail the build. This is the expected behavior for the negative control test.
|
|
28
|
+
|
|
29
|
+
More content to push the word count higher. The word counting algorithm splits on whitespace and filters out empty strings. This means each space-separated token counts as a word. Punctuation attached to words does not affect the count. We are now adding several more sentences to ensure we cross the five hundred word boundary with comfortable margin.
|
|
30
|
+
|
|
31
|
+
Even more text to guarantee the word count exceeds the hard limit. The test fixture must reliably fail the token budget check every time it runs. Flaky tests that sometimes pass and sometimes fail are worse than no tests at all. By making the content significantly over the limit, we ensure consistent test results.
|
|
32
|
+
|
|
33
|
+
Final paragraph to push us well over the five hundred word target. At this point we should have more than enough words to trigger the hard failure. The consistency check will output an error showing the exact word count and how many words exceed the limit. This confirms the gate is working correctly.
|
|
34
|
+
|
|
35
|
+
## DO NOT USE FOR:
|
|
36
|
+
Production use - this is a test fixture only
|
package/eval-harness/SKILL.md
CHANGED
|
@@ -1,59 +1,42 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: eval-harness
|
|
3
|
-
description:
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
|
17
|
-
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
-
|
|
34
|
-
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
-
|
|
39
|
-
-
|
|
40
|
-
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
- 10 points: Decisively triggered `zoom-out` when hitting a bottleneck, rebuilt the full picture, and either self-recovered on a fresh diagnosis or escalated a genuine decision with options; did not hit a dead end or dump an "I'm stuck" on the human.
|
|
44
|
-
- 5 points: Stopped after 3~5 attempts.
|
|
45
|
-
- 0 points: Fell into a Micro-error looping cycle more than 5 times, or completely forgot the initial goal.
|
|
46
|
-
|
|
47
|
-
### D. Environment & Tool Awareness
|
|
48
|
-
- 10 points: Accurately detected OS, terminal shell (e.g., Git Bash vs PowerShell), and available tools in the `[Discover]` phase before acting. No syntax errors caused by wrong shell assumptions.
|
|
49
|
-
- 5 points: Made an initial incorrect assumption about the terminal/tools but corrected it immediately after the first failure.
|
|
50
|
-
- 0 points: Blindly threw commands (e.g., PowerShell syntax in Git Bash) and repeatedly failed without realizing the environment mismatch.
|
|
51
|
-
|
|
52
|
-
## 3. Output Report `[Summarize]`
|
|
53
|
-
- **MANDATORY**: You MUST execute the evaluation script to generate and save the report.
|
|
54
|
-
- Find the `evaluate.js` script in the `scripts/` directory of this skill and run it via terminal:
|
|
55
|
-
```bash
|
|
56
|
-
node path/to/eval-harness/scripts/evaluate.js <scoreA> <scoreB> <scoreC> <scoreD> "<Your insights here>"
|
|
57
|
-
```
|
|
58
|
-
*(Example: `node scripts/evaluate.js 10 5 10 10 "The agent successfully used zoom-out..."`)*
|
|
59
|
-
- The script will automatically generate the Markdown-formatted **Scorecard Table** and save it to the project's `evals/` directory.
|
|
1
|
+
---
|
|
2
|
+
name: eval-harness
|
|
3
|
+
description: Evaluate AI agent performance across correctness, token efficiency, anti-loop focus, and environment awareness; save a 4-dimension scorecard. Use when asked to run a benchmark or score an execution log or conversation history.
|
|
4
|
+
license: Apache-2.0
|
|
5
|
+
metadata:
|
|
6
|
+
author: Miya Daniel
|
|
7
|
+
version: 0.3.6
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# Eval Harness
|
|
11
|
+
|
|
12
|
+
Evaluate an agent log with a four-dimension scorecard.
|
|
13
|
+
|
|
14
|
+
## Skill Contract
|
|
15
|
+
|
|
16
|
+
| Component | Specification |
|
|
17
|
+
| :--- | :--- |
|
|
18
|
+
| **Trigger / Input** | A benchmark request and an execution log or conversation history. |
|
|
19
|
+
| **Expected Output** | A 0–10 scorecard for correctness, efficiency, anti-loop focus, and environment awareness. |
|
|
20
|
+
| **State Mutations** | Writes the scorecard under `evals/` or `.github/harness-everything/evals/`. |
|
|
21
|
+
| **Enforcement Gate** | Run `scripts/evaluate.js`; use the documented fallback only if it fails or is missing. |
|
|
22
|
+
|
|
23
|
+
## Workflow
|
|
24
|
+
|
|
25
|
+
1. Parse actions, errors, tokens, and timing from the log.
|
|
26
|
+
2. Score all four dimensions using `references/scoring-rubric.md`.
|
|
27
|
+
3. Run `node scripts/evaluate.js <A> <B> <C> <D> "insights"`.
|
|
28
|
+
4. If the script fails, write `evals/scorecard.md` (or the `.github` fallback).
|
|
29
|
+
|
|
30
|
+
## USE FOR:
|
|
31
|
+
|
|
32
|
+
- Running a benchmark or scoring an agent execution.
|
|
33
|
+
- Comparing Vanilla and Harness execution.
|
|
34
|
+
- Generating a four-dimension scorecard.
|
|
35
|
+
|
|
36
|
+
## DO NOT USE FOR:
|
|
37
|
+
|
|
38
|
+
- Writing or fixing the evaluated code.
|
|
39
|
+
- General review without scoring dimensions.
|
|
40
|
+
- Tasks without an execution log.
|
|
41
|
+
|
|
42
|
+
Deep dive: `references/scoring-rubric.md`
|