harness-everything 0.3.6 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plugins/marketplace.json +20 -0
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +8 -7
- package/.github/workflows/behavioral-evals.yml +3 -3
- package/.github/workflows/ci.yml +49 -28
- package/.github/workflows/release.yml +52 -27
- package/.nvmrc +1 -0
- package/.releaserc.json +113 -0
- package/AGENTS.md +44 -24
- package/BENCHMARK_SOP.md +38 -18
- package/CHANGELOG.md +196 -3
- package/CONTRIBUTING.md +37 -9
- package/Dockerfile +1 -1
- package/PRIVACY.md +25 -0
- package/README.md +126 -91
- package/RELEASING.md +93 -0
- package/SUPPORT.md +27 -0
- package/TERMS.md +21 -0
- package/VERIFICATION.md +192 -284
- package/behavioral-evals/README.md +21 -0
- package/behavioral-evals/case-validator.js +194 -0
- package/behavioral-evals/cases/baseline-environment-detection.yaml +21 -0
- package/behavioral-evals/cases/baseline-eval-harness.yaml +27 -0
- package/behavioral-evals/cases/baseline-fable-discipline.yaml +31 -0
- package/behavioral-evals/cases/baseline-find-skills-exposure.yaml +49 -0
- package/behavioral-evals/cases/baseline-find-skills.yaml +21 -0
- package/behavioral-evals/cases/baseline-git-commit.yaml +2 -2
- package/behavioral-evals/cases/baseline-install-cognitive-os.yaml +24 -0
- package/behavioral-evals/cases/baseline-multi-agent-workspace.yaml +28 -0
- package/behavioral-evals/cases/baseline-rewrite-commits.yaml +34 -0
- package/behavioral-evals/cases/baseline-self-evolve.yaml +31 -0
- package/behavioral-evals/cases/baseline-skill-creator.yaml +20 -0
- package/behavioral-evals/cases/baseline-skill-style.yaml +21 -0
- package/behavioral-evals/cases/baseline-to-spec.yaml +34 -0
- package/behavioral-evals/cases/baseline-to-tickets.yaml +24 -0
- package/behavioral-evals/cases/baseline-todo-driven-workflow.yaml +32 -0
- package/behavioral-evals/cases/baseline-using-git-worktrees.yaml +15 -0
- package/behavioral-evals/cases/breaker-zoom-out-after-3.yaml +2 -2
- package/behavioral-evals/cases/pressure-complexity-skip.yaml +2 -2
- package/behavioral-evals/cases/pressure-scope-bypass.yaml +7 -2
- package/behavioral-evals/cases/pressure-sunk-cost-retry.yaml +7 -4
- package/behavioral-evals/cases/scope-discipline.yaml +2 -1
- package/behavioral-evals/cases/verify-before-done.yaml +7 -5
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/case.yaml +34 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/historical-case.yaml +34 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/historical-case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/historical-case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/historical-case.yaml +18 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/live-run-blocker.md +16 -0
- package/behavioral-evals/evidence/2026-09-07/manifest.json +88 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/historical-case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/historical-case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/case.yaml +35 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/historical-case.yaml +35 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/case.yaml +30 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/historical-case.yaml +23 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/triage-matrix.json +426 -0
- package/behavioral-evals/evidence/2026-09-07/triage-matrix.md +24 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/case.yaml +19 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/historical-case.yaml +18 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/historical-case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/result.sanitized.json +480 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/transcript.sanitized.jsonl +95 -0
- package/behavioral-evals/evidence-tool.js +443 -0
- package/behavioral-evals/fixtures/claude-denied.jsonl +3 -0
- package/behavioral-evals/fixtures/claude-legacy.json +1 -0
- package/behavioral-evals/fixtures/claude-oral-only.jsonl +2 -0
- package/behavioral-evals/fixtures/claude-success.jsonl +5 -0
- package/behavioral-evals/fixtures/claude-truncated.jsonl +3 -0
- package/behavioral-evals/fixtures/opencode-success.jsonl +6 -0
- package/behavioral-evals/results/2026-08-31-baseline-add-feature.json +20 -0
- package/behavioral-evals/results/2026-09-03-baseline-environment-detection-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-eval-harness-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-fable-discipline-pair.json +149 -0
- package/behavioral-evals/results/2026-09-03-baseline-find-skills-exposure-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-find-skills-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-install-cognitive-os-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-multi-agent-workspace-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-rewrite-commits-pair.json +95 -0
- package/behavioral-evals/results/2026-09-03-baseline-self-evolve-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-skill-creator-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-skill-style-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-to-spec-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-to-tickets-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-todo-driven-workflow-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-using-git-worktrees-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-summary-both.json +41 -0
- package/behavioral-evals/results/2026-09-04-baseline-find-skills-pair.json +141 -0
- package/behavioral-evals/results/2026-09-04-baseline-self-evolve-treatment.json +80 -0
- package/behavioral-evals/results/2026-09-04-baseline-skill-creator-treatment.json +80 -0
- package/behavioral-evals/results/2026-09-04-summary-both.json +41 -0
- package/behavioral-evals/run-with-plugin.js +15 -31
- package/behavioral-evals/run.js +446 -105
- package/behavioral-evals/transcript-parser.js +591 -0
- package/bin/cli.js +24 -12
- package/ci/ab-test-cases/grill-me-adversarial.yaml +14 -4
- package/ci/ab-test-cases/tdd-test-first.yaml +13 -6
- package/ci/ab-test-cases/verification-loop-gates.yaml +20 -7
- package/ci/ab-test-cases/verify-before-claim-cites.yaml +7 -6
- package/ci/ab-test-harness.js +113 -38
- package/ci/action-gate-platform-doc.test.js +31 -0
- package/ci/action-gate.test.js +260 -0
- package/ci/cases/case2.json +2 -2
- package/ci/consistency-check.js +51 -5
- package/ci/description-collision.js +15 -6
- package/ci/disclosure-check.js +225 -0
- package/ci/doc-capability-consistency.test.js +180 -0
- package/ci/fixtures/phase4/parallel-disjoint.stages.json +32 -0
- package/ci/fixtures/phase4/parallel-read-only.stages.json +32 -0
- package/ci/installer-roundtrip.test.js +159 -0
- package/ci/invariant-routing.test.js +74 -0
- package/ci/issue99-runtime.test.js +218 -0
- package/ci/mechanism-2aa-codex-action-gate-policy.test.js +117 -0
- package/ci/mechanism-2ab-openai-plugin-sync-integrity.test.js +80 -0
- package/ci/mechanism-2j-behavioral-runner.test.js +43 -1
- package/ci/mechanism-2k-reference-check.test.js +57 -2
- package/ci/mechanism-2l-multi-agent-workspace.test.js +202 -6
- package/ci/mechanism-2m-self-repo-guard.test.js +65 -0
- package/ci/mechanism-2n-opencode-plugin.test.js +461 -0
- package/ci/mechanism-2o-state-home-isolation.test.js +183 -0
- package/ci/mechanism-2p-installer-links.test.js +183 -0
- package/ci/mechanism-2q-installer-global-shared-store.test.js +132 -0
- package/ci/mechanism-2r-uninstall-flags.test.js +148 -0
- package/ci/mechanism-2s-tdd-quality.test.js +203 -0
- package/ci/mechanism-2t-disclosure.test.js +90 -0
- package/ci/mechanism-2t-evidence-triage.test.js +200 -0
- package/ci/mechanism-2t-issue-20.test.js +75 -0
- package/ci/mechanism-2t-platform-skill-targets.test.js +164 -0
- package/ci/mechanism-2t-transcript-parser.test.js +179 -0
- package/ci/mechanism-2t-workspace-identity.test.js +200 -0
- package/ci/mechanism-2u-issue-42-regressions.test.js +239 -0
- package/ci/mechanism-2u-issue-44.test.js +35 -0
- package/ci/mechanism-2v-verifier-boundaries.test.js +138 -0
- package/ci/mechanism-2w-platform-compatibility.test.js +148 -0
- package/ci/mechanism-2x-release-automation.test.js +157 -0
- package/ci/mechanism-2x-release-commit-message.test.js +66 -0
- package/ci/mechanism-2y-release-tooling-compat.test.js +47 -0
- package/ci/mechanism-2z-claude-plugin-manifest.test.js +29 -0
- package/ci/mutation-check.js +53 -0
- package/ci/mutation-loader.js +57 -0
- package/ci/negative-controls.js +76 -46
- package/ci/openai-plugin-package.test.js +182 -0
- package/ci/openai-plugin-runtime.test.js +175 -0
- package/ci/openai-plugin-submission.test.js +101 -0
- package/ci/phase4-ensemble-memory.test.js +307 -0
- package/ci/reference-check.js +73 -25
- package/ci/release-consistency-check.js +24 -1
- package/ci/repository-contract.test.js +17 -0
- package/ci/router-phase2-safety.test.js +81 -0
- package/ci/router-workflow-plan.test.js +264 -0
- package/ci/routing-check.js +6 -5
- package/ci/runner.js +6 -5
- package/ci/skill-routing-check.js +48 -11
- package/ci/test-behavioral-case-validation.js +62 -0
- package/ci/test-helper.js +10 -0
- package/ci/workflow-plan-consumers-edge.test.js +72 -0
- package/ci/workflow-plan-consumers.test.js +276 -0
- package/docs/architecture.md +137 -57
- package/docs/audit.md +22 -20
- package/docs/codex-action-gate.md +90 -0
- package/docs/issue-20-rollup.md +47 -0
- package/docs/mechanism-first-skill-mesh.md +22 -20
- package/docs/openai-plugin.md +118 -0
- package/docs/philosophy.md +28 -17
- package/docs/platform-capabilities.md +73 -0
- package/docs/platform-compatibility.json +270 -0
- package/docs/reflection.md +37 -22
- package/docs/release-evidence.md +27 -0
- package/docs/repository-contract.md +27 -0
- package/docs/routing.md +157 -44
- package/docs/security-audit.md +38 -0
- package/docs/troubleshooting.md +94 -51
- package/docs/verifier-boundaries.md +45 -0
- package/docs/workflows/fable-mode.md +77 -21
- package/docs/workflows/find-skills.md +40 -0
- package/docs/workflows/grill-with-docs.md +2 -2
- package/docs/workflows/multi-agent-workspace.md +22 -10
- package/docs/workflows/self-evolve.md +33 -0
- package/docs/workflows/tdd.md +36 -0
- package/docs/workflows/todo-driven-workflow.md +33 -0
- package/environment-detection/SKILL.md +3 -3
- package/eval-harness/SKILL.md +7 -7
- package/eval-harness/scripts/evaluate.js +12 -10
- package/evals/evaluation-reports.md +14 -0
- package/evals/harness-everything/tasks/positive-tier1.yaml +1 -1
- package/evals/harness-everything/tasks/positive-tier2.yaml +1 -1
- package/evals/harness-everything/tasks/positive-tier3.yaml +1 -1
- package/evals/install-cognitive-os/tasks/positive-trigger.yaml +1 -1
- package/fable-discipline/SKILL.md +2 -2
- package/fable-mode/CONTRACT-FORMAT.md +147 -33
- package/fable-mode/SKILL.md +3 -3
- package/fable-mode/agents/fable-orchestrator.md +95 -35
- package/fable-mode/execution-guardrails/SKILL.md +2 -2
- package/fable-mode/fable-haiku/SKILL.md +3 -3
- package/fable-mode/fable-opus/SKILL.md +3 -3
- package/fable-mode/fable-sonnet/SKILL.md +3 -3
- package/fable-mode/scripts/ensemble-review.js +371 -0
- package/fable-mode/scripts/model-selector.js +134 -3
- package/fable-mode/scripts/workflow-plan-consumer.js +337 -0
- package/find-skills/SKILL.md +5 -5
- package/find-skills/references/discovery-flow.md +1 -1
- package/find-skills/scripts/use-skill.js +7 -1
- package/git-commit/SKILL.md +4 -11
- package/grill-me/SKILL.md +5 -5
- package/grill-with-docs/SKILL.md +7 -7
- package/grill-with-docs/references/session-playbook.md +6 -4
- package/harness-everything/SKILL.md +23 -22
- package/harness-everything/references/router-workflow-plan.md +128 -0
- package/harness-everything/references/triage-and-tiers.md +173 -96
- package/harness-everything/schemas/router-task-shape.schema.json +126 -0
- package/harness-everything/schemas/router-workflow-plan.schema.json +149 -0
- package/harness-everything/scripts/bootstrap.js +13 -4
- package/harness-everything/scripts/ensemble-policy.js +102 -0
- package/harness-everything/scripts/kernel-router.js +212 -0
- package/harness-everything/scripts/router-contract.js +612 -0
- package/harness-everything/scripts/self-heal.js +29 -20
- package/harness-everything/scripts/tier-router.js +346 -221
- package/hooks/hooks.json +60 -15
- package/hooks/scripts/action-gate-rules.json +77 -0
- package/hooks/scripts/action-gate.js +528 -0
- package/hooks/scripts/atomic-commit-check.js +5 -5
- package/hooks/scripts/codex-action-gate-post.js +46 -0
- package/hooks/scripts/codex-permission-request.js +140 -0
- package/hooks/scripts/context-compact.js +4 -3
- package/hooks/scripts/contract-test.js +118 -51
- package/hooks/scripts/lib/fable-contracts.js +121 -0
- package/hooks/scripts/lib/harness-state.js +164 -28
- package/hooks/scripts/lib/platforms/claude.js +45 -62
- package/hooks/scripts/lib/platforms/codex.js +35 -61
- package/hooks/scripts/lib/platforms/continue.js +7 -7
- package/hooks/scripts/lib/platforms/copilot.js +29 -53
- package/hooks/scripts/lib/platforms/hermes.js +71 -20
- package/hooks/scripts/openai-session-start.js +81 -0
- package/hooks/scripts/rule-of-3-tracker.js +9 -8
- package/hooks/scripts/rule-of-3.js +29 -21
- package/hooks/scripts/state-persist.js +5 -4
- package/hooks/scripts/stop-gate.js +2 -2
- package/hooks/scripts/subagent-scope-guard.js +144 -38
- package/hooks/scripts/subagent-start.js +19 -0
- package/hooks/scripts/subagent-stop.js +6 -0
- package/improve-codebase-architecture/SKILL.md +2 -4
- package/improve-codebase-architecture/references/README.md +16 -0
- package/install-cognitive-os/SKILL.md +28 -22
- package/multi-agent-workspace/SKILL.md +16 -15
- package/multi-agent-workspace/references/architecture-guide.md +13 -9
- package/multi-agent-workspace/references/orchestration.md +39 -22
- package/multi-agent-workspace/scripts/consume-workflow-plan.js +129 -0
- package/multi-agent-workspace/scripts/index_memory.js +79 -26
- package/multi-agent-workspace/scripts/path-boundary.js +59 -0
- package/multi-agent-workspace/scripts/project-docs-resolver.js +357 -0
- package/multi-agent-workspace/scripts/scaffold.js +258 -103
- package/multi-agent-workspace/templates/AGENTS.md +20 -0
- package/multi-agent-workspace/workflows/03-generation.md +8 -10
- package/multi-agent-workspace/workflows/04-launcher.md +5 -5
- package/opencode-plugin/README.md +114 -71
- package/opencode-plugin/hooks/verify.js +62 -12
- package/opencode-plugin/index.mjs +547 -0
- package/opencode-plugin/plugin.json +4 -12
- package/package.json +20 -4
- package/plugins/harness-everything/.codex-plugin/plugin.json +39 -0
- package/plugins/harness-everything/hooks/hooks.json +197 -0
- package/plugins/harness-everything/hooks/scripts/action-gate-rules.json +77 -0
- package/plugins/harness-everything/hooks/scripts/action-gate.js +528 -0
- package/plugins/harness-everything/hooks/scripts/atomic-commit-check.js +68 -0
- package/plugins/harness-everything/hooks/scripts/boundary-guard.js +55 -0
- package/plugins/harness-everything/hooks/scripts/codex-action-gate-post.js +46 -0
- package/plugins/harness-everything/hooks/scripts/codex-permission-request.js +140 -0
- package/plugins/harness-everything/hooks/scripts/context-compact.js +115 -0
- package/plugins/harness-everything/hooks/scripts/contract-test.js +146 -0
- package/plugins/harness-everything/hooks/scripts/depth-guard.js +65 -0
- package/plugins/harness-everything/hooks/scripts/lib/fable-contracts.js +121 -0
- package/plugins/harness-everything/hooks/scripts/lib/harness-state.js +275 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/claude.js +168 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/codex.js +88 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/continue.js +91 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/copilot.js +85 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/cursor.js +89 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/hermes.js +108 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/index.js +17 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/worktrees.js +20 -0
- package/plugins/harness-everything/hooks/scripts/lib/workspace.js +290 -0
- package/plugins/harness-everything/hooks/scripts/openai-session-start.js +81 -0
- package/plugins/harness-everything/hooks/scripts/reset-circuit-breaker.js +30 -0
- package/plugins/harness-everything/hooks/scripts/rule-of-3-tracker.js +125 -0
- package/plugins/harness-everything/hooks/scripts/rule-of-3.js +195 -0
- package/plugins/harness-everything/hooks/scripts/state-persist.js +100 -0
- package/plugins/harness-everything/hooks/scripts/stop-gate.js +80 -0
- package/plugins/harness-everything/hooks/scripts/subagent-scope-guard.js +194 -0
- package/plugins/harness-everything/hooks/scripts/subagent-start.js +19 -0
- package/plugins/harness-everything/hooks/scripts/subagent-stop.js +6 -0
- package/plugins/harness-everything/hooks/session-start.js +81 -0
- package/plugins/harness-everything/plugin.json +43 -0
- package/plugins/harness-everything/scripts/lib/advisory-text.js +174 -0
- package/plugins/harness-everything/skills/environment-detection/SKILL.md +41 -0
- package/plugins/harness-everything/skills/environment-detection/references/shell-syntax-rules.md +76 -0
- package/plugins/harness-everything/skills/environment-detection/scripts/preflight.js +103 -0
- package/plugins/harness-everything/skills/eval-harness/SKILL.md +42 -0
- package/plugins/harness-everything/skills/eval-harness/references/scoring-rubric.md +68 -0
- package/plugins/harness-everything/skills/eval-harness/scripts/evaluate.js +79 -0
- package/plugins/harness-everything/skills/fable-discipline/SKILL.md +41 -0
- package/plugins/harness-everything/skills/fable-discipline/references/discipline-rules.md +22 -0
- package/plugins/harness-everything/skills/fable-mode/CONTRACT-FORMAT.md +165 -0
- package/plugins/harness-everything/skills/fable-mode/SKILL.md +39 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-orchestrator.md +159 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-verifier.md +32 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-haiku.md +52 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-sonnet.md +53 -0
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/SKILL.md +43 -0
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/references/guardrail-rules.md +56 -0
- package/plugins/harness-everything/skills/fable-mode/fable-haiku/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/fable-haiku/references/routing-notes.md +26 -0
- package/plugins/harness-everything/skills/fable-mode/fable-opus/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/fable-sonnet/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/model-matrix.json +32 -0
- package/plugins/harness-everything/skills/fable-mode/references/execution-phases.md +37 -0
- package/plugins/harness-everything/skills/fable-mode/references/model-matrix.md +39 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/ensemble-review.js +371 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/model-selector.js +260 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/workflow-plan-consumer.js +337 -0
- package/plugins/harness-everything/skills/find-skills/SKILL.md +48 -0
- package/plugins/harness-everything/skills/find-skills/references/discovery-flow.md +102 -0
- package/plugins/harness-everything/skills/find-skills/scripts/use-skill.js +112 -0
- package/plugins/harness-everything/skills/git-commit/SKILL.md +38 -0
- package/plugins/harness-everything/skills/git-commit/guides/ANGULAR_STYLE.md +61 -0
- package/plugins/harness-everything/skills/git-commit/guides/COMMIT_GENERATION.md +24 -0
- package/plugins/harness-everything/skills/git-commit/guides/LANGUAGE_DETECTION.md +16 -0
- package/plugins/harness-everything/skills/git-commit/guides/MAIN_REPO.md +15 -0
- package/plugins/harness-everything/skills/git-commit/guides/SUBMODULES.md +27 -0
- package/plugins/harness-everything/skills/git-commit/references/commit-flow.md +30 -0
- package/plugins/harness-everything/skills/grill-me/SKILL.md +43 -0
- package/plugins/harness-everything/skills/grill-me/references/grilling-playbook.md +38 -0
- package/plugins/harness-everything/skills/grill-with-docs/ADR-FORMAT.md +47 -0
- package/plugins/harness-everything/skills/grill-with-docs/CONTEXT-FORMAT.md +63 -0
- package/plugins/harness-everything/skills/grill-with-docs/SKILL.md +41 -0
- package/plugins/harness-everything/skills/grill-with-docs/references/session-playbook.md +99 -0
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +43 -0
- package/plugins/harness-everything/skills/harness-everything/examples/README.md +9 -0
- package/plugins/harness-everything/skills/harness-everything/examples/todo-workflow.md +22 -0
- package/plugins/harness-everything/skills/harness-everything/references/router-workflow-plan.md +128 -0
- package/plugins/harness-everything/skills/harness-everything/references/skill-registry.md +39 -0
- package/plugins/harness-everything/skills/harness-everything/references/triage-and-tiers.md +193 -0
- package/plugins/harness-everything/skills/harness-everything/schemas/router-task-shape.schema.json +126 -0
- package/plugins/harness-everything/skills/harness-everything/schemas/router-workflow-plan.schema.json +149 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/bootstrap.js +212 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/ensemble-policy.js +102 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +212 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/router-contract.js +612 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/routing-keywords.json +320 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/self-heal.js +250 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/tier-router.js +459 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/verify-gate.js +100 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/SKILL.md +38 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/DEEPENING.md +37 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/HTML-REPORT.md +123 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/LANGUAGE.md +53 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/refactoring-workflow.md +59 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/references/README.md +16 -0
- package/plugins/harness-everything/skills/install-cognitive-os/SKILL.md +44 -0
- package/plugins/harness-everything/skills/install-cognitive-os/references/cognitive-loop.md +27 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/NOTICE-agency-agents.md +27 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/SKILL.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/examples/csharp-monorepo.md +123 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/examples/typescript-cli.md +126 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-claude.md +77 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-codex.md +146 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-continue.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-copilot.md +74 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-cursor.md +40 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-hermes.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/role-capabilities.md +108 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/agency-agents.md +31 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/architecture-guide.md +36 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/orchestration.md +65 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/agency-catalog.js +190 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/consume-workflow-plan.js +129 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/index_memory.js +104 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/path-boundary.js +59 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/project-docs-resolver.js +357 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/scaffold.js +408 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/AGENTS.md +20 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/README.md +17 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/agent-launcher.md +68 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-csharp.md +56 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-polyglot.md +62 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-ts.md +57 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/memory-keeper.md +37 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/requirement-analyzer.md +66 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/specialist-agent.md +34 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/task-verifier.md +56 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/tool-maker.md +33 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/01-init.md +22 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/02-analysis.md +23 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/03-generation.md +25 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/04-launcher.md +16 -0
- package/plugins/harness-everything/skills/repo-docs/SKILL.md +42 -0
- package/plugins/harness-everything/skills/repo-docs/references/process-guide.md +60 -0
- package/plugins/harness-everything/skills/repo-docs/templates/agents-template.md +264 -0
- package/plugins/harness-everything/skills/repo-docs/templates/knowledge-agents-template.md +50 -0
- package/plugins/harness-everything/skills/repo-docs/templates/knowledge-readme-template.md +83 -0
- package/plugins/harness-everything/skills/repo-docs/templates/manual-recon.md +117 -0
- package/plugins/harness-everything/skills/repo-docs/templates/multi-skills-readme-template.md +72 -0
- package/plugins/harness-everything/skills/repo-docs/templates/product-agents-template.md +78 -0
- package/plugins/harness-everything/skills/repo-docs/templates/product-readme-template.md +95 -0
- package/plugins/harness-everything/skills/repo-docs/templates/readme-template.md +136 -0
- package/plugins/harness-everything/skills/repo-docs/templates/single-skill-readme-template.md +70 -0
- package/plugins/harness-everything/skills/repo-docs/templates/skills-agents-template.md +60 -0
- package/plugins/harness-everything/skills/rewrite-commits/SKILL.md +37 -0
- package/plugins/harness-everything/skills/rewrite-commits/references/workflow.md +18 -0
- package/plugins/harness-everything/skills/security-review/SKILL.md +47 -0
- package/plugins/harness-everything/skills/security-review/guides/OWASP-PATTERNS.md +31 -0
- package/plugins/harness-everything/skills/security-review/guides/STRIDE-THREAT-MODEL.md +21 -0
- package/plugins/harness-everything/skills/security-review/references/security-checklist.md +478 -0
- package/plugins/harness-everything/skills/security-review/scripts/audit-secrets.js +85 -0
- package/plugins/harness-everything/skills/self-evolve/SKILL.md +45 -0
- package/plugins/harness-everything/skills/self-evolve/references/memory-resolution.md +61 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/persist-memory.js +272 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/register-dynamic-skill.js +253 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/self-regression.js +125 -0
- package/plugins/harness-everything/skills/skill-creator/SKILL.md +47 -0
- package/plugins/harness-everything/skills/skill-creator/references/authoring-workflow.md +14 -0
- package/plugins/harness-everything/skills/skill-creator/references/dynamic-generation-contract.md +21 -0
- package/plugins/harness-everything/skills/skill-creator/references/quality-checklist.md +16 -0
- package/plugins/harness-everything/skills/skill-creator/references/quality-principles.md +81 -0
- package/plugins/harness-everything/skills/skill-creator/references/testing-workflow.md +27 -0
- package/plugins/harness-everything/skills/skill-style/SKILL.md +42 -0
- package/plugins/harness-everything/skills/skill-style/references/style-guide.md +62 -0
- package/plugins/harness-everything/skills/tdd/SKILL.md +38 -0
- package/plugins/harness-everything/skills/tdd/fixtures/pass-evidence.json +100 -0
- package/plugins/harness-everything/skills/tdd/fixtures/quality-scenarios.json +20 -0
- package/plugins/harness-everything/skills/tdd/guides/deep-modules.md +33 -0
- package/plugins/harness-everything/skills/tdd/guides/interface-design.md +31 -0
- package/plugins/harness-everything/skills/tdd/guides/mocking.md +59 -0
- package/plugins/harness-everything/skills/tdd/guides/refactoring.md +10 -0
- package/plugins/harness-everything/skills/tdd/guides/tests.md +61 -0
- package/plugins/harness-everything/skills/tdd/references/core-discipline.md +51 -0
- package/plugins/harness-everything/skills/tdd/references/integration-testing.md +13 -0
- package/plugins/harness-everything/skills/tdd/references/quality-model.md +68 -0
- package/plugins/harness-everything/skills/tdd/references/unit-testing.md +15 -0
- package/plugins/harness-everything/skills/tdd/schemas/quality-evidence.schema.json +211 -0
- package/plugins/harness-everything/skills/tdd/schemas/quality-report.schema.json +110 -0
- package/plugins/harness-everything/skills/tdd/scripts/quality-gate.js +475 -0
- package/plugins/harness-everything/skills/to-spec/SKILL.md +40 -0
- package/plugins/harness-everything/skills/to-spec/references/process.md +77 -0
- package/plugins/harness-everything/skills/to-spec/scripts/check-project-docs.js +239 -0
- package/plugins/harness-everything/skills/to-spec/scripts/project-docs-resolver.js +111 -0
- package/plugins/harness-everything/skills/to-spec/templates/cli-reference.md +31 -0
- package/plugins/harness-everything/skills/to-spec/templates/dev-doc.md +27 -0
- package/plugins/harness-everything/skills/to-spec/templates/feature-spec.md +59 -0
- package/plugins/harness-everything/skills/to-spec/templates/schema-doc.md +31 -0
- package/plugins/harness-everything/skills/to-tickets/SKILL.md +42 -0
- package/plugins/harness-everything/skills/to-tickets/references/publishing-and-templates.md +97 -0
- package/plugins/harness-everything/skills/todo-driven-workflow/SKILL.md +42 -0
- package/plugins/harness-everything/skills/todo-driven-workflow/references/execution-guide.md +27 -0
- package/plugins/harness-everything/skills/using-git-worktrees/SKILL.md +49 -0
- package/plugins/harness-everything/skills/using-git-worktrees/references/workflow-details.md +108 -0
- package/plugins/harness-everything/skills/verification-loop/SKILL.md +46 -0
- package/plugins/harness-everything/skills/verification-loop/references/verification-phases.md +116 -0
- package/plugins/harness-everything/skills/verification-loop/templates/verification-report.template.md +15 -0
- package/plugins/harness-everything/skills/verify-before-claim/SKILL.md +43 -0
- package/plugins/harness-everything/skills/verify-before-claim/references/verification-guide.md +73 -0
- package/plugins/harness-everything/skills/zoom-out/SKILL.md +42 -0
- package/plugins/harness-everything/skills/zoom-out/references/circuit-breaker.md +66 -0
- package/plugins/harness-everything/skills/zoom-out/templates/zoom-out-report.template.md +20 -0
- package/repo-docs/SKILL.md +3 -7
- package/rewrite-commits/SKILL.md +2 -2
- package/scripts/build-openai-submission.js +158 -0
- package/scripts/installer.js +86 -11
- package/scripts/lib/advisory-text.js +25 -18
- package/scripts/lib/execution-contract.js +70 -0
- package/scripts/lib/manifest.js +34 -38
- package/scripts/lib/path-boundary.js +90 -0
- package/scripts/lib/skills.js +124 -70
- package/scripts/lib/workspace.js +269 -7
- package/scripts/repository-contract.js +213 -0
- package/scripts/sync-current-docs.js +12 -0
- package/scripts/sync-openai-plugin.js +91 -0
- package/scripts/sync-release-version.js +212 -0
- package/security-review/SKILL.md +4 -7
- package/security-review/scripts/audit-secrets.js +9 -8
- package/self-evolve/SKILL.md +9 -11
- package/self-evolve/references/memory-resolution.md +4 -1
- package/self-evolve/scripts/persist-memory.js +221 -88
- package/self-evolve/scripts/register-dynamic-skill.js +12 -10
- package/self-evolve/scripts/self-regression.js +23 -8
- package/skill-creator/SKILL.md +12 -14
- package/skill-creator/references/quality-checklist.md +1 -0
- package/skill-style/SKILL.md +4 -4
- package/skill-style/references/style-guide.md +29 -3
- package/submission/openai/README.md +67 -0
- package/submission/openai/listing.json +31 -0
- package/submission/openai/test-cases.json +61 -0
- package/tdd/SKILL.md +20 -26
- package/tdd/fixtures/pass-evidence.json +100 -0
- package/tdd/fixtures/quality-scenarios.json +20 -0
- package/tdd/references/core-discipline.md +2 -0
- package/tdd/references/integration-testing.md +13 -0
- package/tdd/references/quality-model.md +68 -0
- package/tdd/references/unit-testing.md +15 -0
- package/tdd/schemas/quality-evidence.schema.json +211 -0
- package/tdd/schemas/quality-report.schema.json +110 -0
- package/tdd/scripts/quality-gate.js +475 -0
- package/to-spec/SKILL.md +3 -5
- package/to-spec/scripts/check-project-docs.js +42 -62
- package/to-spec/scripts/project-docs-resolver.js +111 -0
- package/to-tickets/SKILL.md +4 -4
- package/todo-driven-workflow/SKILL.md +6 -4
- package/using-git-worktrees/SKILL.md +3 -3
- package/verification-loop/SKILL.md +4 -4
- package/verify-before-claim/SKILL.md +2 -2
- package/zoom-out/SKILL.md +4 -4
- package/ab-test-deep-analysis-2026-08-27.md +0 -383
- package/ab-test-results-2026-08-27.md +0 -152
- package/ab-test-statistical-analysis-31cases.md +0 -193
- package/opencode-plugin/hooks/circuit-breaker.js +0 -121
- package/opencode-plugin/hooks/compliance.js +0 -128
- package/opencode-plugin/hooks/post-edit.js +0 -62
- package/opencode-plugin/hooks/pre-complete.js +0 -65
- package/opencode-plugin/test-hook-firing.js +0 -198
package/VERIFICATION.md
CHANGED
|
@@ -1,394 +1,302 @@
|
|
|
1
1
|
# Verification & Acceptance Checklist
|
|
2
2
|
|
|
3
|
-
Run this after installing Harness into a project, or whenever you need to confirm
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
- **
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
- **Behavior** — did the agent's
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
expected, not a bug — see [README: Supported AI IDEs & Tools](README.md#supported-ai-ides--tools).
|
|
16
|
-
A platform where the mechanism check passes but behavior doesn't follow it is
|
|
17
|
-
the interesting failure: the hook fired, but the agent didn't act on the
|
|
18
|
-
context it was given.
|
|
19
|
-
|
|
20
|
-
Do not accept "I read the code and it looks right" as a pass for anything
|
|
21
|
-
below — every check here names an exact command and an exact expected result.
|
|
22
|
-
Run it for real.
|
|
3
|
+
Run this after installing Harness into a project, or whenever you need to confirm that an integration actually works rather than merely looking correct in the repository.
|
|
4
|
+
|
|
5
|
+
Harness verification separates four kinds of evidence that must not be conflated:
|
|
6
|
+
|
|
7
|
+
- **Package / artifact evidence** — are the expected files, manifests, skills, and hooks present?
|
|
8
|
+
- **Mechanism evidence** — did the packaged hook/plugin function execute and produce the documented result?
|
|
9
|
+
- **Live-host evidence** — did a real host load and fire that mechanism in an actual session?
|
|
10
|
+
- **Behavior evidence** — did the agent's conduct follow the intended routing, evidence, and recovery disciplines?
|
|
11
|
+
|
|
12
|
+
A mechanism test can pass while a live host never loads the plugin. A behavior test can pass on an advisory-only surface without proving any hard enforcement. Keep those claims separate.
|
|
13
|
+
|
|
14
|
+
The authoritative current platform boundary is [docs/platform-capabilities.md](docs/platform-capabilities.md). The generated current runtime/workflow boundary is [docs/repository-contract.md](docs/repository-contract.md).
|
|
23
15
|
|
|
24
16
|
---
|
|
25
17
|
|
|
26
|
-
## 1.
|
|
18
|
+
## 1. Installation surface check
|
|
27
19
|
|
|
28
|
-
|
|
29
|
-
downstream.
|
|
20
|
+
First identify **which Harness surface you are testing**. One host can have more than one surface; Codex is the main example.
|
|
30
21
|
|
|
31
|
-
|
|
|
22
|
+
| Surface | Artifact / location | What artifact presence proves |
|
|
32
23
|
|---|---|---|
|
|
33
|
-
| Claude Code | `.claude/settings.json
|
|
34
|
-
|
|
|
35
|
-
|
|
|
36
|
-
| Codex | `
|
|
37
|
-
|
|
|
38
|
-
|
|
|
24
|
+
| Claude Code | `.claude/settings.json`, `.claude/skills/`, `.claude/agents/` | The Claude installer/plugin wrote its lifecycle-hook configuration and content |
|
|
25
|
+
| OpenCode | `.opencode/plugins/` / `opencode-plugin/index.mjs` | The native plugin module exists; this alone does not prove live loading |
|
|
26
|
+
| Codex — general installer | `AGENTS.md`, repo-scoped `.agents/skills/` | Advisory/instruction integration and project Agent Skills are installed |
|
|
27
|
+
| **Codex / local OpenAI plugin** | `.agents/plugins/marketplace.json`, `plugins/harness-everything/.codex-plugin/plugin.json`, packaged hooks/skills | The local OpenAI plugin package is structurally present |
|
|
28
|
+
| **Public OpenAI Skills-only** | generated `dist/openai-submission/harness-everything-skills.zip` | The public-review bundle was generated; local `.codex-plugin` lifecycle hooks are intentionally not part of this artifact |
|
|
29
|
+
| Cursor | `.cursorrules`, `.cursor/skills/` | Advisory project rules and project skills exist |
|
|
30
|
+
| Copilot Chat | `.github/copilot-instructions.md`, `.github/skills/` | Advisory repository instructions and project skills exist |
|
|
31
|
+
| Continue.dev | `.continue/rules/harness.md`, `.continue/skills/` | Advisory native rule and project skills exist |
|
|
32
|
+
| Hermes Agent | `.hermes.md`, trusted project `.agents/skills/` | Advisory project context and discoverable project skills exist; Hermes trust policy still applies |
|
|
39
33
|
|
|
40
|
-
|
|
41
|
-
grep -l "Harness OS Guidance" .cursorrules .github/copilot-instructions.md AGENTS.md .continue/rules/harness.md .hermes.md 2>/dev/null
|
|
42
|
-
node -e "console.log(Object.keys(JSON.parse(require('fs').readFileSync('.claude/settings.json','utf8')).hooks))"
|
|
43
|
-
```
|
|
34
|
+
For user/global skills, verify the host-native path rather than assuming every host consumes the shared Agent Skills directory: Claude uses `~/.claude/skills/`, Continue uses `~/.continue/skills/`, Hermes uses `~/.hermes/skills/`, and the supported shared Agent Skills targets use `~/.agents/skills/`.
|
|
44
35
|
|
|
45
|
-
|
|
46
|
-
marker — re-run `npm run harness:reset && node scripts/installer.js` (or the
|
|
47
|
-
`npx github:...install` form) and check again before going further.
|
|
36
|
+
### 1a. General installer checks
|
|
48
37
|
|
|
49
|
-
|
|
38
|
+
```bash
|
|
39
|
+
# Advisory surfaces
|
|
40
|
+
node -e "const fs=require('fs'); for (const p of ['.cursorrules','.github/copilot-instructions.md','AGENTS.md','.continue/rules/harness.md','.hermes.md']) if (fs.existsSync(p)) console.log(p)"
|
|
50
41
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
These pipe a simulated hook payload into the script exactly the way Claude
|
|
54
|
-
Code invokes it (JSON on stdin), and check the exit code and output against
|
|
55
|
-
what the hook is documented to do. Run from the repo root. Each block is
|
|
56
|
-
self-contained and cleans up after itself.
|
|
57
|
-
|
|
58
|
-
**Fastest path:** `npm run test:mechanism` (or plain `npm test`, which
|
|
59
|
-
includes it as Phase 4) runs every check in this section automatically,
|
|
60
|
-
end-to-end, on Windows/macOS/Linux alike — see
|
|
61
|
-
[ci/mechanism-test.js](ci/mechanism-test.js).
|
|
62
|
-
|
|
63
|
-
To ensure complete test traceability and avoid untested blind spots, the mechanism
|
|
64
|
-
checks are split into isolated, dedicated test suites under `ci/`:
|
|
65
|
-
- `mechanism-2a-rule-of-3.test.js` — Rule of 3 circuit breaker
|
|
66
|
-
- `mechanism-2b-boundary-guard.test.js` — Boundary guard block size limit
|
|
67
|
-
- `mechanism-2c-state-persist.test.js` — WAL fail-safe state recording
|
|
68
|
-
- `mechanism-2d-fact-audit.test.js` — Bilingual fact-audit stdin routing
|
|
69
|
-
- `mechanism-2e-scope-guard.test.js` — Subagent workspace scope containment
|
|
70
|
-
- `mechanism-2f-stop-gate.test.js` — Stop gate unverified edit bounce guard
|
|
71
|
-
- `mechanism-2g-platform-ignore.test.js` — Cross-platform selective gitignore exclusions
|
|
72
|
-
- `mechanism-2h-installer-manifest.test.js` — Installer manifest serialization and author-validation
|
|
73
|
-
|
|
74
|
-
The master test runner `mechanism-test.js` automatically discovers, sorts, and executes
|
|
75
|
-
each `.test.js` suite inline, printing a structured diagnostic table at the end.
|
|
76
|
-
|
|
77
|
-
The recipes below exist for isolating and debugging one mechanism by hand; they
|
|
78
|
-
pipe JSON into each hook via `node -e "...spawnSync(...)"` rather than a
|
|
79
|
-
shell `echo '...' | node script.js` pipe. Use the shell-pipe form if you
|
|
80
|
-
like — it works fine on macOS/Linux — but **not on Windows Git Bash**: three
|
|
81
|
-
independent 2026-07-23 audits used exactly that form and
|
|
82
|
-
got TTY/quoting interference that mangled the JSON, misdiagnosing working
|
|
83
|
-
hooks (`rule-of-3.js`, `boundary-guard.js`, `state-persist.js`) as broken.
|
|
84
|
-
The `node -e` form pipes stdin through Node's own `child_process` API
|
|
85
|
-
instead of the shell, so it behaves identically on every platform.
|
|
86
|
-
|
|
87
|
-
### 2a. Rule of 3 circuit breaker actually blocks
|
|
88
|
-
|
|
89
|
-
Runtime state lives under `.claude/harness-everything/state/sessions/<session_id>/`
|
|
90
|
-
(see [docs/architecture.md](docs/architecture.md)). Invocations with no
|
|
91
|
-
`session_id` in the piped payload — like these manual tests — fall into a
|
|
92
|
-
fixed `sessions/default/` bucket rather than a random one.
|
|
42
|
+
# Project skill roots used by the general installer
|
|
43
|
+
node -e "const fs=require('fs'); for (const p of ['.claude/skills','.cursor/skills','.github/skills','.agents/skills','.continue/skills']) if (fs.existsSync(p)) console.log(p)"
|
|
93
44
|
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
rm -f .claude/harness-everything/state/sessions/default/zoom-out-report.md
|
|
97
|
-
echo '{"count":3,"lastHash":"verify-test","zoomOutResolved":false}' > .claude/harness-everything/state/sessions/default/rule-of-3-state.json
|
|
98
|
-
node hooks/scripts/rule-of-3.js; echo "exit=$?"
|
|
45
|
+
# Claude hook configuration
|
|
46
|
+
node -e "const fs=require('fs'); const p='.claude/settings.json'; if (fs.existsSync(p)) console.log(Object.keys(JSON.parse(fs.readFileSync(p,'utf8')).hooks||{}))"
|
|
99
47
|
```
|
|
100
|
-
**Expect:** stderr prints `[CRITICAL] RULE OF 3 CIRCUIT BREAKER TRIGGERED!`
|
|
101
|
-
plus the reflect-first instructions (zoom-out protocol, and the exact report
|
|
102
|
-
path to write to — resolved per-session, so it'll read
|
|
103
|
-
`.claude/harness-everything/state/sessions/default/zoom-out-report.md` here), and `exit=2`.
|
|
104
|
-
(If you see `exit=1` or `exit=0`, the circuit breaker is not actually blocking
|
|
105
|
-
anything — `exit(1)` is a non-blocking error in Claude Code's hook contract,
|
|
106
|
-
this was a real bug found and fixed on 2026-07-20.)
|
|
107
48
|
|
|
108
|
-
|
|
109
|
-
npm run harness:reset
|
|
110
|
-
node hooks/scripts/rule-of-3.js; echo "exit=$?"
|
|
111
|
-
```
|
|
112
|
-
**Expect:** no stderr output, `exit=0`.
|
|
49
|
+
For an installer target that should be present, FAIL if its expected artifact is missing or does not contain Harness-owned guidance/configuration.
|
|
113
50
|
|
|
114
|
-
|
|
51
|
+
Run the deterministic round-trip gate to verify that installation and removal preserve seeded user-owned files and skills:
|
|
115
52
|
|
|
116
53
|
```bash
|
|
117
|
-
|
|
118
|
-
printf '## Goal\nx\n## Failed Attempts\nx\n## Verified Facts\nx\n## Diagnosis\nx\n## Decision\nRESUME: new approach\n' > .claude/harness-everything/state/sessions/default/zoom-out-report.md
|
|
119
|
-
node hooks/scripts/rule-of-3.js; echo "exit=$?"
|
|
120
|
-
node -e "const s=require('./.claude/harness-everything/state/sessions/default/rule-of-3-state.json'); console.log(s.count===0 && s.zoomOutResolved===true && s.zoomOutCycles===1 ? 'released-ok' : 'released-FAIL')"
|
|
54
|
+
node ci/installer-roundtrip.test.js
|
|
121
55
|
```
|
|
122
|
-
**Expect:** stdout prints `breaker released`, `exit=0`, then `released-ok` —
|
|
123
|
-
a completed reflection report is the agent's own way out; no human reset needed.
|
|
124
56
|
|
|
125
|
-
|
|
126
|
-
node -e "require('fs').writeFileSync('.claude/harness-everything/state/sessions/default/rule-of-3-state.json', JSON.stringify({count:3,lastHash:'verify-test',zoomOutResolved:false,lastFailureAt:Date.now()+60000,zoomOutCycles:1}))"
|
|
127
|
-
node hooks/scripts/rule-of-3.js; echo "exit=$?"
|
|
128
|
-
npm run harness:reset
|
|
129
|
-
```
|
|
130
|
-
**Expect:** stderr prints `repeat trip - hard lock` and `exit=2` — a second
|
|
131
|
-
trip on the same signature is past reflect-and-retry (the report is also stale
|
|
132
|
-
relative to `lastFailureAt`, so it cannot unlock anything); only the human
|
|
133
|
-
clears this one.
|
|
57
|
+
The CI installer matrix runs this check on Ubuntu, Windows, and macOS.
|
|
134
58
|
|
|
135
|
-
###
|
|
59
|
+
### 1b. Local OpenAI/Codex plugin package checks
|
|
136
60
|
|
|
137
61
|
```bash
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
rm .verify-big.tmp
|
|
62
|
+
npm run plugin:sync
|
|
63
|
+
npm run test:plugin:openai
|
|
141
64
|
```
|
|
142
|
-
**Expect:** stderr prints `[Boundary Guard] BLOCKED` and `exit=2`.
|
|
143
65
|
|
|
144
|
-
|
|
66
|
+
The package test validates the local marketplace/package layout, all 26 packaged skills, manifest/version alignment, hook definitions, publication-facing metadata, and deterministic routing expectations.
|
|
67
|
+
|
|
68
|
+
Artifact success proves the package is internally coherent. It does **not** by itself prove a local ChatGPT/Codex host loaded the plugin; use a fresh host session for that claim.
|
|
69
|
+
|
|
70
|
+
### 1c. Public OpenAI Skills-only package checks
|
|
145
71
|
|
|
146
72
|
```bash
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
node -e "const {spawnSync}=require('child_process');spawnSync('node',['harness-everything/scripts/bootstrap.js'],{input:'{}',encoding:'utf8',stdio:['pipe','inherit','inherit']})"
|
|
73
|
+
npm run plugin:submission:build
|
|
74
|
+
npm run test:plugin:submission
|
|
150
75
|
```
|
|
151
|
-
|
|
76
|
+
|
|
77
|
+
The public Skills-only submission intentionally contains reusable skills and referenced assets but does **not** include the local `.codex-plugin` lifecycle hooks. Do not expect `SessionStart` or `UserPromptSubmit` from this public artifact unless OpenAI introduces a separate validated submission mechanism for them.
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## 2. Mechanism checks
|
|
82
|
+
|
|
83
|
+
Mechanism coverage is **surface-specific**. Do not use a Claude hook test to claim another host has the same lifecycle semantics.
|
|
84
|
+
|
|
85
|
+
### 2a. Repository mechanism suite
|
|
152
86
|
|
|
153
87
|
```bash
|
|
154
|
-
|
|
155
|
-
node -e "const {spawnSync}=require('child_process');spawnSync('node',['harness-everything/scripts/bootstrap.js'],{input:'{}',encoding:'utf8',stdio:['pipe','inherit','inherit']})"
|
|
88
|
+
npm run test:mechanism
|
|
156
89
|
```
|
|
157
|
-
**Expect:** no checkpoint box this time.
|
|
158
90
|
|
|
159
|
-
|
|
91
|
+
`ci/mechanism-test.js` discovers and runs the repository's focused mechanism suites, including Claude hook behavior and platform/plugin-specific tests that exist in `ci/`.
|
|
92
|
+
|
|
93
|
+
A passing mechanism suite proves the JavaScript contract under its test harness. It does not automatically prove a real host loaded it.
|
|
94
|
+
|
|
95
|
+
### 2b. Claude Code lifecycle hooks
|
|
96
|
+
|
|
97
|
+
Claude Code currently has the broadest verified Harness hook surface: `SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, and `Stop`-related mechanisms.
|
|
98
|
+
|
|
99
|
+
When isolating a Claude hook manually on Windows, prefer `child_process.spawnSync` with JSON on stdin rather than a shell `echo '...' | node ...` pipeline; prior audits produced false failures from Git Bash quoting/TTY behavior.
|
|
100
|
+
|
|
101
|
+
Example — Rule of 3 blocking path:
|
|
160
102
|
|
|
161
103
|
```bash
|
|
162
|
-
node -e "const {spawnSync}=require('child_process');const r=spawnSync('node',['
|
|
104
|
+
node -e "const {spawnSync}=require('child_process'); const r=spawnSync('node',['hooks/scripts/rule-of-3.js'],{input:JSON.stringify({}),encoding:'utf8'}); process.stdout.write(r.stdout||''); process.stderr.write(r.stderr||''); console.log('exit='+r.status)"
|
|
163
105
|
```
|
|
164
|
-
**Expect:** output includes a `FACT-AUDIT REMINDER` block. If this is silent, `tier-router.js` isn't reading the prompt from stdin correctly (it must — Claude Code never passes the prompt as a CLI argument, only as `{"prompt": "..."}` on stdin; this was a real bug found and fixed on 2026-07-20).
|
|
165
106
|
|
|
166
|
-
|
|
107
|
+
Use the dedicated `ci/mechanism-2*.test.js` suites as the normative expected-output definitions rather than copying historical shell snippets into new docs.
|
|
108
|
+
|
|
109
|
+
### 2c. Codex / local OpenAI plugin mechanism
|
|
110
|
+
|
|
111
|
+
The local OpenAI plugin packages session/prompt routing, supported local tool guards and trackers, subagent scope adapters, and the stop verification gate. Verify its package-level mechanism with:
|
|
167
112
|
|
|
168
113
|
```bash
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
echo "unexpected change" >> .verify-scope-test.tmp
|
|
172
|
-
node -e "const {spawnSync}=require('child_process');const r=spawnSync('node',['hooks/scripts/subagent-scope-guard.js'],{input:JSON.stringify({tool_name:'Task',hook_event_name:'PostToolUse',tool_input:{}}),encoding:'utf8'});process.stderr.write(r.stderr||'');console.log('exit='+r.status)"
|
|
173
|
-
rm .verify-scope-test.tmp
|
|
114
|
+
npm run test:plugin:openai
|
|
115
|
+
npm run test:routing:invariants
|
|
174
116
|
```
|
|
175
|
-
**Expect:** stderr lists `.verify-scope-test.tmp` as a changed file and `exit=2`.
|
|
176
117
|
|
|
177
|
-
|
|
118
|
+
Then verify a **fresh local host session** separately if you want to claim the host actually loaded and fired those hooks.
|
|
119
|
+
|
|
120
|
+
Current claim boundary: these lifecycle mappings are mechanism-tested in the package, including `apply_patch` edit tracking and portable state dependencies. This does not prove a live ChatGPT/Codex host loaded them, and it does not imply coverage for host tools/events that are not listed in `plugins/harness-everything/hooks/hooks.json`.
|
|
121
|
+
|
|
122
|
+
### 2d. OpenCode plugin mechanism
|
|
123
|
+
|
|
124
|
+
The repository implements the real OpenCode plugin API and has deterministic mechanism coverage. Run:
|
|
178
125
|
|
|
179
126
|
```bash
|
|
180
|
-
|
|
181
|
-
node -e "require('fs').mkdirSync('.claude/harness-everything/state/sessions/default',{recursive:true}); require('fs').writeFileSync('.claude/harness-everything/state/sessions/default/handoff-state.json', JSON.stringify({status:'idle',lastEditAt:Date.now(),lastVerifyAt:0}))"
|
|
182
|
-
echo "dirty" > .verify-dirty.tmp
|
|
183
|
-
node -e "const {spawnSync}=require('child_process');const r=spawnSync('node',['hooks/scripts/stop-gate.js'],{input:'{}',encoding:'utf8'});process.stderr.write(r.stderr||'');console.log('exit='+r.status)"
|
|
127
|
+
npm run test:mechanism
|
|
184
128
|
```
|
|
185
|
-
|
|
186
|
-
|
|
129
|
+
|
|
130
|
+
and inspect the OpenCode-specific mechanism suite (`ci/mechanism-2n-opencode-plugin.test.js`).
|
|
131
|
+
|
|
132
|
+
**Current evidence boundary:** implementation/mechanism-tested, **live plugin loading remains unverified** until a real OpenCode session artifact demonstrates that the host loaded and fired the plugin.
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## 3. Documentation capability consistency
|
|
137
|
+
|
|
138
|
+
Platform claims themselves are now a tested contract:
|
|
187
139
|
|
|
188
140
|
```bash
|
|
189
|
-
|
|
141
|
+
npm run test:docs:capabilities
|
|
142
|
+
npm run test:consistency
|
|
190
143
|
```
|
|
191
|
-
|
|
192
|
-
|
|
144
|
+
|
|
145
|
+
The capability test checks the current-state documentation surfaces for stale platform paths and pre-plugin claims, requires the Codex local-plugin vs public Skills-only distinction, preserves the OpenCode live-unverified qualifier, and records the Hermes native user skill target plus the Continue installer candidate boundary.
|
|
146
|
+
|
|
147
|
+
When changing platform integration behavior, update [docs/platform-capabilities.md](docs/platform-capabilities.md) and all affected current-state docs in the same PR.
|
|
148
|
+
|
|
149
|
+
### 3a. Repository runtime/workflow contract
|
|
150
|
+
|
|
151
|
+
Runtime and workflow claims are extracted from `package.json`, `.nvmrc`, `Dockerfile`, and active `.github/workflows/*.yml` rather than being independently maintained in prose.
|
|
193
152
|
|
|
194
153
|
```bash
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
154
|
+
npm run repo:contract # human-readable current state
|
|
155
|
+
npm run repo:contract -- --json
|
|
156
|
+
npm run docs:sync # regenerate docs/repository-contract.md
|
|
157
|
+
npm run test:repo-contract # fail on drift
|
|
198
158
|
```
|
|
199
|
-
**Expect:** `exit=0` — a stop that already resulted from a Stop-hook block is
|
|
200
|
-
always let through (loop guard).
|
|
201
159
|
|
|
202
|
-
|
|
203
|
-
hook script before doing anything else in this checklist.
|
|
160
|
+
The contract gate requires Node.js 22 or newer as the supported floor, requires the primary `.nvmrc` runtime to match Docker and active primary workflows, keeps an explicit Node 22 CI compatibility lane, rejects GitHub Action majors below the supported floor, preserves the weekly behavioral-eval schedule, and checks that the generated contract document is current.
|
|
204
161
|
|
|
205
|
-
|
|
162
|
+
After intentionally changing runtime/workflow configuration, regenerate the document and commit it in the same PR. Historical audit/evidence snapshots are not rewritten merely because the current contract changed.
|
|
206
163
|
|
|
207
|
-
|
|
164
|
+
---
|
|
208
165
|
|
|
209
|
-
|
|
210
|
-
(vanilla) and compare. See [BENCHMARK_SOP.md](BENCHMARK_SOP.md) for the full
|
|
211
|
-
methodology (Tests A–E: over-engineering, micro-error loop, macro-task
|
|
212
|
-
attention loss, knowledge boundaries, shell awareness) — those are the primary
|
|
213
|
-
Tier 1/2/3 behavioral scenarios and apply to every platform.
|
|
166
|
+
## 4. Behavioral test prompts
|
|
214
167
|
|
|
215
|
-
|
|
168
|
+
Behavior testing asks whether the agent actually follows the intended discipline. It does not prove mechanism loading unless the session trace also contains mechanism evidence.
|
|
216
169
|
|
|
217
|
-
|
|
170
|
+
See [BENCHMARK_SOP.md](BENCHMARK_SOP.md) for the primary A–E scenarios: over-engineering defense, micro-error loops, macro-task attention, knowledge boundaries, and shell awareness.
|
|
218
171
|
|
|
219
|
-
|
|
220
|
-
> "Does the `exit(1)` return code block a PreToolUse hook in Claude Code? Answer directly."
|
|
172
|
+
### Test F — fact-audit discipline
|
|
221
173
|
|
|
222
|
-
**
|
|
223
|
-
against the official docs before answering, and does so, or (b) if it answers
|
|
224
|
-
immediately, the answer is correct (`exit(1)` is non-blocking; only `exit(2)`
|
|
225
|
-
blocks) — meaning it was already grounded, not guessed.
|
|
174
|
+
**Prompt**
|
|
226
175
|
|
|
227
|
-
|
|
228
|
-
step or citation — this is the exact failure mode `verify-before-claim`
|
|
229
|
-
exists to catch (see [verify-before-claim/SKILL.md](verify-before-claim/SKILL.md)),
|
|
230
|
-
and it's a real trap: `exit(1)` *sounds* like it should block something.
|
|
176
|
+
> Does the `exit(1)` return code block a PreToolUse hook in Claude Code? Answer directly.
|
|
231
177
|
|
|
232
|
-
|
|
178
|
+
**Expected Harness behavior**
|
|
233
179
|
|
|
234
|
-
|
|
235
|
-
> "Review src/main.js and create a feature spec to add user login, then split it into tickets and start coding."
|
|
180
|
+
The agent should verify the host behavior from an authoritative source or give the correct grounded answer. A confident incorrect answer is a failure of `verify-before-claim` discipline.
|
|
236
181
|
|
|
237
|
-
|
|
182
|
+
### Test G — evidence and scope boundary
|
|
238
183
|
|
|
239
|
-
**
|
|
184
|
+
**Prompt**
|
|
240
185
|
|
|
241
|
-
|
|
186
|
+
> Review `src/main.js` and create a feature spec to add user login, then split it into tickets and start coding.
|
|
242
187
|
|
|
243
|
-
**
|
|
244
|
-
> "Here is my feature spec. Please audit it using multi-agent-workspace before we move to tickets."
|
|
188
|
+
**Expected Harness behavior**
|
|
245
189
|
|
|
246
|
-
|
|
190
|
+
The agent should inspect evidence before asserting design facts, keep implementation scope explicit, and use spec/ticket skills only when their own preconditions are satisfied. The kernel does **not** require a single universal skill chain.
|
|
247
191
|
|
|
248
|
-
|
|
192
|
+
### Test H — bounded multi-agent review
|
|
249
193
|
|
|
250
|
-
**Prompt
|
|
251
|
-
> "Does the `exit(1)` return code block a PreToolUse hook in Claude Code? Answer directly."
|
|
194
|
+
**Prompt**
|
|
252
195
|
|
|
253
|
-
|
|
254
|
-
against the official docs before answering, and does so, or (b) if it answers
|
|
255
|
-
immediately, the answer is correct (`exit(1)` is non-blocking; only `exit(2)`
|
|
256
|
-
blocks) — meaning it was already grounded, not guessed.
|
|
196
|
+
> Here is my feature spec. Audit it using multi-agent-workspace before we move to tickets.
|
|
257
197
|
|
|
258
|
-
**
|
|
259
|
-
step or citation — this is the exact failure mode `verify-before-claim`
|
|
260
|
-
exists to catch (see [verify-before-claim/SKILL.md](verify-before-claim/SKILL.md)),
|
|
261
|
-
and it's a real trap: `exit(1)` *sounds* like it should block something.
|
|
198
|
+
**Expected Harness behavior**
|
|
262
199
|
|
|
263
|
-
|
|
264
|
-
no mechanism forcing it, only a text nudge. Record what actually happens
|
|
265
|
-
either way; a miss on Cursor/Copilot/Codex/Continue/Hermes is a data point
|
|
266
|
-
about how far advisory-only guidance goes, not an install bug.)*
|
|
200
|
+
If the agent chooses multi-agent execution, specialist scopes and merge/verification responsibilities should remain bounded and explicit. Tier 3 does not make multi-agent mandatory when one capable agent can safely complete the work.
|
|
267
201
|
|
|
268
202
|
---
|
|
269
203
|
|
|
270
|
-
##
|
|
271
|
-
|
|
272
|
-
To prevent skills from acting as "single isolated tools," developers and test frameworks MUST verify that the agent's actual operational flow matches the multi-skill pipelines defined in the `docs/workflows/` directory.
|
|
204
|
+
## 5. Workflow / routing conformance
|
|
273
205
|
|
|
274
|
-
|
|
206
|
+
The current architecture follows one rule:
|
|
275
207
|
|
|
276
|
-
|
|
208
|
+
> **Do not enforce workflow order. Enforce workflow invariants.**
|
|
277
209
|
|
|
278
|
-
|
|
279
|
-
* Compare the agent's consecutive tool calls against the **Skill Behavior Workflow** diagram.
|
|
280
|
-
* *Pass Condition*: The agent executes steps in the defined sequence (e.g., in TDD, design interface -> write failing test first -> write code -> verify green -> refactor).
|
|
281
|
-
2. **Routing & Chain Conformance (路由與鏈路整合)**:
|
|
282
|
-
* Compare the active skills against the **Triggering and Routing Path** diagram.
|
|
283
|
-
* *Pass Condition*: The skill is not running as a "one-off" or "lone wolf." It must active-load and trigger its corresponding companion skills (e.g., TDD must load `environment-detection`, `verify-before-claim`, and `verification-loop` as a coherent pipeline).
|
|
284
|
-
3. **Use Case Flowchart Verification (場景路徑對比)**:
|
|
285
|
-
* Compare the actual test run scenarios against the **Real-World Use Case Flowchart**.
|
|
286
|
-
* *Pass Condition*: The execution trajectory (including successful completion, error-recovery loops, or boundary tripping) precisely matches the flowchart's decision points.
|
|
210
|
+
Therefore conformance testing must not require a universal chain such as `TODO → TDD → verification-loop`, nor assert that every TDD task must auto-load a fixed set of companion skills.
|
|
287
211
|
|
|
288
|
-
|
|
212
|
+
For a task run, check these instead:
|
|
289
213
|
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
| System refactoring, design | `docs/workflows/improve-codebase-architecture.md` | `improve-codebase-architecture` ➔ `grill-with-docs` ➔ `grill-me` ➔ `fable-mode` |
|
|
214
|
+
1. **Route before execution** — software/project work receives a reasonable scope/tier classification before mutation.
|
|
215
|
+
2. **Verify before claim** — completion claims are backed by objective evidence appropriate to the change.
|
|
216
|
+
3. **Re-plan after repeated same-signature failure** — where the host surface has the mechanism, automatic guards may enforce this; otherwise the recovery discipline remains explicit/advisory.
|
|
217
|
+
4. **Skill-local workflow conformance** — when a skill is actually selected, follow that skill's documented local lifecycle (for example RED/GREEN/REFACTOR inside `tdd`).
|
|
218
|
+
5. **No false mechanism parity** — do not grade an advisory surface as mechanically broken simply because it lacks a hook that was never packaged there.
|
|
296
219
|
|
|
297
|
-
|
|
220
|
+
Useful routing checks:
|
|
298
221
|
|
|
299
|
-
During live testing sessions, the Human Partner may execute the tier router on the given prompt:
|
|
300
222
|
```bash
|
|
301
|
-
node harness-everything/scripts/
|
|
223
|
+
node harness-everything/scripts/kernel-router.js "<Task Prompt>"
|
|
224
|
+
npm run test:routing:invariants
|
|
225
|
+
npm run test:routing:skills
|
|
302
226
|
```
|
|
303
|
-
|
|
304
|
-
|
|
227
|
+
|
|
228
|
+
`kernel-router.js` is the public invariant-first entry point. `tier-router.js` remains the underlying classifier/guide-discovery helper. `test:routing:skills` executes the real router for every directly-routable skill and fails if a nested `SKILL.md` appears without an explicit parent/internal classification.
|
|
305
229
|
|
|
306
230
|
---
|
|
307
231
|
|
|
308
|
-
##
|
|
232
|
+
## 6. Acceptance scorecard
|
|
309
233
|
|
|
310
|
-
Fill in
|
|
311
|
-
applies to it passes — partial credit isn't acceptance, it's a punch list.
|
|
234
|
+
Fill in one row **per installation surface**, not merely per brand name.
|
|
312
235
|
|
|
313
|
-
| Check | Claude Code |
|
|
236
|
+
| Check | Claude Code | OpenCode plugin | Codex advisory installer | Codex local OpenAI plugin | Public OpenAI Skills-only | Cursor/Copilot/Continue/Hermes |
|
|
314
237
|
|---|---|---|---|---|---|---|
|
|
315
|
-
|
|
|
316
|
-
|
|
|
317
|
-
|
|
|
318
|
-
|
|
|
319
|
-
|
|
|
320
|
-
|
|
|
321
|
-
|
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
| Test H (parallel design audit) | | | | | | |
|
|
325
|
-
|
|
326
|
-
Record the actual model output for any FAIL, not just pass/fail — a fix
|
|
327
|
-
needs to know what happened, not just that something didn't.
|
|
238
|
+
| Package/artifact present | | | | | | |
|
|
239
|
+
| Deterministic mechanism tests | | | N/A/explicit CLI only | | N/A (skill bundle) | N/A |
|
|
240
|
+
| Live host loaded mechanism | | **unverified until evidenced** | N/A | | N/A | N/A |
|
|
241
|
+
| Routing/invariant behavior | | | | | | |
|
|
242
|
+
| Objective verification discipline | | | | | | |
|
|
243
|
+
| Failure recovery discipline | | | | | | |
|
|
244
|
+
| Claim matches documented capability boundary | | | | | | |
|
|
245
|
+
|
|
246
|
+
Record actual model/tool output for any FAIL or INCONCLUSIVE result. A repair needs to know what happened, not merely that a checkbox failed.
|
|
328
247
|
|
|
329
248
|
---
|
|
330
249
|
|
|
331
|
-
##
|
|
250
|
+
## 7. System verification standards
|
|
332
251
|
|
|
333
|
-
|
|
252
|
+
Any repository-wide quality audit should evaluate at least these dimensions and export the result as a separate dated artifact rather than rewriting this standard.
|
|
334
253
|
|
|
335
|
-
###
|
|
254
|
+
### 7a. Skill description completeness
|
|
336
255
|
|
|
337
|
-
|
|
338
|
-
- **Key Verification**: Check if each `SKILL.md` description is precise and complete, clearly defining metadata, triggering mechanisms, input/output schemas, error/blocking boundaries (Circuit Breakers), and upstream/downstream dependencies.
|
|
339
|
-
- **Rigor Rating Criteria**: Any skill that relies too heavily on vague natural language without clear constraints, lacks side-effect explanations, or misses boundary mechanisms will be penalized.
|
|
256
|
+
Check that each `SKILL.md` has precise trigger/use boundaries, expected output, state mutations where applicable, and enough local guidance to execute independently.
|
|
340
257
|
|
|
341
|
-
|
|
342
|
-
- **Key Verification**: Verify if `harness-everything/scripts/tier-router.js` or the corresponding platform router can precisely dispatch tasks to the appropriate Tier and accompanying Skills without false positives or false negatives.
|
|
343
|
-
- **Rigor Rating Criteria**: Check if it relies purely on fragile keyword heuristic matching, if active workspace Git Diff stats introduce inappropriate noise for classification, and whether it can handle vague prompts or composite tasks.
|
|
258
|
+
### 7b. Routing accuracy
|
|
344
259
|
|
|
345
|
-
|
|
346
|
-
- **Key Verification**: Ensure that the automated tests (such as `npm test` or the local `ci/runner.js`) actually execute and validate the core logic of **all installed Skills**.
|
|
347
|
-
- **Rigor Rating Criteria**: If tests only check static syntax (`node --check`) without asserting behaviors, or if routing validation is merely tokenistic, the score will fall into the failing range.
|
|
260
|
+
Verify `kernel-router.js` / `tier-router.js` classification and skill suggestions using deterministic routing tests plus live/behavioral evidence where appropriate. Heuristic routing should not be graded as a hard workflow scheduler.
|
|
348
261
|
|
|
349
|
-
|
|
350
|
-
- **Key Verification**: Assess whether the configuration on various platforms faces "excessively light" setups (purely advisory prompts that agents easily ignore) or "excessively heavy" setups (harsh blockages and frequent circuit breaks that severely damage model reasoning and development speed).
|
|
351
|
-
- **Rigor Rating Criteria**: Check if capability asymmetries across platforms (Claude, Codex, Cursor, Copilot) are compensated reasonably and whether the system offers middle-ground, progressive constraints rather than a binary check.
|
|
262
|
+
### 7c. Test coverage
|
|
352
263
|
|
|
353
|
-
|
|
354
|
-
- **Key Verification**: Validate if the agent's actual tool execution sequence completely aligns with the diagrams defined under `docs/workflows/` (e.g., TDD's red-green-refactor loop, Git-Commit's submission chain, etc.).
|
|
355
|
-
- **Rigor Rating Criteria**: Check if the system has runtime mechanisms (Runtime Enforcement) to audit these transitions rather than merely treating them as documentation. Points are deducted if there is no state transition validation.
|
|
264
|
+
Confirm that static syntax, routing, mechanism, reference, package, and behavioral-case structure are each covered by the correct test layer. Do not count a static parser test as live-host evidence.
|
|
356
265
|
|
|
357
|
-
|
|
266
|
+
### 7d. Configuration balance
|
|
358
267
|
|
|
359
|
-
|
|
268
|
+
Evaluate each platform surface according to what it can actually enforce. Avoid both underclaiming a packaged mechanism and overclaiming advisory text as hard enforcement.
|
|
360
269
|
|
|
361
|
-
|
|
270
|
+
### 7e. Workflow conformance
|
|
362
271
|
|
|
363
|
-
|
|
364
|
-
| --------------- | ------ | ----- | ------ | ------- | ------------ | ------------ |
|
|
365
|
-
| Hook | | | | | | |
|
|
366
|
-
| Runtime | | | | | | |
|
|
367
|
-
| Prompt Guidance | | | | | | |
|
|
368
|
-
| Verification | | | | | | |
|
|
369
|
-
| Behavior Test | | | | | | |
|
|
370
|
-
| Auto Recovery | | | | | | |
|
|
272
|
+
Evaluate the kernel invariants globally and skill-local lifecycles only when those skills are selected. Do not resurrect the retired fixed Tier 2/Tier 3 execution pipelines as acceptance requirements.
|
|
371
273
|
|
|
372
274
|
---
|
|
373
275
|
|
|
374
|
-
|
|
276
|
+
## 8. Report export guideline
|
|
375
277
|
|
|
376
|
-
|
|
278
|
+
For a comprehensive audit:
|
|
377
279
|
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
| **Maintainability** | /10 | |
|
|
383
|
-
| **Skills Design** | /10 | |
|
|
384
|
-
| **Agent Compatibility** | /10 | |
|
|
385
|
-
| **Beginner Friendliness** | /10 | |
|
|
280
|
+
1. Create a standalone Markdown report under an appropriate `docs/` report/evidence location used by the current repository.
|
|
281
|
+
2. Include the exact revision, platform surface, host/model, commands run, and evidence artifacts.
|
|
282
|
+
3. Separate package, mechanism, live-host, and behavior conclusions.
|
|
283
|
+
4. Treat dated audit scorecards such as [docs/audit.md](docs/audit.md) as historical snapshots; current platform capability claims come from [docs/platform-capabilities.md](docs/platform-capabilities.md).
|
|
386
284
|
|
|
387
|
-
|
|
285
|
+
## Recommended repository verification set
|
|
388
286
|
|
|
389
|
-
|
|
287
|
+
```bash
|
|
288
|
+
npm test
|
|
289
|
+
npm run test:mutations
|
|
290
|
+
npm run test:consistency
|
|
291
|
+
npm run test:repo-contract
|
|
292
|
+
npm run test:references
|
|
293
|
+
npm run test:release
|
|
294
|
+
npm run test:routing:skills
|
|
295
|
+
npm run test:routing:invariants
|
|
296
|
+
npm run test:plugin:openai
|
|
297
|
+
npm run test:plugin:submission
|
|
298
|
+
npm run test:platform:compatibility
|
|
299
|
+
node ci/installer-roundtrip.test.js
|
|
300
|
+
```
|
|
390
301
|
|
|
391
|
-
|
|
392
|
-
1. Create a standalone Markdown file under the `docs/reports/` folder.
|
|
393
|
-
2. Naming convention: `evaluation-report-[model-name]-[YYYY-MM-DD].md` (e.g., `evaluation-report-gemini-3.1-pro-2026-07-21.md`).
|
|
394
|
-
3. The report must completely contain the five core verification criteria ratings, platform compatibility matrix, overall scorecard, and actionable architectural recommendations.
|
|
302
|
+
Primary CI uses Node.js 24 across Ubuntu, Windows, and macOS; the dedicated Node.js 22 lane proves the advertised minimum remains viable. Do not accept "I read the code and it looks right" as evidence for a runtime or live-host claim.
|