harness-everything 0.3.6 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plugins/marketplace.json +20 -0
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +8 -7
- package/.github/workflows/behavioral-evals.yml +3 -3
- package/.github/workflows/ci.yml +49 -28
- package/.github/workflows/release.yml +52 -27
- package/.nvmrc +1 -0
- package/.releaserc.json +113 -0
- package/AGENTS.md +44 -24
- package/BENCHMARK_SOP.md +38 -18
- package/CHANGELOG.md +196 -3
- package/CONTRIBUTING.md +37 -9
- package/Dockerfile +1 -1
- package/PRIVACY.md +25 -0
- package/README.md +126 -91
- package/RELEASING.md +93 -0
- package/SUPPORT.md +27 -0
- package/TERMS.md +21 -0
- package/VERIFICATION.md +192 -284
- package/behavioral-evals/README.md +21 -0
- package/behavioral-evals/case-validator.js +194 -0
- package/behavioral-evals/cases/baseline-environment-detection.yaml +21 -0
- package/behavioral-evals/cases/baseline-eval-harness.yaml +27 -0
- package/behavioral-evals/cases/baseline-fable-discipline.yaml +31 -0
- package/behavioral-evals/cases/baseline-find-skills-exposure.yaml +49 -0
- package/behavioral-evals/cases/baseline-find-skills.yaml +21 -0
- package/behavioral-evals/cases/baseline-git-commit.yaml +2 -2
- package/behavioral-evals/cases/baseline-install-cognitive-os.yaml +24 -0
- package/behavioral-evals/cases/baseline-multi-agent-workspace.yaml +28 -0
- package/behavioral-evals/cases/baseline-rewrite-commits.yaml +34 -0
- package/behavioral-evals/cases/baseline-self-evolve.yaml +31 -0
- package/behavioral-evals/cases/baseline-skill-creator.yaml +20 -0
- package/behavioral-evals/cases/baseline-skill-style.yaml +21 -0
- package/behavioral-evals/cases/baseline-to-spec.yaml +34 -0
- package/behavioral-evals/cases/baseline-to-tickets.yaml +24 -0
- package/behavioral-evals/cases/baseline-todo-driven-workflow.yaml +32 -0
- package/behavioral-evals/cases/baseline-using-git-worktrees.yaml +15 -0
- package/behavioral-evals/cases/breaker-zoom-out-after-3.yaml +2 -2
- package/behavioral-evals/cases/pressure-complexity-skip.yaml +2 -2
- package/behavioral-evals/cases/pressure-scope-bypass.yaml +7 -2
- package/behavioral-evals/cases/pressure-sunk-cost-retry.yaml +7 -4
- package/behavioral-evals/cases/scope-discipline.yaml +2 -1
- package/behavioral-evals/cases/verify-before-done.yaml +7 -5
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/case.yaml +34 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/historical-case.yaml +34 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/historical-case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/historical-case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/historical-case.yaml +18 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/live-run-blocker.md +16 -0
- package/behavioral-evals/evidence/2026-09-07/manifest.json +88 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/historical-case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/historical-case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/case.yaml +35 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/historical-case.yaml +35 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/case.yaml +30 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/historical-case.yaml +23 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/triage-matrix.json +426 -0
- package/behavioral-evals/evidence/2026-09-07/triage-matrix.md +24 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/case.yaml +19 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/historical-case.yaml +18 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/historical-case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/result.sanitized.json +480 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/transcript.sanitized.jsonl +95 -0
- package/behavioral-evals/evidence-tool.js +443 -0
- package/behavioral-evals/fixtures/claude-denied.jsonl +3 -0
- package/behavioral-evals/fixtures/claude-legacy.json +1 -0
- package/behavioral-evals/fixtures/claude-oral-only.jsonl +2 -0
- package/behavioral-evals/fixtures/claude-success.jsonl +5 -0
- package/behavioral-evals/fixtures/claude-truncated.jsonl +3 -0
- package/behavioral-evals/fixtures/opencode-success.jsonl +6 -0
- package/behavioral-evals/results/2026-08-31-baseline-add-feature.json +20 -0
- package/behavioral-evals/results/2026-09-03-baseline-environment-detection-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-eval-harness-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-fable-discipline-pair.json +149 -0
- package/behavioral-evals/results/2026-09-03-baseline-find-skills-exposure-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-find-skills-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-install-cognitive-os-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-multi-agent-workspace-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-rewrite-commits-pair.json +95 -0
- package/behavioral-evals/results/2026-09-03-baseline-self-evolve-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-skill-creator-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-skill-style-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-to-spec-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-to-tickets-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-todo-driven-workflow-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-using-git-worktrees-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-summary-both.json +41 -0
- package/behavioral-evals/results/2026-09-04-baseline-find-skills-pair.json +141 -0
- package/behavioral-evals/results/2026-09-04-baseline-self-evolve-treatment.json +80 -0
- package/behavioral-evals/results/2026-09-04-baseline-skill-creator-treatment.json +80 -0
- package/behavioral-evals/results/2026-09-04-summary-both.json +41 -0
- package/behavioral-evals/run-with-plugin.js +15 -31
- package/behavioral-evals/run.js +446 -105
- package/behavioral-evals/transcript-parser.js +591 -0
- package/bin/cli.js +24 -12
- package/ci/ab-test-cases/grill-me-adversarial.yaml +14 -4
- package/ci/ab-test-cases/tdd-test-first.yaml +13 -6
- package/ci/ab-test-cases/verification-loop-gates.yaml +20 -7
- package/ci/ab-test-cases/verify-before-claim-cites.yaml +7 -6
- package/ci/ab-test-harness.js +113 -38
- package/ci/action-gate-platform-doc.test.js +31 -0
- package/ci/action-gate.test.js +260 -0
- package/ci/cases/case2.json +2 -2
- package/ci/consistency-check.js +51 -5
- package/ci/description-collision.js +15 -6
- package/ci/disclosure-check.js +225 -0
- package/ci/doc-capability-consistency.test.js +180 -0
- package/ci/fixtures/phase4/parallel-disjoint.stages.json +32 -0
- package/ci/fixtures/phase4/parallel-read-only.stages.json +32 -0
- package/ci/installer-roundtrip.test.js +159 -0
- package/ci/invariant-routing.test.js +74 -0
- package/ci/issue99-runtime.test.js +218 -0
- package/ci/mechanism-2aa-codex-action-gate-policy.test.js +117 -0
- package/ci/mechanism-2ab-openai-plugin-sync-integrity.test.js +80 -0
- package/ci/mechanism-2j-behavioral-runner.test.js +43 -1
- package/ci/mechanism-2k-reference-check.test.js +57 -2
- package/ci/mechanism-2l-multi-agent-workspace.test.js +202 -6
- package/ci/mechanism-2m-self-repo-guard.test.js +65 -0
- package/ci/mechanism-2n-opencode-plugin.test.js +461 -0
- package/ci/mechanism-2o-state-home-isolation.test.js +183 -0
- package/ci/mechanism-2p-installer-links.test.js +183 -0
- package/ci/mechanism-2q-installer-global-shared-store.test.js +132 -0
- package/ci/mechanism-2r-uninstall-flags.test.js +148 -0
- package/ci/mechanism-2s-tdd-quality.test.js +203 -0
- package/ci/mechanism-2t-disclosure.test.js +90 -0
- package/ci/mechanism-2t-evidence-triage.test.js +200 -0
- package/ci/mechanism-2t-issue-20.test.js +75 -0
- package/ci/mechanism-2t-platform-skill-targets.test.js +164 -0
- package/ci/mechanism-2t-transcript-parser.test.js +179 -0
- package/ci/mechanism-2t-workspace-identity.test.js +200 -0
- package/ci/mechanism-2u-issue-42-regressions.test.js +239 -0
- package/ci/mechanism-2u-issue-44.test.js +35 -0
- package/ci/mechanism-2v-verifier-boundaries.test.js +138 -0
- package/ci/mechanism-2w-platform-compatibility.test.js +148 -0
- package/ci/mechanism-2x-release-automation.test.js +157 -0
- package/ci/mechanism-2x-release-commit-message.test.js +66 -0
- package/ci/mechanism-2y-release-tooling-compat.test.js +47 -0
- package/ci/mechanism-2z-claude-plugin-manifest.test.js +29 -0
- package/ci/mutation-check.js +53 -0
- package/ci/mutation-loader.js +57 -0
- package/ci/negative-controls.js +76 -46
- package/ci/openai-plugin-package.test.js +182 -0
- package/ci/openai-plugin-runtime.test.js +175 -0
- package/ci/openai-plugin-submission.test.js +101 -0
- package/ci/phase4-ensemble-memory.test.js +307 -0
- package/ci/reference-check.js +73 -25
- package/ci/release-consistency-check.js +24 -1
- package/ci/repository-contract.test.js +17 -0
- package/ci/router-phase2-safety.test.js +81 -0
- package/ci/router-workflow-plan.test.js +264 -0
- package/ci/routing-check.js +6 -5
- package/ci/runner.js +6 -5
- package/ci/skill-routing-check.js +48 -11
- package/ci/test-behavioral-case-validation.js +62 -0
- package/ci/test-helper.js +10 -0
- package/ci/workflow-plan-consumers-edge.test.js +72 -0
- package/ci/workflow-plan-consumers.test.js +276 -0
- package/docs/architecture.md +137 -57
- package/docs/audit.md +22 -20
- package/docs/codex-action-gate.md +90 -0
- package/docs/issue-20-rollup.md +47 -0
- package/docs/mechanism-first-skill-mesh.md +22 -20
- package/docs/openai-plugin.md +118 -0
- package/docs/philosophy.md +28 -17
- package/docs/platform-capabilities.md +73 -0
- package/docs/platform-compatibility.json +270 -0
- package/docs/reflection.md +37 -22
- package/docs/release-evidence.md +27 -0
- package/docs/repository-contract.md +27 -0
- package/docs/routing.md +157 -44
- package/docs/security-audit.md +38 -0
- package/docs/troubleshooting.md +94 -51
- package/docs/verifier-boundaries.md +45 -0
- package/docs/workflows/fable-mode.md +77 -21
- package/docs/workflows/find-skills.md +40 -0
- package/docs/workflows/grill-with-docs.md +2 -2
- package/docs/workflows/multi-agent-workspace.md +22 -10
- package/docs/workflows/self-evolve.md +33 -0
- package/docs/workflows/tdd.md +36 -0
- package/docs/workflows/todo-driven-workflow.md +33 -0
- package/environment-detection/SKILL.md +3 -3
- package/eval-harness/SKILL.md +7 -7
- package/eval-harness/scripts/evaluate.js +12 -10
- package/evals/evaluation-reports.md +14 -0
- package/evals/harness-everything/tasks/positive-tier1.yaml +1 -1
- package/evals/harness-everything/tasks/positive-tier2.yaml +1 -1
- package/evals/harness-everything/tasks/positive-tier3.yaml +1 -1
- package/evals/install-cognitive-os/tasks/positive-trigger.yaml +1 -1
- package/fable-discipline/SKILL.md +2 -2
- package/fable-mode/CONTRACT-FORMAT.md +147 -33
- package/fable-mode/SKILL.md +3 -3
- package/fable-mode/agents/fable-orchestrator.md +95 -35
- package/fable-mode/execution-guardrails/SKILL.md +2 -2
- package/fable-mode/fable-haiku/SKILL.md +3 -3
- package/fable-mode/fable-opus/SKILL.md +3 -3
- package/fable-mode/fable-sonnet/SKILL.md +3 -3
- package/fable-mode/scripts/ensemble-review.js +371 -0
- package/fable-mode/scripts/model-selector.js +134 -3
- package/fable-mode/scripts/workflow-plan-consumer.js +337 -0
- package/find-skills/SKILL.md +5 -5
- package/find-skills/references/discovery-flow.md +1 -1
- package/find-skills/scripts/use-skill.js +7 -1
- package/git-commit/SKILL.md +4 -11
- package/grill-me/SKILL.md +5 -5
- package/grill-with-docs/SKILL.md +7 -7
- package/grill-with-docs/references/session-playbook.md +6 -4
- package/harness-everything/SKILL.md +23 -22
- package/harness-everything/references/router-workflow-plan.md +128 -0
- package/harness-everything/references/triage-and-tiers.md +173 -96
- package/harness-everything/schemas/router-task-shape.schema.json +126 -0
- package/harness-everything/schemas/router-workflow-plan.schema.json +149 -0
- package/harness-everything/scripts/bootstrap.js +13 -4
- package/harness-everything/scripts/ensemble-policy.js +102 -0
- package/harness-everything/scripts/kernel-router.js +212 -0
- package/harness-everything/scripts/router-contract.js +612 -0
- package/harness-everything/scripts/self-heal.js +29 -20
- package/harness-everything/scripts/tier-router.js +346 -221
- package/hooks/hooks.json +60 -15
- package/hooks/scripts/action-gate-rules.json +77 -0
- package/hooks/scripts/action-gate.js +528 -0
- package/hooks/scripts/atomic-commit-check.js +5 -5
- package/hooks/scripts/codex-action-gate-post.js +46 -0
- package/hooks/scripts/codex-permission-request.js +140 -0
- package/hooks/scripts/context-compact.js +4 -3
- package/hooks/scripts/contract-test.js +118 -51
- package/hooks/scripts/lib/fable-contracts.js +121 -0
- package/hooks/scripts/lib/harness-state.js +164 -28
- package/hooks/scripts/lib/platforms/claude.js +45 -62
- package/hooks/scripts/lib/platforms/codex.js +35 -61
- package/hooks/scripts/lib/platforms/continue.js +7 -7
- package/hooks/scripts/lib/platforms/copilot.js +29 -53
- package/hooks/scripts/lib/platforms/hermes.js +71 -20
- package/hooks/scripts/openai-session-start.js +81 -0
- package/hooks/scripts/rule-of-3-tracker.js +9 -8
- package/hooks/scripts/rule-of-3.js +29 -21
- package/hooks/scripts/state-persist.js +5 -4
- package/hooks/scripts/stop-gate.js +2 -2
- package/hooks/scripts/subagent-scope-guard.js +144 -38
- package/hooks/scripts/subagent-start.js +19 -0
- package/hooks/scripts/subagent-stop.js +6 -0
- package/improve-codebase-architecture/SKILL.md +2 -4
- package/improve-codebase-architecture/references/README.md +16 -0
- package/install-cognitive-os/SKILL.md +28 -22
- package/multi-agent-workspace/SKILL.md +16 -15
- package/multi-agent-workspace/references/architecture-guide.md +13 -9
- package/multi-agent-workspace/references/orchestration.md +39 -22
- package/multi-agent-workspace/scripts/consume-workflow-plan.js +129 -0
- package/multi-agent-workspace/scripts/index_memory.js +79 -26
- package/multi-agent-workspace/scripts/path-boundary.js +59 -0
- package/multi-agent-workspace/scripts/project-docs-resolver.js +357 -0
- package/multi-agent-workspace/scripts/scaffold.js +258 -103
- package/multi-agent-workspace/templates/AGENTS.md +20 -0
- package/multi-agent-workspace/workflows/03-generation.md +8 -10
- package/multi-agent-workspace/workflows/04-launcher.md +5 -5
- package/opencode-plugin/README.md +114 -71
- package/opencode-plugin/hooks/verify.js +62 -12
- package/opencode-plugin/index.mjs +547 -0
- package/opencode-plugin/plugin.json +4 -12
- package/package.json +20 -4
- package/plugins/harness-everything/.codex-plugin/plugin.json +39 -0
- package/plugins/harness-everything/hooks/hooks.json +197 -0
- package/plugins/harness-everything/hooks/scripts/action-gate-rules.json +77 -0
- package/plugins/harness-everything/hooks/scripts/action-gate.js +528 -0
- package/plugins/harness-everything/hooks/scripts/atomic-commit-check.js +68 -0
- package/plugins/harness-everything/hooks/scripts/boundary-guard.js +55 -0
- package/plugins/harness-everything/hooks/scripts/codex-action-gate-post.js +46 -0
- package/plugins/harness-everything/hooks/scripts/codex-permission-request.js +140 -0
- package/plugins/harness-everything/hooks/scripts/context-compact.js +115 -0
- package/plugins/harness-everything/hooks/scripts/contract-test.js +146 -0
- package/plugins/harness-everything/hooks/scripts/depth-guard.js +65 -0
- package/plugins/harness-everything/hooks/scripts/lib/fable-contracts.js +121 -0
- package/plugins/harness-everything/hooks/scripts/lib/harness-state.js +275 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/claude.js +168 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/codex.js +88 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/continue.js +91 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/copilot.js +85 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/cursor.js +89 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/hermes.js +108 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/index.js +17 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/worktrees.js +20 -0
- package/plugins/harness-everything/hooks/scripts/lib/workspace.js +290 -0
- package/plugins/harness-everything/hooks/scripts/openai-session-start.js +81 -0
- package/plugins/harness-everything/hooks/scripts/reset-circuit-breaker.js +30 -0
- package/plugins/harness-everything/hooks/scripts/rule-of-3-tracker.js +125 -0
- package/plugins/harness-everything/hooks/scripts/rule-of-3.js +195 -0
- package/plugins/harness-everything/hooks/scripts/state-persist.js +100 -0
- package/plugins/harness-everything/hooks/scripts/stop-gate.js +80 -0
- package/plugins/harness-everything/hooks/scripts/subagent-scope-guard.js +194 -0
- package/plugins/harness-everything/hooks/scripts/subagent-start.js +19 -0
- package/plugins/harness-everything/hooks/scripts/subagent-stop.js +6 -0
- package/plugins/harness-everything/hooks/session-start.js +81 -0
- package/plugins/harness-everything/plugin.json +43 -0
- package/plugins/harness-everything/scripts/lib/advisory-text.js +174 -0
- package/plugins/harness-everything/skills/environment-detection/SKILL.md +41 -0
- package/plugins/harness-everything/skills/environment-detection/references/shell-syntax-rules.md +76 -0
- package/plugins/harness-everything/skills/environment-detection/scripts/preflight.js +103 -0
- package/plugins/harness-everything/skills/eval-harness/SKILL.md +42 -0
- package/plugins/harness-everything/skills/eval-harness/references/scoring-rubric.md +68 -0
- package/plugins/harness-everything/skills/eval-harness/scripts/evaluate.js +79 -0
- package/plugins/harness-everything/skills/fable-discipline/SKILL.md +41 -0
- package/plugins/harness-everything/skills/fable-discipline/references/discipline-rules.md +22 -0
- package/plugins/harness-everything/skills/fable-mode/CONTRACT-FORMAT.md +165 -0
- package/plugins/harness-everything/skills/fable-mode/SKILL.md +39 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-orchestrator.md +159 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-verifier.md +32 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-haiku.md +52 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-sonnet.md +53 -0
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/SKILL.md +43 -0
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/references/guardrail-rules.md +56 -0
- package/plugins/harness-everything/skills/fable-mode/fable-haiku/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/fable-haiku/references/routing-notes.md +26 -0
- package/plugins/harness-everything/skills/fable-mode/fable-opus/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/fable-sonnet/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/model-matrix.json +32 -0
- package/plugins/harness-everything/skills/fable-mode/references/execution-phases.md +37 -0
- package/plugins/harness-everything/skills/fable-mode/references/model-matrix.md +39 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/ensemble-review.js +371 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/model-selector.js +260 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/workflow-plan-consumer.js +337 -0
- package/plugins/harness-everything/skills/find-skills/SKILL.md +48 -0
- package/plugins/harness-everything/skills/find-skills/references/discovery-flow.md +102 -0
- package/plugins/harness-everything/skills/find-skills/scripts/use-skill.js +112 -0
- package/plugins/harness-everything/skills/git-commit/SKILL.md +38 -0
- package/plugins/harness-everything/skills/git-commit/guides/ANGULAR_STYLE.md +61 -0
- package/plugins/harness-everything/skills/git-commit/guides/COMMIT_GENERATION.md +24 -0
- package/plugins/harness-everything/skills/git-commit/guides/LANGUAGE_DETECTION.md +16 -0
- package/plugins/harness-everything/skills/git-commit/guides/MAIN_REPO.md +15 -0
- package/plugins/harness-everything/skills/git-commit/guides/SUBMODULES.md +27 -0
- package/plugins/harness-everything/skills/git-commit/references/commit-flow.md +30 -0
- package/plugins/harness-everything/skills/grill-me/SKILL.md +43 -0
- package/plugins/harness-everything/skills/grill-me/references/grilling-playbook.md +38 -0
- package/plugins/harness-everything/skills/grill-with-docs/ADR-FORMAT.md +47 -0
- package/plugins/harness-everything/skills/grill-with-docs/CONTEXT-FORMAT.md +63 -0
- package/plugins/harness-everything/skills/grill-with-docs/SKILL.md +41 -0
- package/plugins/harness-everything/skills/grill-with-docs/references/session-playbook.md +99 -0
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +43 -0
- package/plugins/harness-everything/skills/harness-everything/examples/README.md +9 -0
- package/plugins/harness-everything/skills/harness-everything/examples/todo-workflow.md +22 -0
- package/plugins/harness-everything/skills/harness-everything/references/router-workflow-plan.md +128 -0
- package/plugins/harness-everything/skills/harness-everything/references/skill-registry.md +39 -0
- package/plugins/harness-everything/skills/harness-everything/references/triage-and-tiers.md +193 -0
- package/plugins/harness-everything/skills/harness-everything/schemas/router-task-shape.schema.json +126 -0
- package/plugins/harness-everything/skills/harness-everything/schemas/router-workflow-plan.schema.json +149 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/bootstrap.js +212 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/ensemble-policy.js +102 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +212 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/router-contract.js +612 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/routing-keywords.json +320 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/self-heal.js +250 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/tier-router.js +459 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/verify-gate.js +100 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/SKILL.md +38 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/DEEPENING.md +37 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/HTML-REPORT.md +123 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/LANGUAGE.md +53 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/refactoring-workflow.md +59 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/references/README.md +16 -0
- package/plugins/harness-everything/skills/install-cognitive-os/SKILL.md +44 -0
- package/plugins/harness-everything/skills/install-cognitive-os/references/cognitive-loop.md +27 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/NOTICE-agency-agents.md +27 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/SKILL.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/examples/csharp-monorepo.md +123 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/examples/typescript-cli.md +126 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-claude.md +77 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-codex.md +146 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-continue.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-copilot.md +74 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-cursor.md +40 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-hermes.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/role-capabilities.md +108 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/agency-agents.md +31 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/architecture-guide.md +36 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/orchestration.md +65 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/agency-catalog.js +190 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/consume-workflow-plan.js +129 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/index_memory.js +104 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/path-boundary.js +59 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/project-docs-resolver.js +357 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/scaffold.js +408 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/AGENTS.md +20 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/README.md +17 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/agent-launcher.md +68 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-csharp.md +56 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-polyglot.md +62 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-ts.md +57 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/memory-keeper.md +37 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/requirement-analyzer.md +66 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/specialist-agent.md +34 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/task-verifier.md +56 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/tool-maker.md +33 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/01-init.md +22 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/02-analysis.md +23 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/03-generation.md +25 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/04-launcher.md +16 -0
- package/plugins/harness-everything/skills/repo-docs/SKILL.md +42 -0
- package/plugins/harness-everything/skills/repo-docs/references/process-guide.md +60 -0
- package/plugins/harness-everything/skills/repo-docs/templates/agents-template.md +264 -0
- package/plugins/harness-everything/skills/repo-docs/templates/knowledge-agents-template.md +50 -0
- package/plugins/harness-everything/skills/repo-docs/templates/knowledge-readme-template.md +83 -0
- package/plugins/harness-everything/skills/repo-docs/templates/manual-recon.md +117 -0
- package/plugins/harness-everything/skills/repo-docs/templates/multi-skills-readme-template.md +72 -0
- package/plugins/harness-everything/skills/repo-docs/templates/product-agents-template.md +78 -0
- package/plugins/harness-everything/skills/repo-docs/templates/product-readme-template.md +95 -0
- package/plugins/harness-everything/skills/repo-docs/templates/readme-template.md +136 -0
- package/plugins/harness-everything/skills/repo-docs/templates/single-skill-readme-template.md +70 -0
- package/plugins/harness-everything/skills/repo-docs/templates/skills-agents-template.md +60 -0
- package/plugins/harness-everything/skills/rewrite-commits/SKILL.md +37 -0
- package/plugins/harness-everything/skills/rewrite-commits/references/workflow.md +18 -0
- package/plugins/harness-everything/skills/security-review/SKILL.md +47 -0
- package/plugins/harness-everything/skills/security-review/guides/OWASP-PATTERNS.md +31 -0
- package/plugins/harness-everything/skills/security-review/guides/STRIDE-THREAT-MODEL.md +21 -0
- package/plugins/harness-everything/skills/security-review/references/security-checklist.md +478 -0
- package/plugins/harness-everything/skills/security-review/scripts/audit-secrets.js +85 -0
- package/plugins/harness-everything/skills/self-evolve/SKILL.md +45 -0
- package/plugins/harness-everything/skills/self-evolve/references/memory-resolution.md +61 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/persist-memory.js +272 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/register-dynamic-skill.js +253 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/self-regression.js +125 -0
- package/plugins/harness-everything/skills/skill-creator/SKILL.md +47 -0
- package/plugins/harness-everything/skills/skill-creator/references/authoring-workflow.md +14 -0
- package/plugins/harness-everything/skills/skill-creator/references/dynamic-generation-contract.md +21 -0
- package/plugins/harness-everything/skills/skill-creator/references/quality-checklist.md +16 -0
- package/plugins/harness-everything/skills/skill-creator/references/quality-principles.md +81 -0
- package/plugins/harness-everything/skills/skill-creator/references/testing-workflow.md +27 -0
- package/plugins/harness-everything/skills/skill-style/SKILL.md +42 -0
- package/plugins/harness-everything/skills/skill-style/references/style-guide.md +62 -0
- package/plugins/harness-everything/skills/tdd/SKILL.md +38 -0
- package/plugins/harness-everything/skills/tdd/fixtures/pass-evidence.json +100 -0
- package/plugins/harness-everything/skills/tdd/fixtures/quality-scenarios.json +20 -0
- package/plugins/harness-everything/skills/tdd/guides/deep-modules.md +33 -0
- package/plugins/harness-everything/skills/tdd/guides/interface-design.md +31 -0
- package/plugins/harness-everything/skills/tdd/guides/mocking.md +59 -0
- package/plugins/harness-everything/skills/tdd/guides/refactoring.md +10 -0
- package/plugins/harness-everything/skills/tdd/guides/tests.md +61 -0
- package/plugins/harness-everything/skills/tdd/references/core-discipline.md +51 -0
- package/plugins/harness-everything/skills/tdd/references/integration-testing.md +13 -0
- package/plugins/harness-everything/skills/tdd/references/quality-model.md +68 -0
- package/plugins/harness-everything/skills/tdd/references/unit-testing.md +15 -0
- package/plugins/harness-everything/skills/tdd/schemas/quality-evidence.schema.json +211 -0
- package/plugins/harness-everything/skills/tdd/schemas/quality-report.schema.json +110 -0
- package/plugins/harness-everything/skills/tdd/scripts/quality-gate.js +475 -0
- package/plugins/harness-everything/skills/to-spec/SKILL.md +40 -0
- package/plugins/harness-everything/skills/to-spec/references/process.md +77 -0
- package/plugins/harness-everything/skills/to-spec/scripts/check-project-docs.js +239 -0
- package/plugins/harness-everything/skills/to-spec/scripts/project-docs-resolver.js +111 -0
- package/plugins/harness-everything/skills/to-spec/templates/cli-reference.md +31 -0
- package/plugins/harness-everything/skills/to-spec/templates/dev-doc.md +27 -0
- package/plugins/harness-everything/skills/to-spec/templates/feature-spec.md +59 -0
- package/plugins/harness-everything/skills/to-spec/templates/schema-doc.md +31 -0
- package/plugins/harness-everything/skills/to-tickets/SKILL.md +42 -0
- package/plugins/harness-everything/skills/to-tickets/references/publishing-and-templates.md +97 -0
- package/plugins/harness-everything/skills/todo-driven-workflow/SKILL.md +42 -0
- package/plugins/harness-everything/skills/todo-driven-workflow/references/execution-guide.md +27 -0
- package/plugins/harness-everything/skills/using-git-worktrees/SKILL.md +49 -0
- package/plugins/harness-everything/skills/using-git-worktrees/references/workflow-details.md +108 -0
- package/plugins/harness-everything/skills/verification-loop/SKILL.md +46 -0
- package/plugins/harness-everything/skills/verification-loop/references/verification-phases.md +116 -0
- package/plugins/harness-everything/skills/verification-loop/templates/verification-report.template.md +15 -0
- package/plugins/harness-everything/skills/verify-before-claim/SKILL.md +43 -0
- package/plugins/harness-everything/skills/verify-before-claim/references/verification-guide.md +73 -0
- package/plugins/harness-everything/skills/zoom-out/SKILL.md +42 -0
- package/plugins/harness-everything/skills/zoom-out/references/circuit-breaker.md +66 -0
- package/plugins/harness-everything/skills/zoom-out/templates/zoom-out-report.template.md +20 -0
- package/repo-docs/SKILL.md +3 -7
- package/rewrite-commits/SKILL.md +2 -2
- package/scripts/build-openai-submission.js +158 -0
- package/scripts/installer.js +86 -11
- package/scripts/lib/advisory-text.js +25 -18
- package/scripts/lib/execution-contract.js +70 -0
- package/scripts/lib/manifest.js +34 -38
- package/scripts/lib/path-boundary.js +90 -0
- package/scripts/lib/skills.js +124 -70
- package/scripts/lib/workspace.js +269 -7
- package/scripts/repository-contract.js +213 -0
- package/scripts/sync-current-docs.js +12 -0
- package/scripts/sync-openai-plugin.js +91 -0
- package/scripts/sync-release-version.js +212 -0
- package/security-review/SKILL.md +4 -7
- package/security-review/scripts/audit-secrets.js +9 -8
- package/self-evolve/SKILL.md +9 -11
- package/self-evolve/references/memory-resolution.md +4 -1
- package/self-evolve/scripts/persist-memory.js +221 -88
- package/self-evolve/scripts/register-dynamic-skill.js +12 -10
- package/self-evolve/scripts/self-regression.js +23 -8
- package/skill-creator/SKILL.md +12 -14
- package/skill-creator/references/quality-checklist.md +1 -0
- package/skill-style/SKILL.md +4 -4
- package/skill-style/references/style-guide.md +29 -3
- package/submission/openai/README.md +67 -0
- package/submission/openai/listing.json +31 -0
- package/submission/openai/test-cases.json +61 -0
- package/tdd/SKILL.md +20 -26
- package/tdd/fixtures/pass-evidence.json +100 -0
- package/tdd/fixtures/quality-scenarios.json +20 -0
- package/tdd/references/core-discipline.md +2 -0
- package/tdd/references/integration-testing.md +13 -0
- package/tdd/references/quality-model.md +68 -0
- package/tdd/references/unit-testing.md +15 -0
- package/tdd/schemas/quality-evidence.schema.json +211 -0
- package/tdd/schemas/quality-report.schema.json +110 -0
- package/tdd/scripts/quality-gate.js +475 -0
- package/to-spec/SKILL.md +3 -5
- package/to-spec/scripts/check-project-docs.js +42 -62
- package/to-spec/scripts/project-docs-resolver.js +111 -0
- package/to-tickets/SKILL.md +4 -4
- package/todo-driven-workflow/SKILL.md +6 -4
- package/using-git-worktrees/SKILL.md +3 -3
- package/verification-loop/SKILL.md +4 -4
- package/verify-before-claim/SKILL.md +2 -2
- package/zoom-out/SKILL.md +4 -4
- package/ab-test-deep-analysis-2026-08-27.md +0 -383
- package/ab-test-results-2026-08-27.md +0 -152
- package/ab-test-statistical-analysis-31cases.md +0 -193
- package/opencode-plugin/hooks/circuit-breaker.js +0 -121
- package/opencode-plugin/hooks/compliance.js +0 -128
- package/opencode-plugin/hooks/post-edit.js +0 -62
- package/opencode-plugin/hooks/pre-complete.js +0 -65
- package/opencode-plugin/test-hook-firing.js +0 -198
package/BENCHMARK_SOP.md
CHANGED
|
@@ -2,7 +2,11 @@
|
|
|
2
2
|
|
|
3
3
|
This guide helps you objectively test the performance difference between an AI Agent **with the Harness framework** and **without the Harness framework (Vanilla)**. You can switch between different underlying models (e.g., Haiku, Sonnet, Opus) to test whether the framework truly prevents "over-engineering" and breaks through the "reasoning ceiling".
|
|
4
4
|
|
|
5
|
-
**What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether
|
|
5
|
+
**What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether an underlying hook/plugin mechanism actually loaded, fired, or blocked anything. A well-behaved model can pass every scenario below even if a mechanism silently no-ops — that is why behavior evidence and mechanism/live-host evidence are tracked separately in [VERIFICATION.md](VERIFICATION.md).
|
|
6
|
+
|
|
7
|
+
Do not classify mechanism support by product name alone. A host can expose more than one Harness surface. For example, the general Codex installer path is instruction-oriented, while the **Codex / local OpenAI plugin** packages `SessionStart` and `UserPromptSubmit` invariant hooks; the public OpenAI **Skills-only** artifact does not include those local lifecycle hooks. OpenCode has a real plugin implementation with deterministic mechanism coverage, but live plugin loading remains unverified. Use [docs/platform-capabilities.md](docs/platform-capabilities.md) as the current source of truth.
|
|
8
|
+
|
|
9
|
+
This SOP is therefore a **behavior** benchmark. Pair it with the mechanism/package/live-host checks that apply to the exact installation surface under test before making an enforcement claim.
|
|
6
10
|
|
|
7
11
|
## Testing Process
|
|
8
12
|
|
|
@@ -10,17 +14,17 @@ To maintain controlled variables, please follow these steps for testing:
|
|
|
10
14
|
|
|
11
15
|
1. **Prepare the Test Environment**: Prepare a small project containing an intentionally flawed bug or requiring refactoring.
|
|
12
16
|
2. **Control Group (Vanilla Session)**:
|
|
13
|
-
- Open a **clean session without
|
|
17
|
+
- Open a **clean session without Harness skills/runtime integration loaded**.
|
|
14
18
|
- Provide the test task instructions (see scenarios below).
|
|
15
19
|
- Record: Was it successful? How many Tokens (or steps) were used? Did it hit a dead end?
|
|
16
|
-
- Export the conversation
|
|
20
|
+
- Export or preserve the conversation/session evidence required by your benchmark workflow.
|
|
17
21
|
3. **Experimental Group (Harness Session)**:
|
|
18
|
-
- Open a session with
|
|
22
|
+
- Open a session with the exact Harness installation surface being tested.
|
|
19
23
|
- Provide the exact same test task instructions.
|
|
20
|
-
- Record: Was it successful? How many Tokens were used?
|
|
21
|
-
-
|
|
24
|
+
- Record: Was it successful? How many Tokens were used? Which Harness skills/mechanisms were actually observed rather than merely expected?
|
|
25
|
+
- Preserve the corresponding session evidence.
|
|
22
26
|
4. **Automated Scoring (Scoring)**:
|
|
23
|
-
-
|
|
27
|
+
- Use `eval-harness` or the repository benchmark tooling to score the paired evidence against a declared rubric.
|
|
24
28
|
|
|
25
29
|
---
|
|
26
30
|
|
|
@@ -30,33 +34,49 @@ To maintain controlled variables, please follow these steps for testing:
|
|
|
30
34
|
- **Scenario**: The project has a `README.md` with an obvious spelling error ("Instalation" -> "Installation").
|
|
31
35
|
- **User Prompt**: "Help me fix the typo in the README."
|
|
32
36
|
- **Expected Difference**:
|
|
33
|
-
- **Vanilla**:
|
|
34
|
-
- **Harness**: Should
|
|
37
|
+
- **Vanilla**: May over-read or over-plan relative to the tiny change.
|
|
38
|
+
- **Harness**: Should classify this as a small task and prefer direct execution with only enough discovery/verification to support the final claim.
|
|
35
39
|
|
|
36
40
|
### Test B: Micro-Error Loop Defense Test (Tier 2 Task)
|
|
37
|
-
- **Scenario**: Provide a sorting algorithm containing a boundary condition error (e.g., crashing on empty arrays) and intentionally set a
|
|
41
|
+
- **Scenario**: Provide a sorting algorithm containing a boundary condition error (e.g., crashing on empty arrays) and intentionally set a hard-to-spot logic error.
|
|
38
42
|
- **User Prompt**: "This sort function has a bug, help me fix it."
|
|
39
43
|
- **Expected Difference**:
|
|
40
|
-
- **Vanilla**:
|
|
41
|
-
- **Harness**:
|
|
44
|
+
- **Vanilla**: May repeatedly micro-adjust the code after failed verification.
|
|
45
|
+
- **Harness**: Should use evidence-driven debugging and verification. If the selected installation surface packages an automatic Rule-of-3 mechanism, repeated same-signature failures can trigger it mechanically; otherwise the same recovery principle is behavioral/advisory rather than an automatic hook claim. `tdd` may be selected when executable behavior benefits from it, but Tier 2 does not force TDD universally.
|
|
42
46
|
|
|
43
47
|
### Test C: Attention Loss and Hallucination Test (Tier 3 Task)
|
|
44
48
|
- **Scenario**: In a project without any defined interfaces, ask for a large-scale abstraction.
|
|
45
49
|
- **User Prompt**: "Refactor all database connections in this project into a Dependency Injection architecture."
|
|
46
50
|
- **Expected Difference**:
|
|
47
|
-
- **Vanilla**:
|
|
48
|
-
- **Harness**:
|
|
51
|
+
- **Vanilla**: May read too broadly, lose constraints, or make unsupported architectural assumptions.
|
|
52
|
+
- **Harness**: Should classify the work as broad/architectural, discover evidence before mutation, keep scope/decisions explicit, and gather verification evidence. `fable-mode`, `improve-codebase-architecture`, `grill-with-docs`, or multi-agent execution are possible tactics when useful; Tier 3 does not mandate one fixed chain.
|
|
49
53
|
|
|
50
54
|
### Test D: Knowledge Boundary and Hallucination Test (Knowledge Boundary Test)
|
|
51
55
|
- **Scenario**: Intentionally ask about current events unrelated to the project, e.g., "Updates on the Russia-Ukraine war", "Current US-China relations", "Who is the current US Secretary of State", "Latest on US-Iran conflicts".
|
|
52
56
|
- **User Prompt**: "Summarize the latest updates on the Russia-Ukraine war and US-China relations for me."
|
|
53
57
|
- **Expected Difference**:
|
|
54
|
-
- **Vanilla**:
|
|
55
|
-
- **Harness**:
|
|
58
|
+
- **Vanilla**: May answer from stale assumptions or provide weakly grounded current claims.
|
|
59
|
+
- **Harness**: The relevant evidence/knowledge-boundary discipline should make the agent recognize when current external verification is needed and avoid fabricating unavailable facts. Note that ordinary non-software web/Q&A may bypass the software-work kernel; this scenario tests evidence discipline rather than Tier routing.
|
|
56
60
|
|
|
57
61
|
### Test E: Environment & Tool Awareness Test (Terminal Detection)
|
|
58
62
|
- **Scenario**: The user is running a specific terminal on Windows (e.g., Git Bash instead of the default Command Prompt or PowerShell).
|
|
59
63
|
- **User Prompt**: "Run a terminal command to list all environment variables and save them to env_list.txt."
|
|
60
64
|
- **Expected Difference**:
|
|
61
|
-
- **Vanilla**:
|
|
62
|
-
- **Harness**:
|
|
65
|
+
- **Vanilla**: May assume the default OS shell and choose incompatible syntax.
|
|
66
|
+
- **Harness**: Should determine the execution environment before relying on shell-specific commands, then use syntax appropriate to the actual shell. Whether environment context is injected automatically or discovered explicitly depends on the installation surface under test.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Reporting Rules
|
|
71
|
+
|
|
72
|
+
For each run, record at least:
|
|
73
|
+
|
|
74
|
+
- repository revision;
|
|
75
|
+
- model/host and exact Harness installation surface;
|
|
76
|
+
- scenario and prompt;
|
|
77
|
+
- success/failure plus objective task evidence;
|
|
78
|
+
- observed skill/mechanism events, clearly distinguishing expected behavior from trace evidence;
|
|
79
|
+
- token/step metrics when available;
|
|
80
|
+
- links/hashes for retained session evidence.
|
|
81
|
+
|
|
82
|
+
Do not turn a behavioral PASS into a statement such as "the hook fired" unless the trace proves that hook/plugin event. Likewise, a surface without a particular automatic gate is not a mechanism failure if [docs/platform-capabilities.md](docs/platform-capabilities.md) never claimed that gate was packaged there.
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,149 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
All notable changes to this project are documented in this file.
|
|
4
|
+
|
|
5
|
+
## [0.4.1](https://github.com/dyphn1/Harness-everything/compare/v0.4.0...v0.4.1) (2026-09-16)
|
|
6
|
+
|
|
7
|
+
## [0.4.0](https://github.com/dyphn1/Harness-everything/compare/v0.3.6...v0.4.0) (2026-09-16)
|
|
8
|
+
|
|
9
|
+
### Features
|
|
10
|
+
|
|
11
|
+
* **action-gate:** add destructive action rule table ([fc472b0](https://github.com/dyphn1/Harness-everything/commit/fc472b04dcba1b2b27faf42e2ba69a28e49191f9))
|
|
12
|
+
* **action-gate:** enforce pre-action approval ([fca6ccb](https://github.com/dyphn1/Harness-everything/commit/fca6ccba4d07a18f46fa1d43b214cb10a8be122c))
|
|
13
|
+
* **action-gate:** mirror destructive action rule table ([4ad59aa](https://github.com/dyphn1/Harness-everything/commit/4ad59aa51e2450a761e31db524f03e0ac754b1a6))
|
|
14
|
+
* **action-gate:** mirror pre-action approval enforcement ([5b254ea](https://github.com/dyphn1/Harness-everything/commit/5b254eaace8e438c3e1aa969ad150256f6e07161))
|
|
15
|
+
* **action-gate:** wire Claude lifecycle enforcement ([eadca2f](https://github.com/dyphn1/Harness-everything/commit/eadca2fcc8a98a5467a12774f0cad7983a4252f8))
|
|
16
|
+
* **action-gate:** wire plugin lifecycle enforcement ([f15a500](https://github.com/dyphn1/Harness-everything/commit/f15a50074fc564ebb8cdc72bc6b2979fe323c0f2))
|
|
17
|
+
* **codex:** observe native permission requests ([2cb472c](https://github.com/dyphn1/Harness-everything/commit/2cb472cee3786e2f510f0e1356f7163bcd37026d))
|
|
18
|
+
* **codex:** package native permission observer ([c7c279c](https://github.com/dyphn1/Harness-everything/commit/c7c279c5e187ec02830c91b8925129377fe338eb))
|
|
19
|
+
* **codex:** wire native permission request observer ([318c6ca](https://github.com/dyphn1/Harness-everything/commit/318c6ca14ef16f1846a0f1b993b1ce0a8e13d588))
|
|
20
|
+
* **compatibility:** Add official platform capability matrix ([3b423e2](https://github.com/dyphn1/Harness-everything/commit/3b423e233536406a6ef58609a1471b604ed1b461))
|
|
21
|
+
* **fable:** add bounded ensemble evidence synthesis ([df18314](https://github.com/dyphn1/Harness-everything/commit/df1831495344cc738617c320216799146192f6af))
|
|
22
|
+
* **fable:** add workflow plan consumer and run-scoped contracts ([215fe26](https://github.com/dyphn1/Harness-everything/commit/215fe266a748591870c98873100e05c05d3ce1b5))
|
|
23
|
+
* **hooks:** add run-scoped fable contract helpers ([0fa5b39](https://github.com/dyphn1/Harness-everything/commit/0fa5b3936184c98c293d6dbe265371cd0a5f47eb))
|
|
24
|
+
* **hooks:** correlate stage checks with run-scoped contracts ([2be79a9](https://github.com/dyphn1/Harness-everything/commit/2be79a9f9e50417c5211698bae380c4f080c50ee))
|
|
25
|
+
* **hooks:** validate subagent diffs against declared write sets ([56a1521](https://github.com/dyphn1/Harness-everything/commit/56a152135b100e432a6b2666531d26f930d6a0eb))
|
|
26
|
+
* **installer:** Add canonical per-scope skill store with link-aware uninstall ([4ec59dd](https://github.com/dyphn1/Harness-everything/commit/4ec59dd8702abf99574aab2524e65c84ad698bef)), closes [#49](https://github.com/dyphn1/Harness-everything/issues/49)
|
|
27
|
+
* **memory:** screen persisted rules and record provenance ([8f0759a](https://github.com/dyphn1/Harness-everything/commit/8f0759acf013281552440990977abc87d2d7ad4e))
|
|
28
|
+
* **openai:** 完成外掛生命週期支援 ([bf5d75b](https://github.com/dyphn1/Harness-everything/commit/bf5d75b19c5a979cd10a90a43cba61c09e2326ca)), closes [#72](https://github.com/dyphn1/Harness-everything/issues/72)
|
|
29
|
+
* **plugin:** add deterministic skills submission bundle builder ([a6fc321](https://github.com/dyphn1/Harness-everything/commit/a6fc3214460fe48961ee8f6cbadda4b3f0a335d4))
|
|
30
|
+
* **plugin:** add OpenAI submission listing source ([f0915dc](https://github.com/dyphn1/Harness-everything/commit/f0915dc650214691f7e89af2f19ea73cd1309530))
|
|
31
|
+
* **plugin:** mirror bounded ensemble evidence synthesis ([3b6c3f0](https://github.com/dyphn1/Harness-everything/commit/3b6c3f0f63c205882222747434395d8fb547847e))
|
|
32
|
+
* **plugin:** mirror bounded ensemble modifier policy ([aa9ee20](https://github.com/dyphn1/Harness-everything/commit/aa9ee20be35582567a9877e85b6337d2dded7198))
|
|
33
|
+
* **plugin:** mirror bounded ensemble schema ([e8b138e](https://github.com/dyphn1/Harness-everything/commit/e8b138e5018d4a011094432acc6ceb59cf127bd1))
|
|
34
|
+
* **plugin:** mirror kernel ensemble modifier ([389146a](https://github.com/dyphn1/Harness-everything/commit/389146a36037d3e1cf6ecbf4317ab0d887c254c7))
|
|
35
|
+
* **plugin:** mirror memory persistence screening ([0b707a9](https://github.com/dyphn1/Harness-everything/commit/0b707a91a89eb3da46091314f0a3977b5a9d90e9))
|
|
36
|
+
* **plugin:** package Harness Everything for ChatGPT and Codex ([0ce65d0](https://github.com/dyphn1/Harness-everything/commit/0ce65d0165b4890c847f561f79f95ebfd25d81e5))
|
|
37
|
+
* **plugin:** prepare Harness Everything for OpenAI publication ([8e827f6](https://github.com/dyphn1/Harness-everything/commit/8e827f6b8a0027c388bb18d94a8ae127a1d06141))
|
|
38
|
+
* **release:** add deterministic version synchronizer ([9a75283](https://github.com/dyphn1/Harness-everything/commit/9a75283718b624b98c98da6871ec80e8e397f031))
|
|
39
|
+
* **release:** align OpenCode plugin version ([e1a3e87](https://github.com/dyphn1/Harness-everything/commit/e1a3e87d79ec50701107e6e4cd29afa7a4c7eaf5))
|
|
40
|
+
* **release:** configure semantic release ([93020dc](https://github.com/dyphn1/Harness-everything/commit/93020dcedc0f58eabe7b05b27963be71cfa52e60))
|
|
41
|
+
* **release:** publish from conventional commit history ([585516b](https://github.com/dyphn1/Harness-everything/commit/585516bf4fa138d71860e0c25c01bef51cb31b32))
|
|
42
|
+
* **release:** remove beta Claude plugin version ([cb24975](https://github.com/dyphn1/Harness-everything/commit/cb24975f64c009cd84d571b9d994eb94899a7f32))
|
|
43
|
+
* **release:** remove beta Codex plugin version ([4c11024](https://github.com/dyphn1/Harness-everything/commit/4c11024f6f0895238447bb229cffbf4676bbb104))
|
|
44
|
+
* **release:** remove beta marketplace version ([85d48a7](https://github.com/dyphn1/Harness-everything/commit/85d48a790049460ad83e565674898f01d1592915))
|
|
45
|
+
* **release:** remove beta package version ([e5c24dd](https://github.com/dyphn1/Harness-everything/commit/e5c24ddd2f01451735b3961463b9139eb0608111))
|
|
46
|
+
* **release:** remove beta portable plugin version ([f94ff42](https://github.com/dyphn1/Harness-everything/commit/f94ff424830137a1d20e02277af461dfdb3475b0))
|
|
47
|
+
* **router:** add bounded ensemble modifier policy ([ae25dc5](https://github.com/dyphn1/Harness-everything/commit/ae25dc5096aba4254fc42c16f7dc746be5af668d))
|
|
48
|
+
* **router:** add task-shape schema ([5185d75](https://github.com/dyphn1/Harness-everything/commit/5185d759e68687eb32bd45f868581405ad8be59b))
|
|
49
|
+
* **router:** add versioned workflow plan contract ([714bb3f](https://github.com/dyphn1/Harness-everything/commit/714bb3fb4db4d47e3808c41e2eb133eb85df4510))
|
|
50
|
+
* **router:** add workflow plan schema ([1ffa89b](https://github.com/dyphn1/Harness-everything/commit/1ffa89b99d3d57497031e2673a966efe26304474))
|
|
51
|
+
* **router:** attach ensemble modifier at kernel contract boundary ([29841fc](https://github.com/dyphn1/Harness-everything/commit/29841fce11a15de1746517488eef218d4f1bf74c))
|
|
52
|
+
* **router:** consume structured tier contract ([6472bde](https://github.com/dyphn1/Harness-everything/commit/6472bdefc20659347256086600b29eadf358f727))
|
|
53
|
+
* **router:** derive task shape and safety modifiers ([904c0bf](https://github.com/dyphn1/Harness-everything/commit/904c0bf364d57443a8b1006d6e83709b3d2171ed))
|
|
54
|
+
* **router:** emit structured shadow routing contract ([49fcc72](https://github.com/dyphn1/Harness-everything/commit/49fcc7267bc4e6f91b92151e8009455b37905d77))
|
|
55
|
+
* **router:** extend task-shape schema for phase 2 ([21d98ec](https://github.com/dyphn1/Harness-everything/commit/21d98ec3d902e27181ae2e80b402e046376bdc3b))
|
|
56
|
+
* **router:** extend workflow-plan schema for selected strategies ([69b4f15](https://github.com/dyphn1/Harness-everything/commit/69b4f1506da1c5624f124085215c882f64c96f77))
|
|
57
|
+
* **router:** make kernel consume selected workflow plan ([f9162e5](https://github.com/dyphn1/Harness-everything/commit/f9162e55aa623c25023a30ccb84dbbe601ac6d26))
|
|
58
|
+
* **router:** select deterministic phase 2 strategies ([b85b0e9](https://github.com/dyphn1/Harness-everything/commit/b85b0e94c3fd1c749f5d563264bd8ce06b038b9f))
|
|
59
|
+
* **schema:** define bounded ensemble modifier contract ([6e1fa26](https://github.com/dyphn1/Harness-everything/commit/6e1fa266bee0665d7fa7713ba00697cd66b4edfb))
|
|
60
|
+
* **tdd:** Add quantified quality gate ([0fd4abf](https://github.com/dyphn1/Harness-everything/commit/0fd4abf19479ab3bb250d080622ddedd61593f31)), closes [#58](https://github.com/dyphn1/Harness-everything/issues/58)
|
|
61
|
+
* **workspace:** consume workspace portion of workflow plan ([5b75680](https://github.com/dyphn1/Harness-everything/commit/5b756807eae8b56ebcc2ecc2b312b91e42b84d37))
|
|
62
|
+
|
|
63
|
+
### Bug Fixes
|
|
64
|
+
|
|
65
|
+
* **action-gate:** Defer matched commands to Claude Code permission modes ([7628da1](https://github.com/dyphn1/Harness-everything/commit/7628da1cbadd2b5ed2916514e8172b281aaf6e80)), closes [#107](https://github.com/dyphn1/Harness-everything/issues/107)
|
|
66
|
+
* **action-gate:** preserve fail-closed runtime semantics ([a959316](https://github.com/dyphn1/Harness-everything/commit/a9593166e1c6210d2c5cafe1b3f688a53b68ca1b))
|
|
67
|
+
* **action-gate:** sync shared runtime into plugin ([ed40429](https://github.com/dyphn1/Harness-everything/commit/ed4042973bd79afaaa0e8d1c84c067eb85ef4f3e))
|
|
68
|
+
* **behavioral-evals:** Correct the find-skills verdict claimed in PR [#51](https://github.com/dyphn1/Harness-everything/issues/51) ([6afa34f](https://github.com/dyphn1/Harness-everything/commit/6afa34f5f3601c32d7d04a8437ae373a9bcb514a)), closes [#52](https://github.com/dyphn1/Harness-everything/issues/52)
|
|
69
|
+
* **behavioral-evals:** Grant claude engine real headless autonomy, correct rubric bugs found live ([90e36c2](https://github.com/dyphn1/Harness-everything/commit/90e36c2071fb5a71e16cdc99b1060cd3d699c43b)), closes [#52](https://github.com/dyphn1/Harness-everything/issues/52) [#52](https://github.com/dyphn1/Harness-everything/issues/52)
|
|
70
|
+
* **behavioral-evals:** harden evidence triage ([f7bd8ec](https://github.com/dyphn1/Harness-everything/commit/f7bd8ec8fe6b8f02f55e8b45e45af5e233f4d224)), closes [#56](https://github.com/dyphn1/Harness-everything/issues/56) [#56](https://github.com/dyphn1/Harness-everything/issues/56)
|
|
71
|
+
* **behavioral-evals:** Harden evidence triage ([ad45453](https://github.com/dyphn1/Harness-everything/commit/ad45453a8f08d28256ae569735aa2c7125d901e8))
|
|
72
|
+
* **behavioral-evals:** Make execution evidence grading command-aware ([aa2235d](https://github.com/dyphn1/Harness-everything/commit/aa2235d49a97764dd6406ce5f12fc057366faa17))
|
|
73
|
+
* **behavioral-evals:** parse Claude stream evidence ([77efb06](https://github.com/dyphn1/Harness-everything/commit/77efb06ae91b28d07b60e7833ba96940bae612cb)), closes [#52](https://github.com/dyphn1/Harness-everything/issues/52)
|
|
74
|
+
* **behavioral-evals:** parse Claude stream evidence ([f5c3a46](https://github.com/dyphn1/Harness-everything/commit/f5c3a4639b15464017b344b7329c990912480516)), closes [#52](https://github.com/dyphn1/Harness-everything/issues/52)
|
|
75
|
+
* **behavioral-evals:** Support --model for the claude engine, correct find-skills-exposure claim ([15e4c7c](https://github.com/dyphn1/Harness-everything/commit/15e4c7c98717dea615052114be9e8033f9e4feb5))
|
|
76
|
+
* **boundaries:** Add shared verifier contracts ([c093c50](https://github.com/dyphn1/Harness-everything/commit/c093c50de7cd12260f0fdee3f6e6d4f6a49d2d0b))
|
|
77
|
+
* **ci:** Derive the release baseline and gate tag pushes with the full CI suite ([0ec04af](https://github.com/dyphn1/Harness-everything/commit/0ec04af3f4fc100eced3b464c7be919f31a8e969)), closes [#39](https://github.com/dyphn1/Harness-everything/issues/39)
|
|
78
|
+
* **ci:** Fetch history for historical evidence tests ([aeeb69f](https://github.com/dyphn1/Harness-everything/commit/aeeb69f43cbec76f1a4d9f8e3e6cd1483bf33664))
|
|
79
|
+
* **ci:** Harden issue 20 verification gates ([b482e41](https://github.com/dyphn1/Harness-everything/commit/b482e41a9448efe0504032472b86eeb1f983c6cd))
|
|
80
|
+
* **ci:** Make SKILL.md path notation file-relative and fail on unknown placeholders ([543c057](https://github.com/dyphn1/Harness-everything/commit/543c057b676c9e4e98f745b1758b666a6756a95a)), closes [#44](https://github.com/dyphn1/Harness-everything/issues/44)
|
|
81
|
+
* **ci:** Publish the workspace skip-list instead of paying tokens to inline it ([cda087c](https://github.com/dyphn1/Harness-everything/commit/cda087cff423763f2e14fd2940cc1331396de313)), closes [#44](https://github.com/dyphn1/Harness-everything/issues/44)
|
|
82
|
+
* **ci:** Spend path markers only where the base is not the current file ([8b2a73e](https://github.com/dyphn1/Harness-everything/commit/8b2a73ed3c2a9b83009ab3273c7fa00ccd45645a)), closes [#44](https://github.com/dyphn1/Harness-everything/issues/44)
|
|
83
|
+
* **claude-plugin:** avoid duplicate standard hooks registration ([314bcf1](https://github.com/dyphn1/Harness-everything/commit/314bcf1663a24f9eb32fdf342a3b1ed58c2f0006))
|
|
84
|
+
* **claude:** anchor plugin hooks to CLAUDE_PLUGIN_ROOT ([6a4462b](https://github.com/dyphn1/Harness-everything/commit/6a4462b85fbc79464da6be0e0e27092a36fa9fa6))
|
|
85
|
+
* **claude:** resolve plugin-root hook commands during install ([fdd2dd7](https://github.com/dyphn1/Harness-everything/commit/fdd2dd7cacf07e46699dec582cd734966addea88))
|
|
86
|
+
* **claude:** sync installer hook-root resolver into plugin ([308949a](https://github.com/dyphn1/Harness-everything/commit/308949a7840bdc26179c4772dd4fa424b84d8f7e))
|
|
87
|
+
* **codex:** align hook lifecycle with supported events ([8f90425](https://github.com/dyphn1/Harness-everything/commit/8f904258d6bf5e46e939facff3d734ad2baf7e63))
|
|
88
|
+
* **codex:** canonicalize action-gate post adapter ([82b56af](https://github.com/dyphn1/Harness-everything/commit/82b56af6195d05790573b3e75ffcf16684011d73))
|
|
89
|
+
* **codex:** infer action-gate outcome from PostToolUse ([0c79b11](https://github.com/dyphn1/Harness-everything/commit/0c79b110f5af9f311c80bdfaf42e94f26c5c1298))
|
|
90
|
+
* **fable:** serialize overlapping staged work without rejecting plan ([33e4c1a](https://github.com/dyphn1/Harness-everything/commit/33e4c1ae9dc5d74ee229c1e2b1758153f25a1f45))
|
|
91
|
+
* **find-skills:** Add tested host-agent exposure for ephemeral skills ([#35](https://github.com/dyphn1/Harness-everything/issues/35)) ([11fe847](https://github.com/dyphn1/Harness-everything/commit/11fe847bf6d8a1643b9487106b5519aae87b1e5a))
|
|
92
|
+
* **find-skills:** Trim SKILL.md under the 500-token waza hard limit ([5d66c0a](https://github.com/dyphn1/Harness-everything/commit/5d66c0a8a35fd6495b137b5578a511541ad88c30)), closes [#35](https://github.com/dyphn1/Harness-everything/issues/35)
|
|
93
|
+
* Harden PR [#66](https://github.com/dyphn1/Harness-everything/issues/66) behavioral evidence triage ([fcf0015](https://github.com/dyphn1/Harness-everything/commit/fcf00157597ed17ead63dcfb68e35c913b21fc2e))
|
|
94
|
+
* **hooks:** keep contract correlation session and path safe ([6655d83](https://github.com/dyphn1/Harness-everything/commit/6655d836394acd0e83145e35f1f20f3bd33e17aa))
|
|
95
|
+
* **hooks:** treat pending stage contracts as active ([ec6c265](https://github.com/dyphn1/Harness-everything/commit/ec6c2653afc97dd48ff2f9b8051d4ef57a28e03b))
|
|
96
|
+
* **installer:** discover native Continue and Hermes manifests ([d836760](https://github.com/dyphn1/Harness-everything/commit/d836760179e09cd08ec5aa206e439624d68d1973))
|
|
97
|
+
* **installer:** fail closed on malformed Claude settings ([fdec691](https://github.com/dyphn1/Harness-everything/commit/fdec691ef682ca0c4c28a4490d1be8b539ffbe9f))
|
|
98
|
+
* **installer:** Honor explicit uninstall scope flags over the interactive menu ([9f791d9](https://github.com/dyphn1/Harness-everything/commit/9f791d9ac39fa20a24e82ee996e71312f187b437)), closes [#48](https://github.com/dyphn1/Harness-everything/issues/48) [#49](https://github.com/dyphn1/Harness-everything/issues/49)
|
|
99
|
+
* **installer:** preserve pre-existing advisory files ([0299db0](https://github.com/dyphn1/Harness-everything/commit/0299db0724a789b83d8c5f3ed8b4d1a32bf68f14))
|
|
100
|
+
* **installer:** protect global Codex prompt ownership ([db67ac6](https://github.com/dyphn1/Harness-everything/commit/db67ac697e9f4d5beb50280f9db96f7114b99829))
|
|
101
|
+
* **installer:** protect global Copilot prompt ownership ([5f36651](https://github.com/dyphn1/Harness-everything/commit/5f36651d80550acea4a5d2a7c5bb532dece89e4a))
|
|
102
|
+
* **installer:** track Hermes global install ownership ([6967a3e](https://github.com/dyphn1/Harness-everything/commit/6967a3e33f0d2506192a16a5b705a0b04a774663))
|
|
103
|
+
* **installer:** use Codex native project skills path ([b4a372c](https://github.com/dyphn1/Harness-everything/commit/b4a372c6d9749c9ebd22b379bd45ce61358fffc6))
|
|
104
|
+
* **installer:** use Continue native global skills path ([cb0202c](https://github.com/dyphn1/Harness-everything/commit/cb0202cc518facf3836af1e377b58c10320fc6fb))
|
|
105
|
+
* **installer:** use Hermes native skill scopes ([d7c8275](https://github.com/dyphn1/Harness-everything/commit/d7c8275db04e71c5e6cf6b12e7b474da2fc8ecd9))
|
|
106
|
+
* **multi-agent-workspace:** Harden document resolution and migration ([6361f96](https://github.com/dyphn1/Harness-everything/commit/6361f96bec03741ea4a743f7284e0fdc7febeaa0)), closes [#43](https://github.com/dyphn1/Harness-everything/issues/43) [#43](https://github.com/dyphn1/Harness-everything/issues/43)
|
|
107
|
+
* **opencode-plugin:** Declare all five hooks and gate them in CI ([30e2e6d](https://github.com/dyphn1/Harness-everything/commit/30e2e6d6e868323d9b5af9a8cf8cb7015515879e)), closes [#37](https://github.com/dyphn1/Harness-everything/issues/37) [#20](https://github.com/dyphn1/Harness-everything/issues/20)
|
|
108
|
+
* **opencode-plugin:** Port enforcement plugin to opencode's real plugin API ([4ae960c](https://github.com/dyphn1/Harness-everything/commit/4ae960c15e582a7c4b23e41471d875120da8dcd6)), closes [#37](https://github.com/dyphn1/Harness-everything/issues/37) [#37](https://github.com/dyphn1/Harness-everything/issues/37)
|
|
109
|
+
* **opencode:** enforce reflection lifecycle ([d790e7f](https://github.com/dyphn1/Harness-everything/commit/d790e7f71d9738cea28d8d623e9c2cf857dd2793)), closes [#37](https://github.com/dyphn1/Harness-everything/issues/37)
|
|
110
|
+
* **opencode:** enforce reflection lifecycle ([74ffed3](https://github.com/dyphn1/Harness-everything/commit/74ffed3513aa7e9b51251a44d7c2a39298a990d1)), closes [#37](https://github.com/dyphn1/Harness-everything/issues/37)
|
|
111
|
+
* **opencode:** Harden reflection enforcement ([8989ab4](https://github.com/dyphn1/Harness-everything/commit/8989ab4c81ad8887359c8428eb8c3e3b5d7cc458))
|
|
112
|
+
* Parse top-level tool events in behavioral transcripts ([31fe8e1](https://github.com/dyphn1/Harness-everything/commit/31fe8e1cb6b193d21079dcc2c64edb712b40ac6d))
|
|
113
|
+
* **plugin:** align public long description with submission listing ([ab2e283](https://github.com/dyphn1/Harness-everything/commit/ab2e283424c34328f57b0373e9bc7ebab894f96a))
|
|
114
|
+
* **plugin:** match plugin.json manifest schema for repository and agents ([ab770ad](https://github.com/dyphn1/Harness-everything/commit/ab770ad1e9ec5489e83dbd06228fc10bd875806a)), closes [#86](https://github.com/dyphn1/Harness-everything/issues/86)
|
|
115
|
+
* **plugin:** Point agents at individual .md files, not a directory ([10e7512](https://github.com/dyphn1/Harness-everything/commit/10e75129ee5b19ee5c5d8872ee73195cc6c4109d))
|
|
116
|
+
* **release:** initialize conventional commits preset ([a0dbbf9](https://github.com/dyphn1/Harness-everything/commit/a0dbbf96f87a79de6674c95c7b99dd549d62809b))
|
|
117
|
+
* **release:** keep release commit compatible with commitlint ([#116](https://github.com/dyphn1/Harness-everything/issues/116)) ([27b1595](https://github.com/dyphn1/Harness-everything/commit/27b1595aef715aa696a67eaa58fc1367f0f3eba4)), closes [#115](https://github.com/dyphn1/Harness-everything/issues/115)
|
|
118
|
+
* **release:** make version rules explicit ([469a4ed](https://github.com/dyphn1/Harness-everything/commit/469a4edc3e2402cfd124c7764cc1b41e3387b319))
|
|
119
|
+
* **release:** remove beta from active skill metadata ([df2201c](https://github.com/dyphn1/Harness-everything/commit/df2201cd6508335d44fb5b14c0879808016ac742))
|
|
120
|
+
* **release:** use compatible changelog preset for [#113](https://github.com/dyphn1/Harness-everything/issues/113) ([9a181cf](https://github.com/dyphn1/Harness-everything/commit/9a181cf607797cd35a84a7b0e540387feda2f196))
|
|
121
|
+
* **release:** version changed skills during release ([333bb33](https://github.com/dyphn1/Harness-everything/commit/333bb33ceacbd5bc20be546d751bd98468e74db9))
|
|
122
|
+
* **release:** version only skills changed since last release ([6c74155](https://github.com/dyphn1/Harness-everything/commit/6c741557150c9156a0b31ed5713353f5a15ed21e))
|
|
123
|
+
* **router:** preserve invariant-first human-readable contract ([5a9baf0](https://github.com/dyphn1/Harness-everything/commit/5a9baf02e906c29e027affc9b735366526b54c3e))
|
|
124
|
+
* **router:** serialize unsafe explicit parallel requests ([fd02c97](https://github.com/dyphn1/Harness-everything/commit/fd02c977607b7400d9a61cbdc6648b79770a5a0b))
|
|
125
|
+
* **self-evolve:** Stop gating simple-rule persistence on the meta-repo's own CI ([4ab1bd8](https://github.com/dyphn1/Harness-everything/commit/4ab1bd88e11f55194e25f6bb636b164a57794eea)), closes [#54](https://github.com/dyphn1/Harness-everything/issues/54) [#52](https://github.com/dyphn1/Harness-everything/issues/52)
|
|
126
|
+
* **self-heal:** Identify the harness repo by package name, not source path ([f7ad891](https://github.com/dyphn1/Harness-everything/commit/f7ad891940cfa18b951d41ea3bcac80a98972e9c)), closes [#40](https://github.com/dyphn1/Harness-everything/issues/40)
|
|
127
|
+
* **skill-creator:** Require USE FOR/DO NOT USE FOR in the Quality Checklist ([68d9923](https://github.com/dyphn1/Harness-everything/commit/68d992350001d97fd3258a37dcc1e0d894a86c0b)), closes [#55](https://github.com/dyphn1/Harness-everything/issues/55)
|
|
128
|
+
* **skill:** Keep workspace contract within token budget ([7cd7956](https://github.com/dyphn1/Harness-everything/commit/7cd7956146be01221bc5e6b8bbf0d2fe21b027a0))
|
|
129
|
+
* **skill:** Keep workspace contract within token budget ([d360b8c](https://github.com/dyphn1/Harness-everything/commit/d360b8c70d89eaca8687821f616fe3cd9007f5ba))
|
|
130
|
+
* **skills:** Close Issue [#44](https://github.com/dyphn1/Harness-everything/issues/44) disclosure and path normalization defects ([27cb486](https://github.com/dyphn1/Harness-everything/commit/27cb486d0b7fbb0148df711bcfa8a1d6a355077d))
|
|
131
|
+
* **skills:** Trim SKILL.md bodies to fit waza's real 500-token budget ([3eeadf5](https://github.com/dyphn1/Harness-everything/commit/3eeadf5cf49ae4b00967a7901e6e157a2256ce20))
|
|
132
|
+
* **state:** Converge Claude/opencode runtime state onto a single global root ([ee348cd](https://github.com/dyphn1/Harness-everything/commit/ee348cdab972e2cfe08ec3277f1a89e86ecb2a74)), closes [#42](https://github.com/dyphn1/Harness-everything/issues/42)
|
|
133
|
+
* **state:** Fix zero-walk and cwd-default scatter in fable-mode and multi-agent-workspace ([6ac579e](https://github.com/dyphn1/Harness-everything/commit/6ac579e95632e9119a81260d4965974118c199be)), closes [#42](https://github.com/dyphn1/Harness-everything/issues/42)
|
|
134
|
+
* **state:** Harden workspace identity and migration ([0f3149d](https://github.com/dyphn1/Harness-everything/commit/0f3149da848b931d0c575311396b7f747122f452)), closes [#42](https://github.com/dyphn1/Harness-everything/issues/42)
|
|
135
|
+
* **state:** stabilize workspace identity and migration ([9dd2e3d](https://github.com/dyphn1/Harness-everything/commit/9dd2e3df3c4bfe91d8651c8344f568842687fbc1))
|
|
136
|
+
* **state:** stabilize workspace identity and migration ([d28a055](https://github.com/dyphn1/Harness-everything/commit/d28a055c1822a35ecc0ee96e7f578d8cb1363e96))
|
|
137
|
+
* **test:** close Codex Stop hook assertion ([ac1a8b6](https://github.com/dyphn1/Harness-everything/commit/ac1a8b600abc399cfa217e63a9c22ea8f08df362))
|
|
138
|
+
* **uninstall:** normalize global skill scope protection ([ab7ae46](https://github.com/dyphn1/Harness-everything/commit/ab7ae460a2cbf3abcf7f0da35436883d72fd0856))
|
|
139
|
+
* **uninstall:** remove orphaned Harness reference artifacts ([0a4d5b2](https://github.com/dyphn1/Harness-everything/commit/0a4d5b2caf0883f48aa3a22cdb97ade1f2d59823))
|
|
140
|
+
* **verifier:** export normalizeRepoPath for behavioral runner ([3e7b580](https://github.com/dyphn1/Harness-everything/commit/3e7b58024e9bef78299349e75f35f0c7cee37ea8))
|
|
141
|
+
* **verifier:** Harden consumer boundaries and mutation gates ([ef010ba](https://github.com/dyphn1/Harness-everything/commit/ef010bae835dcef736231ff07f9f09e0a6a886e8))
|
|
142
|
+
* **workspace:** keep skill contract compact and references explicit ([cce02bc](https://github.com/dyphn1/Harness-everything/commit/cce02bc4a1273428eae4dad586ab0a42eda2b2a7))
|
|
143
|
+
* **workspace:** preserve state mutation contract wording ([4cc11be](https://github.com/dyphn1/Harness-everything/commit/4cc11be2b8d94c7388320af4421e568ca51857d2))
|
|
144
|
+
|
|
145
|
+
# Changelog
|
|
146
|
+
|
|
3
147
|
All notable changes to this project will be documented in this file.
|
|
4
148
|
|
|
5
149
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
@@ -7,6 +151,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
151
|
|
|
8
152
|
---
|
|
9
153
|
|
|
154
|
+
## [Unreleased]
|
|
155
|
+
|
|
156
|
+
### Changed
|
|
157
|
+
- **Action gate defers to Claude Code permission modes** (#107): on Claude Code, a matched destructive command no longer returns a forced `permissionDecision: "ask"`, which prompted even in auto mode. The gate returns no decision, so the session's permission mode, rules and auto-mode classifier decide. It still records the matched rule and the command SHA-256 (`deferred-to-host`, then `executed` or `rejected`). An internal error on Claude also defers and shows a `systemMessage` warning. `HARNESS_ACTION_GATE_POLICY=always-ask` restores the forced prompt. Hosts without an ask decision keep the exit-2 block.
|
|
158
|
+
|
|
159
|
+
## [0.3.7-beta] - Unreleased
|
|
160
|
+
|
|
161
|
+
### Changed
|
|
162
|
+
- Prepare the next beta for the reviewed open-issue implementations (#20, #37, #42, #43, #44, #52, #56, #74, #75, #78, #82). Individual changes and verification evidence are recorded as they are integrated.
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
### Added
|
|
167
|
+
- **Portable Agent Plugin compatibility and platform matrix** (#75, #82): added the portable root `plugin.json`, official-source compatibility data for standalone skills versus plugins across eight agent platforms, explicit installer-target contracts, conservative status vocabulary, and deterministic gates that keep package/mechanism evidence separate from live-host evidence.
|
|
168
|
+
- **OpenAI/Codex lifecycle runtime compatibility** (#72): extended the local `.codex-plugin` package from session/prompt hooks to mechanism-tested `PreToolUse`, `PostToolUse`, `SubagentStart`, `SubagentStop`, and `Stop` adapters, including portable state dependencies and `apply_patch` edit tracking. Package synchronization now normalizes text EOLs for reproducible Windows/Unix output.
|
|
169
|
+
- **OpenAI/Codex plugin packaging and public Skills-only submission** (#74): added the repository marketplace (`.agents/plugins/marketplace.json`), local `.codex-plugin` package, synchronized copies of all 26 canonical skills, local `SessionStart` / `UserPromptSubmit` invariant hooks, OpenAI package/submission tests, deterministic Skills-only ZIP generation, and reviewer listing/test-case inputs. The local plugin mechanism boundary is explicitly separate from the public Skills-only artifact, which does not include the local lifecycle hooks.
|
|
170
|
+
- **Progressive disclosure and explicit skill paths** (#44): require short routed skill contracts, lazy-loaded workflow/reference detail, meaningful Mermaid coverage, and explicit `<this-skill-dir>/`, `<skills-repo-root>/`, or `<workspace>/` path bases.
|
|
171
|
+
- **Behavioral-eval evidence triage** (`behavioral-evals/`, #56): added structural case validation, ordered execution-evidence grading, sanitized historical archives with current-versus-historical hashes, and paired-rerun status tracking for old failures and never-run cases.
|
|
172
|
+
- **Shared verifier boundary contracts** (`scripts/lib/path-boundary.js`, `scripts/lib/execution-contract.js`): physical containment checks reject traversal and junction/symlink aliases, while execution evidence uses exact command/argv matching and all-or-nothing patch authorization. Deterministic mechanism coverage protects the contract for downstream verifier, workspace, installer, and plugin fixes.
|
|
173
|
+
- **Machine-verifiable TDD scoring gate** (`tdd/`, #58): the TDD skill routes behavior through a Mermaid path to separate unit or integration standards. Added profile-tagged source evidence, published canonical weights and calculation inputs, per-requirement and aggregate scores, mandatory PASS/FAIL gates, strict integration rerun comparison, auditable skip/N/A handling, and separate `NON_CONFORMANT`, `SOURCE_DEFECT`, and `SOURCE_CONFLICT` outcomes. `npm run tdd:quality -- <evidence.json> --output <report.json>` is the agent self-check; CI fixtures lock exact Unit and Integration scores plus malformed, incomplete, conflicting, skipped, and nondeterministic failures.
|
|
174
|
+
- **`--model` support for the `claude` engine in `behavioral-evals/run.js`**: `BEHAVIORAL_MODEL` now also passes `--model <value>` to the `claude -p` invocation (previously only the `opencode` engine read it). Lets a live run target a specific model, e.g. `BEHAVIORAL_MODEL=haiku node behavioral-evals/run.js run --case <id> --arm both`.
|
|
175
|
+
- **Full behavioral-eval skill coverage** (`behavioral-evals/cases/`, #34): added a fixture-bound baseline case for each of the 14 root skills that previously had only routing-precision coverage (`environment-detection`, `eval-harness`, `fable-discipline`, `find-skills`, `install-cognitive-os`, `multi-agent-workspace`, `rewrite-commits`, `self-evolve`, `skill-creator`, `skill-style`, `todo-driven-workflow`, `to-spec`, `to-tickets`, `using-git-worktrees`). `npm run eval:behavioral:validate` now covers all 26 shipped root skills instead of 12/26; `npm run eval:behavioral run` remains on-demand/token-costing, never CI. Live-verified on Haiku (`--engine claude`, see the harness-bug fix below) after two correction rounds: 5 EFFECTIVE (`eval-harness`, `skill-style`, `using-git-worktrees`, `to-tickets`, `to-spec`), 2 legitimate INCONCLUSIVE (`install-cognitive-os`, `fable-discipline`), 2 genuine INEFFECTIVE findings not rubric bugs (`self-evolve` treatment never wrote anything; `skill-creator` treatment wrote the Skill Contract table but omitted the required `USE FOR` section), and 5 still open (`environment-detection`, `todo-driven-workflow`, `multi-agent-workspace`, `rewrite-commits`, `find-skills` - blocked on #52 and/or an unverified prompt/turn-budget edit made after the last live run; see each case file's own note). Post-merge correction (2026-09-04): `find-skills` was briefly reported EFFECTIVE from a pre-permission-fix run where a denied tool call's permission-request text happened to mention the search command; re-run after the fix shows the skill actually works (a real search, presented correctly, never auto-applied) but #52 hides that from the grader - moved to the open/blocked list.
|
|
176
|
+
|
|
177
|
+
### Fixed
|
|
178
|
+
- **Verifier consumer boundary repair** (`behavioral-evals/`, `multi-agent-workspace/`, `opencode-plugin/`, `scripts/installer.js`): reject historical refs that cannot resolve, protect archive and legacy migration writes from junction/symlink aliases, require exact command evidence and atomic reflection patches, reject fixture traversal, and make global installation work outside a git checkout. Added a focused mutation gate so these protections are required by executable tests.
|
|
179
|
+
- **Issue #20 deterministic gate hardening**: behavioral case parsing now preserves block scalars from CRLF files, malformed skill YAML fails closed in collision detection, negative controls execute the real gates against bad fixtures, the opencode plugin runner's validation path works, and local routing checks gracefully skip when waza is unavailable. Live opencode loading and paid behavioral evidence remain explicitly unverified.
|
|
180
|
+
- **Claude behavioral-eval transcript measurement** (`behavioral-evals/run.js`, `behavioral-evals/transcript-parser.js`, #52): switched headless Claude runs to verbose `stream-json`, preserved legacy JSON parsing with unavailable tool visibility, and added structured tool-attempt/completion/denial evidence with fail-closed grading for partial or unrecognized streams. Paired aggregation now excludes inconclusive arms from effectiveness statistics and preserves unknown tool-count deltas as `null`.
|
|
181
|
+
- **Execution-safe behavioral-eval rubrics** (`behavioral-evals/transcript-parser.js`, `behavioral-evals/cases/`): command-keyed expectations now match the recorded command (including safe prefixes and script path arguments), legacy/user transcript prose cannot masquerade as execution evidence, and execution-focused cases use structured completed-tool assertions.
|
|
182
|
+
- **Multi-agent workspace path split** (#43): moved generated runtime metadata, roles, state, logs, router, and memory indexes into the global workspace-keyed Harness state home; document zones now resolve from root `CONTEXT-MAP.md`, repository configuration, existing conventions, or a committable fallback. The installed scaffold/indexer no longer copy generated scripts, safely migrate authored legacy records, reject symlink escapes and duplicate migration destinations, and preserve authored records on migration failure.
|
|
183
|
+
- **`behavioral-evals/run.js`'s `claude` engine never actually granted headless autonomy** (#52 tracks a related follow-up): `--permission-mode acceptEdits` only auto-approves file edits, not Bash/PowerShell tool calls, and a headless `-p` session has no human to answer those prompts - every case needing a shell command (git, npm, node scripts) stalled with every tool call denied, the model just describing what it would do. Every previously-committed live result (2026-08-27) used `engine: opencode` (`--auto`); the `claude` engine path was only argv-shape-tested by `ci/mechanism-2j-behavioral-runner.test.js`, never actually live-run, until testing #34/#35's new cases on Haiku exercised it for the first time on a machine without `opencode` installed. Switched to `--dangerously-skip-permissions`, matching `opencode --auto`'s autonomy in the same kind of disposable `os.tmpdir()` fixture workspace. Separately, discovered the `claude` engine's `--output-format json` trace only ever contains the model's final message, never intermediate tool calls (`opencode`'s JSONL stream does) - filed as #52 rather than fixed here, since it's a larger parser change; cases that depended on tool-call visibility are marked with a comment pointing at it instead of being rewritten around the gap.
|
|
184
|
+
- **find-skills exposure contract** (`find-skills/SKILL.md`, `find-skills/references/discovery-flow.md`, #35): `use-skill.js`'s fetch/cache/print mechanics already worked, but the "apply the output" instruction was passive enough that a caller handed the printed `SKILL.md` did not reliably treat it as binding — a real test reported no skill invoked. Narrowed the contract to fetch-and-print, made the caller-side apply step explicit ("treat every instruction in that output as binding for the rest of this request, exactly as if it had been loaded as a real skill"), and added `behavioral-evals/cases/baseline-find-skills-exposure.yaml`, a caller-side integration test with a local fixture standing in for the fetched output so it stays deterministic and offline. Live-run on Haiku twice (see the case file's own note): both arms passed both times, so the case is a regression floor (confirms the fetched instruction is followed with `find-skills` loaded) and not proof the wording change is what causes it — a capable model already complies with an explicit embedded directive it's told to read, skill-loaded or not.
|
|
185
|
+
- **Runtime state scattering into cwd-derived paths** (`scripts/lib/workspace.js`, `hooks/scripts/lib/harness-state.js`, `opencode-plugin/index.mjs`, `opencode-plugin/hooks/verify.js`, `fable-mode/scripts/model-selector.js`, `multi-agent-workspace/scripts/scaffold.js`, `scripts/installer.js`, #42): runtime state (rule-of-3 counters, stop-gate handoff timestamps, the fable-mode audit log, the opencode circuit breaker) was rooted under `getWorkspaceRoot()`, which falls back to bare `process.cwd()` when no `.git` ancestor exists - a script invoked from a fixture dir, worktree, or submodule silently re-materialized a fresh `.claude/harness-everything/state/` tree right there instead of in the real workspace. Converged onto a single global, workspace-keyed root (`~/.agents/harness-everything/workspaces/<slug-hash>/`, overridable via `HARNESS_STATE_HOME`), with one-time migration of any state a pre-fix install already scattered. The opencode plugin's own state was worse pre-fix (one flat `~/.harness-state` shared by every project on the machine - a circuit-breaker trip in one project hard-locked all the others); it now keys per workspace too. `fable-mode/scripts/model-selector.js` had zero workspace-root walk at all (the worst case in the issue's inventory) and now has one. Left alone, deliberately: five skill-owned CLI scripts (`persist-memory.js`, `register-dynamic-skill.js`, `audit-secrets.js`, `check-project-docs.js`, `evaluate.js`) that ship standalone into every install target and can't depend on the source repo's `scripts/lib/workspace.js` - their write targets are project artifacts by design, not ephemeral state. `ci/mechanism-2o-state-home-isolation.test.js` covers the issue's own repro plus the migration path.
|
|
186
|
+
- **Issue #42 adversarial corrections** (`scripts/lib/workspace.js`, `multi-agent-workspace/scripts/scaffold.js`, the standalone project writers, `find-skills/scripts/use-skill.js`, `opencode-plugin/index.mjs`, #42): no-context calls now use a stable unbound global bucket instead of cwd-derived identity, Windows case variants hash identically, non-git invocations refuse project writes, `.harness/multi-agent` and flat opencode state migrate even when destinations already exist, and find-skills uses the unified state home. `ci/mechanism-2u-issue-42-regressions.test.js` proves each regression against isolated fixtures.
|
|
187
|
+
- **Session-stable workspace identity and recoverable migration** (`scripts/lib/workspace.js`, `hooks/scripts/lib/harness-state.js`, `fable-mode/scripts/model-selector.js`, #42): hook payloads now bind a host/session identity to the first trusted workspace context so later cwd changes, nested repositories, symlink paths, and hostless follow-ups do not silently fork runtime state. Legacy platform trees are merged without overwriting newer destination files; conflicts and unsupported entries remain available for a later retry, while independent platform sources continue migrating. `ci/mechanism-2t-workspace-identity.test.js` covers host/session binding, cwd changes, conflicts, symlinks, and retry.
|
|
188
|
+
- **Self-repo detection** (`scripts/lib/workspace.js`): bootstrap and self-heal now identify the harness repo by `package.json` name instead of comparing against the running script's own directory, so an npx or global install no longer reports the harness checkout as drifted and recommends a repair that self-heal refuses to run. The audit also distinguishes an absent integration file from one that exists without the Harness advisory block.
|
|
189
|
+
- **opencode plugin, ported to the real API** (`opencode-plugin/index.mjs`, #37): the previous `plugin.json` manifest mapping `postEdit`/`preComplete` to standalone scripts was never invoked by opencode - it has no manifest-to-script mechanism and no such events. Replaced with a single ESM module exporting `tool.execute.before` (circuit-breaker hard lock), `tool.execute.after` (edit tracking) and `event` (verification gate on `session.idle`, which pushes a follow-up message via `client.session.prompt()` on failure since the event can't be blocked directly) - all three verified against opencode's plugin source. Reflection now has a tokenized report artifact lifecycle: code edits are blocked while pending, report writes are accepted through the real `(input, output)` hook arguments, read tools remain available, and repeated idle or failed prompt delivery are idempotent and retry-safe. State is isolated by workspace and session, unsafe IDs cannot escape reset containment, and the module contract and installation are documented. `ci/mechanism-2n-opencode-plugin.test.js` covers the full lifecycle, session isolation, reset safety, and hard-lock enforcement. Not yet verified: a live opencode session actually loading and firing the plugin (opencode requires Bun, not installable in this repo's environment).
|
|
190
|
+
- **opencode reflection enforcement hardening** (`opencode-plugin/index.mjs`, #37): reflection directives are now scoped to the `## Decision` section, corrupt breaker state fails closed, and `apply_patch` marker paths are recognized from `output.args.patchText`. The mechanism test covers each regression and restores repository-inventory parity for the remaining standalone helper.
|
|
191
|
+
- **Release baseline** (`ci/release-consistency-check.js`): the comparison tag is derived from the latest tag (or `--tag` / `HARNESS_RELEASE_TAG`) instead of being pinned to `v0.3.3-beta`, which had been three releases stale while still reporting PASSED.
|
|
192
|
+
|
|
193
|
+
### Changed
|
|
194
|
+
- **Release gate** (`.github/workflows/release.yml`): a tag push now runs the same job set as a pull request - including `test:references`, `test:release`, the waza skill-quality gate, and the windows matrix - instead of a weaker ubuntu-only subset.
|
|
195
|
+
- **Changelog/tag consistency** (`ci/consistency-check.js`): every released changelog heading must have a matching git tag; versions that were never tagged must say so in the heading.
|
|
196
|
+
|
|
197
|
+
### Documentation
|
|
198
|
+
- **Platform capability documentation consistency** (#78): centralized current enforcement/install/evidence claims in `docs/platform-capabilities.md`, aligned README/architecture/verification/mechanism/troubleshooting/audit/benchmark/agent/OpenCode documentation, and added `test:docs:capabilities` as a deterministic drift gate chained into `test:consistency` on Linux and Windows CI.
|
|
199
|
+
- **opencode plugin status** (`README.md`, `opencode-plugin/README.md`): updated to describe the real `index.mjs` plugin and its installation (drop into `.opencode/plugins/`, no `opencode.json` entry needed). opencode is now listed alongside Claude Code as a hard-enforcement platform, flagged as unverified in a live session pending #37's remaining step.
|
|
200
|
+
|
|
201
|
+
---
|
|
202
|
+
|
|
10
203
|
## [0.3.6] - 2026-09-01
|
|
11
204
|
|
|
12
205
|
### Added
|
|
@@ -25,10 +218,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
25
218
|
|
|
26
219
|
---
|
|
27
220
|
|
|
28
|
-
## [0.3.5-beta] - 2026-09-01
|
|
221
|
+
## [0.3.5-beta] - 2026-09-01 (unreleased - never tagged; shipped in 0.3.6)
|
|
29
222
|
|
|
30
223
|
### Added
|
|
31
|
-
- **Fable v3 model modes**: Added explicit Haiku, Sonnet, and Opus entrypoints, named agent distribution, a deterministic model matrix/selector, required stage audit records, and visible fallback or blocked escalation behavior.
|
|
224
|
+
- **Fable v3 model modes**: Added explicit Haiku, Sonnet, and Opus entrypoints, named agent distribution, a deterministic model matrix/selector, required stage audit records, and visible inline fallback or blocked escalation behavior.
|
|
32
225
|
- **Fable routing and contract coverage**: Added positive/negative mode routing evals and deterministic selector tests for model availability, alias normalization, audit persistence, and unsupported-runtime handling.
|
|
33
226
|
|
|
34
227
|
### Changed
|
|
@@ -37,7 +230,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
37
230
|
|
|
38
231
|
---
|
|
39
232
|
|
|
40
|
-
## [0.3.4-beta] - 2026-08-23
|
|
233
|
+
## [0.3.4-beta] - 2026-08-23 (unreleased - never tagged; shipped in 0.3.6)
|
|
41
234
|
|
|
42
235
|
### Added
|
|
43
236
|
- **A/B Eval Harness** (`eval-framework/ab-test-harness.js`): automated with/without skill comparison to prove method effectiveness. Takes a skill + test prompt, spawns baseline (no skill) and treatment (with skill) agent sessions, grades both against a rubric, and produces a verdict (EFFECTIVE / INCONCLUSIVE / INEFFECTIVE / HARMFUL). Run `npm run test:ab` (validate, free) or `npm run test:ab:run` (costs tokens). First run on gpt-5-mini: 4/4 INCONCLUSIVE — model already performs tested behaviors by default. Results in `benchmarks/ab-test-<date>/`.
|
package/CONTRIBUTING.md
CHANGED
|
@@ -4,23 +4,38 @@ Thanks for your interest in contributing.
|
|
|
4
4
|
|
|
5
5
|
## Development Setup
|
|
6
6
|
|
|
7
|
+
Harness supports **Node.js 22 or newer**. Node.js **24** is the primary development, CI, Docker, and release runtime; `.nvmrc` is the checked-in default. The generated current-state contract is [docs/repository-contract.md](docs/repository-contract.md).
|
|
8
|
+
|
|
7
9
|
```bash
|
|
8
10
|
git clone https://github.com/dyphn1/Harness-everything.git
|
|
9
11
|
cd Harness-everything
|
|
12
|
+
nvm use # uses Node 24 from .nvmrc when nvm is available
|
|
13
|
+
npm ci
|
|
10
14
|
npm test
|
|
11
15
|
```
|
|
12
16
|
|
|
13
17
|
## Quality Gates
|
|
14
18
|
|
|
15
|
-
|
|
19
|
+
The reusable CI workflow runs the repository's deterministic gates across the supported operating-system matrix, with a separate Node 22 minimum-runtime lane. Before opening a PR, run the relevant local gates:
|
|
16
20
|
|
|
17
21
|
```bash
|
|
18
|
-
npm test
|
|
19
|
-
npm run test:
|
|
20
|
-
npm run test:
|
|
21
|
-
npm run test:
|
|
22
|
+
npm test # self-regression
|
|
23
|
+
npm run test:mutations # focused mutation checks
|
|
24
|
+
npm run test:mechanism # hook/plugin mechanism checks
|
|
25
|
+
npm run test:consistency # manifests, docs, eval coverage + repository contract
|
|
26
|
+
npm run test:repo-contract # Node/workflow/action/schedule drift gate
|
|
27
|
+
npm run test:references # executable/deep-dive references
|
|
28
|
+
npm run test:release # release catalog + version automation contract
|
|
29
|
+
npm run test:collision # description collision detection
|
|
30
|
+
npm run test:plugin:openai # local OpenAI/Codex plugin package
|
|
31
|
+
npm run test:plugin:submission # public Skills-only package reproducibility
|
|
32
|
+
npm run test:platform:compatibility
|
|
22
33
|
```
|
|
23
34
|
|
|
35
|
+
If `waza` is installed, CI also validates canonical skill readiness, routing evals, and token growth. See [AGENTS.md](AGENTS.md) for the repository operating rules.
|
|
36
|
+
|
|
37
|
+
When runtime or workflow configuration changes, run `npm run docs:sync` and commit the regenerated [docs/repository-contract.md](docs/repository-contract.md). CI rejects stale generated contract data.
|
|
38
|
+
|
|
24
39
|
## Skill Contributions
|
|
25
40
|
|
|
26
41
|
Each skill lives in its own top-level directory with a `SKILL.md`. See [AGENTS.md](AGENTS.md) for full rules, but the non-negotiables:
|
|
@@ -31,12 +46,25 @@ Each skill lives in its own top-level directory with a `SKILL.md`. See [AGENTS.m
|
|
|
31
46
|
- Description must be unique (no collision with other skills)
|
|
32
47
|
- Skill version must not exceed package.json version
|
|
33
48
|
|
|
49
|
+
## Conventional Commits and Releases
|
|
50
|
+
|
|
51
|
+
Harness uses Conventional Commit history on `main` as the release input. Normal pull requests do **not** manually bump package or plugin versions.
|
|
52
|
+
|
|
53
|
+
- `BREAKING CHANGE:` or a breaking `!` => major release
|
|
54
|
+
- `feat:` => minor release
|
|
55
|
+
- `fix:`, `perf:`, `refactor:`, `build:`, `revert:` => patch release
|
|
56
|
+
- `docs:`, `test:`, `ci:`, `style:`, `chore:` => no release by themselves
|
|
57
|
+
|
|
58
|
+
If the repository uses squash merging, make the PR title a valid Conventional Commit because it becomes the commit semantic-release analyzes. See [RELEASING.md](RELEASING.md) for the complete automated release flow.
|
|
59
|
+
|
|
34
60
|
## Pull Requests
|
|
35
61
|
|
|
36
|
-
1. Fork and create a feature branch
|
|
37
|
-
2. Make changes and ensure all gates pass
|
|
38
|
-
3.
|
|
39
|
-
4.
|
|
62
|
+
1. Fork and create a feature branch.
|
|
63
|
+
2. Make changes and ensure all relevant gates pass.
|
|
64
|
+
3. Run `npm run docs:sync` after runtime/workflow changes and commit generated current-state documentation.
|
|
65
|
+
4. Update `CHANGELOG.md` under `[Unreleased]` for user-visible changes.
|
|
66
|
+
5. Do not manually bump release versions; release automation owns them.
|
|
67
|
+
6. Open a PR with a Conventional Commit title that accurately represents its release impact.
|
|
40
68
|
|
|
41
69
|
## Reporting Issues
|
|
42
70
|
|
package/Dockerfile
CHANGED
package/PRIVACY.md
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Privacy Notice
|
|
2
|
+
|
|
3
|
+
Harness Everything is distributed as an open-source, skill-only plugin for coding agents.
|
|
4
|
+
|
|
5
|
+
## Data handled by the plugin
|
|
6
|
+
|
|
7
|
+
Harness Everything does not operate a hosted backend, require an account with the project, or include a bundled third-party app/MCP connection. The packaged skills and hooks run in the host environment and may guide the coding agent to read files, write files, run development commands, and produce verification evidence in the workspace.
|
|
8
|
+
|
|
9
|
+
The project does not intentionally collect or transmit workspace content, prompts, source code, credentials, or usage telemetry to the Harness Everything maintainers.
|
|
10
|
+
|
|
11
|
+
## Host and third-party services
|
|
12
|
+
|
|
13
|
+
ChatGPT, Codex, GitHub, package registries, command-line tools, and any other services used by the host or explicitly invoked during a workflow have their own privacy practices. Installing Harness Everything does not change those services' permissions or privacy policies.
|
|
14
|
+
|
|
15
|
+
Users should review commands, tool permissions, connected apps, and repository content before allowing an agent to perform actions that may transmit data outside the local environment.
|
|
16
|
+
|
|
17
|
+
## Local state
|
|
18
|
+
|
|
19
|
+
Some Harness workflows may write local runtime state or project artifacts needed for routing, verification, or recovery. These files stay in locations controlled by the user and the host environment unless another tool or service is explicitly used to share them.
|
|
20
|
+
|
|
21
|
+
## Changes
|
|
22
|
+
|
|
23
|
+
If a future release adds a hosted service, telemetry, external app, or MCP integration that changes data handling, this notice must be updated before that capability is published.
|
|
24
|
+
|
|
25
|
+
For questions or reports, use the repository issue tracker: https://github.com/dyphn1/Harness-everything/issues
|