harness-everything 0.3.6 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plugins/marketplace.json +20 -0
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +8 -7
- package/.github/workflows/ci.yml +23 -18
- package/.github/workflows/release.yml +53 -27
- package/.releaserc.json +113 -0
- package/AGENTS.md +34 -23
- package/BENCHMARK_SOP.md +38 -18
- package/CHANGELOG.md +194 -3
- package/CONTRIBUTING.md +17 -4
- package/PRIVACY.md +25 -0
- package/README.md +118 -87
- package/RELEASING.md +82 -0
- package/SUPPORT.md +27 -0
- package/TERMS.md +21 -0
- package/VERIFICATION.md +175 -286
- package/behavioral-evals/README.md +21 -0
- package/behavioral-evals/case-validator.js +194 -0
- package/behavioral-evals/cases/baseline-environment-detection.yaml +21 -0
- package/behavioral-evals/cases/baseline-eval-harness.yaml +27 -0
- package/behavioral-evals/cases/baseline-fable-discipline.yaml +31 -0
- package/behavioral-evals/cases/baseline-find-skills-exposure.yaml +49 -0
- package/behavioral-evals/cases/baseline-find-skills.yaml +21 -0
- package/behavioral-evals/cases/baseline-git-commit.yaml +2 -2
- package/behavioral-evals/cases/baseline-install-cognitive-os.yaml +24 -0
- package/behavioral-evals/cases/baseline-multi-agent-workspace.yaml +28 -0
- package/behavioral-evals/cases/baseline-rewrite-commits.yaml +34 -0
- package/behavioral-evals/cases/baseline-self-evolve.yaml +31 -0
- package/behavioral-evals/cases/baseline-skill-creator.yaml +20 -0
- package/behavioral-evals/cases/baseline-skill-style.yaml +21 -0
- package/behavioral-evals/cases/baseline-to-spec.yaml +34 -0
- package/behavioral-evals/cases/baseline-to-tickets.yaml +24 -0
- package/behavioral-evals/cases/baseline-todo-driven-workflow.yaml +32 -0
- package/behavioral-evals/cases/baseline-using-git-worktrees.yaml +15 -0
- package/behavioral-evals/cases/breaker-zoom-out-after-3.yaml +2 -2
- package/behavioral-evals/cases/pressure-complexity-skip.yaml +2 -2
- package/behavioral-evals/cases/pressure-scope-bypass.yaml +7 -2
- package/behavioral-evals/cases/pressure-sunk-cost-retry.yaml +7 -4
- package/behavioral-evals/cases/scope-discipline.yaml +2 -1
- package/behavioral-evals/cases/verify-before-done.yaml +7 -5
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-debugging/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/case.yaml +34 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/historical-case.yaml +34 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-performance/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/historical-case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-security-review/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/historical-case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/baseline-simple-bugfix/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/historical-case.yaml +18 -0
- package/behavioral-evals/evidence/2026-09-07/grill-me-adversarial/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/live-run-blocker.md +16 -0
- package/behavioral-evals/evidence/2026-09-07/manifest.json +88 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/historical-case.yaml +28 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-scope-bypass/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/historical-case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-docs/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/case.yaml +35 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/historical-case.yaml +35 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-error-handling/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-23-pressure-skip-verification/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/historical-case.yaml +43 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-skip-verification--2026-08-27-pressure-skip-verification/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-23-pressure-sunk-cost-retry/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/case.yaml +50 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/historical-case.yaml +47 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/pressure-sunk-cost-retry--2026-08-27-pressure-sunk-cost-retry/result.sanitized.json +24 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/case.yaml +30 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/historical-case.yaml +23 -0
- package/behavioral-evals/evidence/2026-09-07/tdd-test-first/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/triage-matrix.json +426 -0
- package/behavioral-evals/evidence/2026-09-07/triage-matrix.md +24 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/case.yaml +19 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/historical-case.yaml +18 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-claim-cites/provenance.json +30 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-23-verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/historical-case.yaml +31 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-07/verify-before-done--2026-08-27-verify-before-done/result.sanitized.json +20 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/historical-case.yaml +33 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/provenance.json +42 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/result.sanitized.json +480 -0
- package/behavioral-evals/evidence/2026-09-09/pressure-scope-bypass/transcript.sanitized.jsonl +95 -0
- package/behavioral-evals/evidence-tool.js +443 -0
- package/behavioral-evals/fixtures/claude-denied.jsonl +3 -0
- package/behavioral-evals/fixtures/claude-legacy.json +1 -0
- package/behavioral-evals/fixtures/claude-oral-only.jsonl +2 -0
- package/behavioral-evals/fixtures/claude-success.jsonl +5 -0
- package/behavioral-evals/fixtures/claude-truncated.jsonl +3 -0
- package/behavioral-evals/fixtures/opencode-success.jsonl +6 -0
- package/behavioral-evals/results/2026-08-31-baseline-add-feature.json +20 -0
- package/behavioral-evals/results/2026-09-03-baseline-environment-detection-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-eval-harness-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-fable-discipline-pair.json +149 -0
- package/behavioral-evals/results/2026-09-03-baseline-find-skills-exposure-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-find-skills-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-install-cognitive-os-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-multi-agent-workspace-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-rewrite-commits-pair.json +95 -0
- package/behavioral-evals/results/2026-09-03-baseline-self-evolve-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-skill-creator-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-skill-style-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-to-spec-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-baseline-to-tickets-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-todo-driven-workflow-pair.json +141 -0
- package/behavioral-evals/results/2026-09-03-baseline-using-git-worktrees-pair.json +133 -0
- package/behavioral-evals/results/2026-09-03-summary-both.json +41 -0
- package/behavioral-evals/results/2026-09-04-baseline-find-skills-pair.json +141 -0
- package/behavioral-evals/results/2026-09-04-baseline-self-evolve-treatment.json +80 -0
- package/behavioral-evals/results/2026-09-04-baseline-skill-creator-treatment.json +80 -0
- package/behavioral-evals/results/2026-09-04-summary-both.json +41 -0
- package/behavioral-evals/run-with-plugin.js +15 -31
- package/behavioral-evals/run.js +446 -105
- package/behavioral-evals/transcript-parser.js +591 -0
- package/bin/cli.js +24 -12
- package/ci/ab-test-cases/grill-me-adversarial.yaml +14 -4
- package/ci/ab-test-cases/tdd-test-first.yaml +13 -6
- package/ci/ab-test-cases/verification-loop-gates.yaml +20 -7
- package/ci/ab-test-cases/verify-before-claim-cites.yaml +7 -6
- package/ci/ab-test-harness.js +113 -38
- package/ci/action-gate-platform-doc.test.js +31 -0
- package/ci/action-gate.test.js +260 -0
- package/ci/cases/case2.json +2 -2
- package/ci/consistency-check.js +51 -5
- package/ci/description-collision.js +15 -6
- package/ci/disclosure-check.js +225 -0
- package/ci/doc-capability-consistency.test.js +180 -0
- package/ci/fixtures/phase4/parallel-disjoint.stages.json +32 -0
- package/ci/fixtures/phase4/parallel-read-only.stages.json +32 -0
- package/ci/installer-roundtrip.test.js +159 -0
- package/ci/invariant-routing.test.js +74 -0
- package/ci/issue99-runtime.test.js +218 -0
- package/ci/mechanism-2aa-codex-action-gate-policy.test.js +117 -0
- package/ci/mechanism-2ab-openai-plugin-sync-integrity.test.js +80 -0
- package/ci/mechanism-2j-behavioral-runner.test.js +43 -1
- package/ci/mechanism-2k-reference-check.test.js +57 -2
- package/ci/mechanism-2l-multi-agent-workspace.test.js +202 -6
- package/ci/mechanism-2m-self-repo-guard.test.js +65 -0
- package/ci/mechanism-2n-opencode-plugin.test.js +461 -0
- package/ci/mechanism-2o-state-home-isolation.test.js +183 -0
- package/ci/mechanism-2p-installer-links.test.js +183 -0
- package/ci/mechanism-2q-installer-global-shared-store.test.js +132 -0
- package/ci/mechanism-2r-uninstall-flags.test.js +148 -0
- package/ci/mechanism-2s-tdd-quality.test.js +203 -0
- package/ci/mechanism-2t-disclosure.test.js +90 -0
- package/ci/mechanism-2t-evidence-triage.test.js +200 -0
- package/ci/mechanism-2t-issue-20.test.js +75 -0
- package/ci/mechanism-2t-platform-skill-targets.test.js +164 -0
- package/ci/mechanism-2t-transcript-parser.test.js +179 -0
- package/ci/mechanism-2t-workspace-identity.test.js +200 -0
- package/ci/mechanism-2u-issue-42-regressions.test.js +239 -0
- package/ci/mechanism-2u-issue-44.test.js +35 -0
- package/ci/mechanism-2v-verifier-boundaries.test.js +138 -0
- package/ci/mechanism-2w-platform-compatibility.test.js +148 -0
- package/ci/mechanism-2x-release-automation.test.js +157 -0
- package/ci/mechanism-2x-release-commit-message.test.js +66 -0
- package/ci/mechanism-2y-release-tooling-compat.test.js +47 -0
- package/ci/mechanism-2z-claude-plugin-manifest.test.js +29 -0
- package/ci/mutation-check.js +53 -0
- package/ci/mutation-loader.js +57 -0
- package/ci/negative-controls.js +76 -46
- package/ci/openai-plugin-package.test.js +182 -0
- package/ci/openai-plugin-runtime.test.js +175 -0
- package/ci/openai-plugin-submission.test.js +101 -0
- package/ci/phase4-ensemble-memory.test.js +307 -0
- package/ci/reference-check.js +73 -25
- package/ci/release-consistency-check.js +24 -1
- package/ci/router-phase2-safety.test.js +81 -0
- package/ci/router-workflow-plan.test.js +264 -0
- package/ci/routing-check.js +6 -5
- package/ci/runner.js +6 -5
- package/ci/skill-routing-check.js +48 -11
- package/ci/test-behavioral-case-validation.js +62 -0
- package/ci/test-helper.js +10 -0
- package/ci/workflow-plan-consumers-edge.test.js +72 -0
- package/ci/workflow-plan-consumers.test.js +276 -0
- package/docs/architecture.md +137 -57
- package/docs/audit.md +22 -20
- package/docs/codex-action-gate.md +90 -0
- package/docs/issue-20-rollup.md +47 -0
- package/docs/mechanism-first-skill-mesh.md +22 -20
- package/docs/openai-plugin.md +118 -0
- package/docs/philosophy.md +28 -17
- package/docs/platform-capabilities.md +73 -0
- package/docs/platform-compatibility.json +270 -0
- package/docs/reflection.md +37 -22
- package/docs/release-evidence.md +25 -0
- package/docs/routing.md +157 -44
- package/docs/security-audit.md +38 -0
- package/docs/troubleshooting.md +82 -46
- package/docs/verifier-boundaries.md +45 -0
- package/docs/workflows/fable-mode.md +77 -21
- package/docs/workflows/find-skills.md +40 -0
- package/docs/workflows/grill-with-docs.md +2 -2
- package/docs/workflows/multi-agent-workspace.md +22 -10
- package/docs/workflows/self-evolve.md +33 -0
- package/docs/workflows/tdd.md +36 -0
- package/docs/workflows/todo-driven-workflow.md +33 -0
- package/environment-detection/SKILL.md +3 -3
- package/eval-harness/SKILL.md +7 -7
- package/eval-harness/scripts/evaluate.js +12 -10
- package/evals/evaluation-reports.md +14 -0
- package/evals/harness-everything/tasks/positive-tier1.yaml +1 -1
- package/evals/harness-everything/tasks/positive-tier2.yaml +1 -1
- package/evals/harness-everything/tasks/positive-tier3.yaml +1 -1
- package/evals/install-cognitive-os/tasks/positive-trigger.yaml +1 -1
- package/fable-discipline/SKILL.md +2 -2
- package/fable-mode/CONTRACT-FORMAT.md +147 -33
- package/fable-mode/SKILL.md +3 -3
- package/fable-mode/agents/fable-orchestrator.md +95 -35
- package/fable-mode/execution-guardrails/SKILL.md +2 -2
- package/fable-mode/fable-haiku/SKILL.md +3 -3
- package/fable-mode/fable-opus/SKILL.md +3 -3
- package/fable-mode/fable-sonnet/SKILL.md +3 -3
- package/fable-mode/scripts/ensemble-review.js +371 -0
- package/fable-mode/scripts/model-selector.js +134 -3
- package/fable-mode/scripts/workflow-plan-consumer.js +337 -0
- package/find-skills/SKILL.md +5 -5
- package/find-skills/references/discovery-flow.md +1 -1
- package/find-skills/scripts/use-skill.js +7 -1
- package/git-commit/SKILL.md +4 -11
- package/grill-me/SKILL.md +5 -5
- package/grill-with-docs/SKILL.md +7 -7
- package/grill-with-docs/references/session-playbook.md +6 -4
- package/harness-everything/SKILL.md +23 -22
- package/harness-everything/references/router-workflow-plan.md +128 -0
- package/harness-everything/references/triage-and-tiers.md +173 -96
- package/harness-everything/schemas/router-task-shape.schema.json +126 -0
- package/harness-everything/schemas/router-workflow-plan.schema.json +149 -0
- package/harness-everything/scripts/bootstrap.js +13 -4
- package/harness-everything/scripts/ensemble-policy.js +102 -0
- package/harness-everything/scripts/kernel-router.js +212 -0
- package/harness-everything/scripts/router-contract.js +612 -0
- package/harness-everything/scripts/self-heal.js +29 -20
- package/harness-everything/scripts/tier-router.js +346 -221
- package/hooks/hooks.json +60 -15
- package/hooks/scripts/action-gate-rules.json +77 -0
- package/hooks/scripts/action-gate.js +528 -0
- package/hooks/scripts/atomic-commit-check.js +5 -5
- package/hooks/scripts/codex-action-gate-post.js +46 -0
- package/hooks/scripts/codex-permission-request.js +140 -0
- package/hooks/scripts/context-compact.js +4 -3
- package/hooks/scripts/contract-test.js +118 -51
- package/hooks/scripts/lib/fable-contracts.js +121 -0
- package/hooks/scripts/lib/harness-state.js +164 -28
- package/hooks/scripts/lib/platforms/claude.js +45 -62
- package/hooks/scripts/lib/platforms/codex.js +35 -61
- package/hooks/scripts/lib/platforms/continue.js +7 -7
- package/hooks/scripts/lib/platforms/copilot.js +29 -53
- package/hooks/scripts/lib/platforms/hermes.js +71 -20
- package/hooks/scripts/openai-session-start.js +81 -0
- package/hooks/scripts/rule-of-3-tracker.js +9 -8
- package/hooks/scripts/rule-of-3.js +29 -21
- package/hooks/scripts/state-persist.js +5 -4
- package/hooks/scripts/stop-gate.js +2 -2
- package/hooks/scripts/subagent-scope-guard.js +144 -38
- package/hooks/scripts/subagent-start.js +19 -0
- package/hooks/scripts/subagent-stop.js +6 -0
- package/improve-codebase-architecture/SKILL.md +2 -4
- package/improve-codebase-architecture/references/README.md +16 -0
- package/install-cognitive-os/SKILL.md +28 -22
- package/multi-agent-workspace/SKILL.md +16 -15
- package/multi-agent-workspace/references/architecture-guide.md +13 -9
- package/multi-agent-workspace/references/orchestration.md +39 -22
- package/multi-agent-workspace/scripts/consume-workflow-plan.js +129 -0
- package/multi-agent-workspace/scripts/index_memory.js +79 -26
- package/multi-agent-workspace/scripts/path-boundary.js +59 -0
- package/multi-agent-workspace/scripts/project-docs-resolver.js +357 -0
- package/multi-agent-workspace/scripts/scaffold.js +258 -103
- package/multi-agent-workspace/templates/AGENTS.md +20 -0
- package/multi-agent-workspace/workflows/03-generation.md +8 -10
- package/multi-agent-workspace/workflows/04-launcher.md +5 -5
- package/opencode-plugin/README.md +114 -71
- package/opencode-plugin/hooks/verify.js +62 -12
- package/opencode-plugin/index.mjs +547 -0
- package/opencode-plugin/plugin.json +4 -12
- package/package.json +16 -3
- package/plugins/harness-everything/.codex-plugin/plugin.json +39 -0
- package/plugins/harness-everything/hooks/hooks.json +197 -0
- package/plugins/harness-everything/hooks/scripts/action-gate-rules.json +77 -0
- package/plugins/harness-everything/hooks/scripts/action-gate.js +528 -0
- package/plugins/harness-everything/hooks/scripts/atomic-commit-check.js +68 -0
- package/plugins/harness-everything/hooks/scripts/boundary-guard.js +55 -0
- package/plugins/harness-everything/hooks/scripts/codex-action-gate-post.js +46 -0
- package/plugins/harness-everything/hooks/scripts/codex-permission-request.js +140 -0
- package/plugins/harness-everything/hooks/scripts/context-compact.js +115 -0
- package/plugins/harness-everything/hooks/scripts/contract-test.js +146 -0
- package/plugins/harness-everything/hooks/scripts/depth-guard.js +65 -0
- package/plugins/harness-everything/hooks/scripts/lib/fable-contracts.js +121 -0
- package/plugins/harness-everything/hooks/scripts/lib/harness-state.js +275 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/claude.js +168 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/codex.js +88 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/continue.js +91 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/copilot.js +85 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/cursor.js +89 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/hermes.js +108 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/index.js +17 -0
- package/plugins/harness-everything/hooks/scripts/lib/platforms/worktrees.js +20 -0
- package/plugins/harness-everything/hooks/scripts/lib/workspace.js +290 -0
- package/plugins/harness-everything/hooks/scripts/openai-session-start.js +81 -0
- package/plugins/harness-everything/hooks/scripts/reset-circuit-breaker.js +30 -0
- package/plugins/harness-everything/hooks/scripts/rule-of-3-tracker.js +125 -0
- package/plugins/harness-everything/hooks/scripts/rule-of-3.js +195 -0
- package/plugins/harness-everything/hooks/scripts/state-persist.js +100 -0
- package/plugins/harness-everything/hooks/scripts/stop-gate.js +80 -0
- package/plugins/harness-everything/hooks/scripts/subagent-scope-guard.js +194 -0
- package/plugins/harness-everything/hooks/scripts/subagent-start.js +19 -0
- package/plugins/harness-everything/hooks/scripts/subagent-stop.js +6 -0
- package/plugins/harness-everything/hooks/session-start.js +81 -0
- package/plugins/harness-everything/plugin.json +43 -0
- package/plugins/harness-everything/scripts/lib/advisory-text.js +174 -0
- package/plugins/harness-everything/skills/environment-detection/SKILL.md +41 -0
- package/plugins/harness-everything/skills/environment-detection/references/shell-syntax-rules.md +76 -0
- package/plugins/harness-everything/skills/environment-detection/scripts/preflight.js +103 -0
- package/plugins/harness-everything/skills/eval-harness/SKILL.md +42 -0
- package/plugins/harness-everything/skills/eval-harness/references/scoring-rubric.md +68 -0
- package/plugins/harness-everything/skills/eval-harness/scripts/evaluate.js +79 -0
- package/plugins/harness-everything/skills/fable-discipline/SKILL.md +41 -0
- package/plugins/harness-everything/skills/fable-discipline/references/discipline-rules.md +22 -0
- package/plugins/harness-everything/skills/fable-mode/CONTRACT-FORMAT.md +165 -0
- package/plugins/harness-everything/skills/fable-mode/SKILL.md +39 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-orchestrator.md +159 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-verifier.md +32 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-haiku.md +52 -0
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-sonnet.md +53 -0
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/SKILL.md +43 -0
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/references/guardrail-rules.md +56 -0
- package/plugins/harness-everything/skills/fable-mode/fable-haiku/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/fable-haiku/references/routing-notes.md +26 -0
- package/plugins/harness-everything/skills/fable-mode/fable-opus/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/fable-sonnet/SKILL.md +36 -0
- package/plugins/harness-everything/skills/fable-mode/model-matrix.json +32 -0
- package/plugins/harness-everything/skills/fable-mode/references/execution-phases.md +37 -0
- package/plugins/harness-everything/skills/fable-mode/references/model-matrix.md +39 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/ensemble-review.js +371 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/model-selector.js +260 -0
- package/plugins/harness-everything/skills/fable-mode/scripts/workflow-plan-consumer.js +337 -0
- package/plugins/harness-everything/skills/find-skills/SKILL.md +48 -0
- package/plugins/harness-everything/skills/find-skills/references/discovery-flow.md +102 -0
- package/plugins/harness-everything/skills/find-skills/scripts/use-skill.js +112 -0
- package/plugins/harness-everything/skills/git-commit/SKILL.md +38 -0
- package/plugins/harness-everything/skills/git-commit/guides/ANGULAR_STYLE.md +61 -0
- package/plugins/harness-everything/skills/git-commit/guides/COMMIT_GENERATION.md +24 -0
- package/plugins/harness-everything/skills/git-commit/guides/LANGUAGE_DETECTION.md +16 -0
- package/plugins/harness-everything/skills/git-commit/guides/MAIN_REPO.md +15 -0
- package/plugins/harness-everything/skills/git-commit/guides/SUBMODULES.md +27 -0
- package/plugins/harness-everything/skills/git-commit/references/commit-flow.md +30 -0
- package/plugins/harness-everything/skills/grill-me/SKILL.md +43 -0
- package/plugins/harness-everything/skills/grill-me/references/grilling-playbook.md +38 -0
- package/plugins/harness-everything/skills/grill-with-docs/ADR-FORMAT.md +47 -0
- package/plugins/harness-everything/skills/grill-with-docs/CONTEXT-FORMAT.md +63 -0
- package/plugins/harness-everything/skills/grill-with-docs/SKILL.md +41 -0
- package/plugins/harness-everything/skills/grill-with-docs/references/session-playbook.md +99 -0
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +43 -0
- package/plugins/harness-everything/skills/harness-everything/examples/README.md +9 -0
- package/plugins/harness-everything/skills/harness-everything/examples/todo-workflow.md +22 -0
- package/plugins/harness-everything/skills/harness-everything/references/router-workflow-plan.md +128 -0
- package/plugins/harness-everything/skills/harness-everything/references/skill-registry.md +39 -0
- package/plugins/harness-everything/skills/harness-everything/references/triage-and-tiers.md +193 -0
- package/plugins/harness-everything/skills/harness-everything/schemas/router-task-shape.schema.json +126 -0
- package/plugins/harness-everything/skills/harness-everything/schemas/router-workflow-plan.schema.json +149 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/bootstrap.js +212 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/ensemble-policy.js +102 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +212 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/router-contract.js +612 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/routing-keywords.json +320 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/self-heal.js +250 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/tier-router.js +459 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/verify-gate.js +100 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/SKILL.md +38 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/DEEPENING.md +37 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/HTML-REPORT.md +123 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/INTERFACE-DESIGN.md +44 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/LANGUAGE.md +53 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/guides/refactoring-workflow.md +59 -0
- package/plugins/harness-everything/skills/improve-codebase-architecture/references/README.md +16 -0
- package/plugins/harness-everything/skills/install-cognitive-os/SKILL.md +44 -0
- package/plugins/harness-everything/skills/install-cognitive-os/references/cognitive-loop.md +27 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/NOTICE-agency-agents.md +27 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/SKILL.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/examples/csharp-monorepo.md +123 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/examples/typescript-cli.md +126 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-claude.md +77 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-codex.md +146 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-continue.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-copilot.md +74 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-cursor.md +40 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/platform-hermes.md +38 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/guidelines/role-capabilities.md +108 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/agency-agents.md +31 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/architecture-guide.md +36 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/references/orchestration.md +65 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/agency-catalog.js +190 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/consume-workflow-plan.js +129 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/index_memory.js +104 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/path-boundary.js +59 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/project-docs-resolver.js +357 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/scaffold.js +408 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/AGENTS.md +20 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/README.md +17 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/agent-launcher.md +68 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-csharp.md +56 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-polyglot.md +62 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/backend-developer-ts.md +57 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/memory-keeper.md +37 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/requirement-analyzer.md +66 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/specialist-agent.md +34 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/task-verifier.md +56 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/templates/tool-maker.md +33 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/01-init.md +22 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/02-analysis.md +23 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/03-generation.md +25 -0
- package/plugins/harness-everything/skills/multi-agent-workspace/workflows/04-launcher.md +16 -0
- package/plugins/harness-everything/skills/repo-docs/SKILL.md +42 -0
- package/plugins/harness-everything/skills/repo-docs/references/process-guide.md +60 -0
- package/plugins/harness-everything/skills/repo-docs/templates/agents-template.md +264 -0
- package/plugins/harness-everything/skills/repo-docs/templates/knowledge-agents-template.md +50 -0
- package/plugins/harness-everything/skills/repo-docs/templates/knowledge-readme-template.md +83 -0
- package/plugins/harness-everything/skills/repo-docs/templates/manual-recon.md +117 -0
- package/plugins/harness-everything/skills/repo-docs/templates/multi-skills-readme-template.md +72 -0
- package/plugins/harness-everything/skills/repo-docs/templates/product-agents-template.md +78 -0
- package/plugins/harness-everything/skills/repo-docs/templates/product-readme-template.md +95 -0
- package/plugins/harness-everything/skills/repo-docs/templates/readme-template.md +136 -0
- package/plugins/harness-everything/skills/repo-docs/templates/single-skill-readme-template.md +70 -0
- package/plugins/harness-everything/skills/repo-docs/templates/skills-agents-template.md +60 -0
- package/plugins/harness-everything/skills/rewrite-commits/SKILL.md +37 -0
- package/plugins/harness-everything/skills/rewrite-commits/references/workflow.md +18 -0
- package/plugins/harness-everything/skills/security-review/SKILL.md +47 -0
- package/plugins/harness-everything/skills/security-review/guides/OWASP-PATTERNS.md +31 -0
- package/plugins/harness-everything/skills/security-review/guides/STRIDE-THREAT-MODEL.md +21 -0
- package/plugins/harness-everything/skills/security-review/references/security-checklist.md +478 -0
- package/plugins/harness-everything/skills/security-review/scripts/audit-secrets.js +85 -0
- package/plugins/harness-everything/skills/self-evolve/SKILL.md +45 -0
- package/plugins/harness-everything/skills/self-evolve/references/memory-resolution.md +61 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/persist-memory.js +272 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/register-dynamic-skill.js +253 -0
- package/plugins/harness-everything/skills/self-evolve/scripts/self-regression.js +125 -0
- package/plugins/harness-everything/skills/skill-creator/SKILL.md +47 -0
- package/plugins/harness-everything/skills/skill-creator/references/authoring-workflow.md +14 -0
- package/plugins/harness-everything/skills/skill-creator/references/dynamic-generation-contract.md +21 -0
- package/plugins/harness-everything/skills/skill-creator/references/quality-checklist.md +16 -0
- package/plugins/harness-everything/skills/skill-creator/references/quality-principles.md +81 -0
- package/plugins/harness-everything/skills/skill-creator/references/testing-workflow.md +27 -0
- package/plugins/harness-everything/skills/skill-style/SKILL.md +42 -0
- package/plugins/harness-everything/skills/skill-style/references/style-guide.md +62 -0
- package/plugins/harness-everything/skills/tdd/SKILL.md +38 -0
- package/plugins/harness-everything/skills/tdd/fixtures/pass-evidence.json +100 -0
- package/plugins/harness-everything/skills/tdd/fixtures/quality-scenarios.json +20 -0
- package/plugins/harness-everything/skills/tdd/guides/deep-modules.md +33 -0
- package/plugins/harness-everything/skills/tdd/guides/interface-design.md +31 -0
- package/plugins/harness-everything/skills/tdd/guides/mocking.md +59 -0
- package/plugins/harness-everything/skills/tdd/guides/refactoring.md +10 -0
- package/plugins/harness-everything/skills/tdd/guides/tests.md +61 -0
- package/plugins/harness-everything/skills/tdd/references/core-discipline.md +51 -0
- package/plugins/harness-everything/skills/tdd/references/integration-testing.md +13 -0
- package/plugins/harness-everything/skills/tdd/references/quality-model.md +68 -0
- package/plugins/harness-everything/skills/tdd/references/unit-testing.md +15 -0
- package/plugins/harness-everything/skills/tdd/schemas/quality-evidence.schema.json +211 -0
- package/plugins/harness-everything/skills/tdd/schemas/quality-report.schema.json +110 -0
- package/plugins/harness-everything/skills/tdd/scripts/quality-gate.js +475 -0
- package/plugins/harness-everything/skills/to-spec/SKILL.md +40 -0
- package/plugins/harness-everything/skills/to-spec/references/process.md +77 -0
- package/plugins/harness-everything/skills/to-spec/scripts/check-project-docs.js +239 -0
- package/plugins/harness-everything/skills/to-spec/scripts/project-docs-resolver.js +111 -0
- package/plugins/harness-everything/skills/to-spec/templates/cli-reference.md +31 -0
- package/plugins/harness-everything/skills/to-spec/templates/dev-doc.md +27 -0
- package/plugins/harness-everything/skills/to-spec/templates/feature-spec.md +59 -0
- package/plugins/harness-everything/skills/to-spec/templates/schema-doc.md +31 -0
- package/plugins/harness-everything/skills/to-tickets/SKILL.md +42 -0
- package/plugins/harness-everything/skills/to-tickets/references/publishing-and-templates.md +97 -0
- package/plugins/harness-everything/skills/todo-driven-workflow/SKILL.md +42 -0
- package/plugins/harness-everything/skills/todo-driven-workflow/references/execution-guide.md +27 -0
- package/plugins/harness-everything/skills/using-git-worktrees/SKILL.md +49 -0
- package/plugins/harness-everything/skills/using-git-worktrees/references/workflow-details.md +108 -0
- package/plugins/harness-everything/skills/verification-loop/SKILL.md +46 -0
- package/plugins/harness-everything/skills/verification-loop/references/verification-phases.md +116 -0
- package/plugins/harness-everything/skills/verification-loop/templates/verification-report.template.md +15 -0
- package/plugins/harness-everything/skills/verify-before-claim/SKILL.md +43 -0
- package/plugins/harness-everything/skills/verify-before-claim/references/verification-guide.md +73 -0
- package/plugins/harness-everything/skills/zoom-out/SKILL.md +42 -0
- package/plugins/harness-everything/skills/zoom-out/references/circuit-breaker.md +66 -0
- package/plugins/harness-everything/skills/zoom-out/templates/zoom-out-report.template.md +20 -0
- package/repo-docs/SKILL.md +3 -7
- package/rewrite-commits/SKILL.md +2 -2
- package/scripts/build-openai-submission.js +158 -0
- package/scripts/installer.js +86 -11
- package/scripts/lib/advisory-text.js +25 -18
- package/scripts/lib/execution-contract.js +70 -0
- package/scripts/lib/manifest.js +34 -38
- package/scripts/lib/path-boundary.js +90 -0
- package/scripts/lib/skills.js +124 -70
- package/scripts/lib/workspace.js +269 -7
- package/scripts/sync-openai-plugin.js +91 -0
- package/scripts/sync-release-version.js +212 -0
- package/security-review/SKILL.md +4 -7
- package/security-review/scripts/audit-secrets.js +9 -8
- package/self-evolve/SKILL.md +9 -11
- package/self-evolve/references/memory-resolution.md +4 -1
- package/self-evolve/scripts/persist-memory.js +221 -88
- package/self-evolve/scripts/register-dynamic-skill.js +12 -10
- package/self-evolve/scripts/self-regression.js +23 -8
- package/skill-creator/SKILL.md +12 -14
- package/skill-creator/references/quality-checklist.md +1 -0
- package/skill-style/SKILL.md +4 -4
- package/skill-style/references/style-guide.md +29 -3
- package/submission/openai/README.md +67 -0
- package/submission/openai/listing.json +31 -0
- package/submission/openai/test-cases.json +61 -0
- package/tdd/SKILL.md +20 -26
- package/tdd/fixtures/pass-evidence.json +100 -0
- package/tdd/fixtures/quality-scenarios.json +20 -0
- package/tdd/references/core-discipline.md +2 -0
- package/tdd/references/integration-testing.md +13 -0
- package/tdd/references/quality-model.md +68 -0
- package/tdd/references/unit-testing.md +15 -0
- package/tdd/schemas/quality-evidence.schema.json +211 -0
- package/tdd/schemas/quality-report.schema.json +110 -0
- package/tdd/scripts/quality-gate.js +475 -0
- package/to-spec/SKILL.md +3 -5
- package/to-spec/scripts/check-project-docs.js +42 -62
- package/to-spec/scripts/project-docs-resolver.js +111 -0
- package/to-tickets/SKILL.md +4 -4
- package/todo-driven-workflow/SKILL.md +6 -4
- package/using-git-worktrees/SKILL.md +3 -3
- package/verification-loop/SKILL.md +4 -4
- package/verify-before-claim/SKILL.md +2 -2
- package/zoom-out/SKILL.md +4 -4
- package/ab-test-deep-analysis-2026-08-27.md +0 -383
- package/ab-test-results-2026-08-27.md +0 -152
- package/ab-test-statistical-analysis-31cases.md +0 -193
- package/opencode-plugin/hooks/circuit-breaker.js +0 -121
- package/opencode-plugin/hooks/compliance.js +0 -128
- package/opencode-plugin/hooks/post-edit.js +0 -62
- package/opencode-plugin/hooks/pre-complete.js +0 -65
- package/opencode-plugin/test-hook-firing.js +0 -198
package/docs/audit.md
CHANGED
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Maintainer-facing audit record. If you just want to know whether Harness works and how to verify it yourself, start at [VERIFICATION.md](../VERIFICATION.md) instead — this file records *how the scores below were obtained* and what changed between audit cycles.
|
|
4
4
|
|
|
5
|
+
> **Historical snapshot boundary:** the scorecard below is the 2026-09-01 audit and is intentionally preserved as evidence of what was measured on that date. Platform capabilities have changed since then (including the OpenAI/Codex local plugin work and the OpenCode plugin implementation). Do not treat the 2026-09-01 platform rows as current-state documentation; use [platform-capabilities.md](platform-capabilities.md) for the current matrix.
|
|
6
|
+
|
|
5
7
|
**Last self-audited: 2026-09-01**, by running the actual test suite and the VERIFICATION.md recipes on Windows 11 — not by reading the code and assuming it works.
|
|
6
8
|
|
|
7
9
|
## The 2026-07-23 mis-measurement incident
|
|
@@ -14,40 +16,40 @@ This incident is why VERIFICATION.md's recipes were rewritten to be Windows-safe
|
|
|
14
16
|
|
|
15
17
|
| Criterion | Score | Verified basis |
|
|
16
18
|
|---|---|---|
|
|
17
|
-
| **Skill Description Completeness** | 10/10 | All 26
|
|
18
|
-
| **Routing Accuracy** | 9/10 | The local router
|
|
19
|
-
| **Test Coverage of All Skills** | 9.5/10 | `npm test`
|
|
20
|
-
| **Configuration Balance** | 8.5/10 |
|
|
21
|
-
| **Workflow Conformance** | 8.5/10 | Every
|
|
19
|
+
| **Skill Description Completeness** | 10/10 | All 26 skill directories present in the 2026-09-01 snapshot had a complete four-row Skill Contract and the required trigger sections. Four PreToolUse hooks remained internal mechanisms, not standalone skills. |
|
|
20
|
+
| **Routing Accuracy** | 9/10 | The local router passed 34/34 positive prompts across all 26 skills in that snapshot, including the foundation and meta routes added in the audit. Waza verification was not runnable in that Windows checkout; ambiguous/metaphorical generalization remained unmeasured. |
|
|
21
|
+
| **Test Coverage of All Skills** | 9.5/10 | `npm test` included syntax, CLI, routing matrix, positive route coverage, references, behavioral-case structure, Fable contract, and mechanism checks at the time. Waza and live model sessions remained separate CI/on-demand evidence. |
|
|
22
|
+
| **Configuration Balance** | 8.5/10 | **Historical 2026-09-01 result:** hard `exit(2)` blocking hooks were measured on Claude Code and the other then-installed platform paths were evaluated as advisory. This row predates the later OpenAI/Codex local plugin and OpenCode plugin work; see `docs/platform-capabilities.md` for current status. |
|
|
23
|
+
| **Workflow Conformance** | 8.5/10 | Every skill in the snapshot had a matching workflow document, and `ci/skill-routing-check.js` executed every positive route case against the real router. Runtime tool-call sequence conformance remained host-dependent and was not claimed. |
|
|
22
24
|
|
|
23
25
|
## Overall Scorecard
|
|
24
26
|
|
|
25
27
|
| Category | Score | Notes |
|
|
26
28
|
|---|---|---|
|
|
27
|
-
| **Architecture** | 9/10 | Per-session state isolation, per-platform state directories, fail-open-by-default hooks — all re-verified directly against `hooks/scripts/lib/harness-state.js
|
|
28
|
-
| **Test Coverage** | 9/10 |
|
|
29
|
-
| **README Completeness** | 9.5/10 | Catalog counts, layer labels, route coverage, and the local baseline
|
|
30
|
-
| **Maintainability** | 9/10 | The custom cross-session todo CLI and obsolete multi-agent compatibility stubs
|
|
31
|
-
| **Skills Design** | 9.5/10 | All 26
|
|
32
|
-
| **Agent Compatibility** | 9/10 |
|
|
33
|
-
| **Beginner Friendliness** | 7.5/10 | Quick Start
|
|
29
|
+
| **Architecture** | 9/10 | Per-session state isolation, per-platform state directories, fail-open-by-default hooks — all re-verified directly against `hooks/scripts/lib/harness-state.js` in the 2026-09-01 audit. |
|
|
30
|
+
| **Test Coverage** | 9/10 | Deterministic checks covered syntax, routing, skill references, release-tag catalog, install tree/version drift, behavioral-case structure, and 12 mechanism suites at the time. Live model evaluation remained on-demand and was not represented by a local simulation. |
|
|
31
|
+
| **README Completeness** | 9.5/10 | Catalog counts, layer labels, route coverage, and the local baseline matched the then-current 26-skill tree. |
|
|
32
|
+
| **Maintainability** | 9/10 | The custom cross-session todo CLI and obsolete multi-agent compatibility stubs were absent; route coverage was a deterministic regression gate. |
|
|
33
|
+
| **Skills Design** | 9.5/10 | All 26 skills in the snapshot used the same four-row Skill Contract and exact trigger-section contract; internal hooks remained intentionally non-routable. |
|
|
34
|
+
| **Agent Compatibility** | 9/10 | **Historical 2026-09-01 result:** full hard-mechanism support was measured on Claude Code and advisory fallback on the other installer targets then covered by the audit. This is not the current platform matrix. |
|
|
35
|
+
| **Beginner Friendliness** | 7.5/10 | Quick Start was genuinely 10 seconds, but Tier Routing / Rule of 3 / session-scoped state were load-bearing concepts a newcomer had to absorb before the system's behavior made sense. Not addressed in that audit round. |
|
|
34
36
|
|
|
35
37
|
## Change log by audit cycle
|
|
36
38
|
|
|
37
39
|
### 2026-09-01
|
|
38
|
-
- Fable v3 integration
|
|
39
|
-
- The
|
|
40
|
+
- Fable v3 integration exposed explicit Haiku, Sonnet, and Opus entrypoints, with deterministic model selection and visible inline fallback or blocked escalation in `fable-mode/model-matrix.json` and `fable-mode/scripts/model-selector.js`.
|
|
41
|
+
- The workflow used native host TODO tracking or Markdown checklists. The older shared TODO CLI/state-machine references below are retained only as historical audit evidence; no current runtime, test, or active workflow depends on them.
|
|
40
42
|
- Strict catalog audit: removed obsolete compatibility directories and stale workflow; added the missing `find-skills` workflow.
|
|
41
|
-
- Corrected
|
|
42
|
-
- Added `ci/skill-routing-check.js`; all 34 positive routing cases across the 26-skill catalog
|
|
43
|
+
- Corrected documentation and runtime references to the per-session `.claude/harness-everything/state/` path, aligned README/registry layer labels, and removed unimplemented workflow claims.
|
|
44
|
+
- Added `ci/skill-routing-check.js`; all 34 positive routing cases across the 26-skill catalog passed locally, and all positive eval descriptions exactly matched their skill frontmatter.
|
|
43
45
|
|
|
44
46
|
### 2026-07-26
|
|
45
|
-
- `verify-gate.js`
|
|
46
|
-
- CI added: `.github/workflows/ci.yml`
|
|
47
|
+
- `verify-gate.js` was no longer a simulated stub: it discovered the nearest `package.json` (scoped to the enclosing git repo), ran its real `lint`/`test` scripts via the detected package manager (npm/pnpm/yarn/bun), and blocked completion on failure. The `.verify-fail.tmp` injection hook was kept for hermetic mechanism tests, and `HARNESS_SKIP_PROJECT_CHECKS=1` guarded against self-recursion in projects whose test suite itself drove the Harness runtime. When no runnable scripts existed it exited 0 but explicitly warned that no mechanical checks ran — the exit code was not evidence.
|
|
48
|
+
- CI added: `.github/workflows/ci.yml` ran `npm test` on push/PR, closing the "local-gate-only" gap called out in the previous cycle.
|
|
47
49
|
- README restructured to be user-facing (problem → quick start → what gets installed/uninstall → platform support); audit narrative moved to this file.
|
|
48
|
-
- `harness-everything/SKILL.md`: tier-router output wording and the Skill Contract
|
|
50
|
+
- `harness-everything/SKILL.md`: tier-router output wording and the Skill Contract were aligned (router output is the default route, overridable with a one-line reason — previously the contract said "MUST follow" while the script itself said "default, not an order"); section numbering fixed (§5 registry now precedes §6).
|
|
49
51
|
|
|
50
52
|
### 2026-07-23
|
|
51
53
|
- VERIFICATION.md §2's mechanism checks automated (`npm run test:mechanism`).
|
|
52
54
|
- VERIFICATION.md's test recipes rewritten to be Windows-safe (`node -e` stdin instead of a shell `echo` pipe).
|
|
53
|
-
- Workflow progress
|
|
55
|
+
- Workflow progress was recorded in the host TODO tracker or a Markdown checklist.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# Codex action-gate policy
|
|
2
|
+
|
|
3
|
+
This document records the Codex-specific action-gate contract. It intentionally differs from Claude Code because the two hosts expose different approval semantics.
|
|
4
|
+
|
|
5
|
+
## Current official capability boundary
|
|
6
|
+
|
|
7
|
+
Harness treats the current Codex hooks documentation as the release-behavior contract.
|
|
8
|
+
|
|
9
|
+
### PreToolUse
|
|
10
|
+
|
|
11
|
+
For Codex, Harness may safely block a covered local tool call with a supported deny/block result or exit code 2. Harness must not emit `permissionDecision: "ask"`: Codex currently parses that value but does not support it as an authorization decision, so relying on it would be fail-open.
|
|
12
|
+
|
|
13
|
+
The packaged action gate therefore keeps the existing fail-closed behavior for a destructive rule match:
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
covered local tool
|
|
17
|
+
-> PreToolUse
|
|
18
|
+
-> Harness destructive rule match
|
|
19
|
+
-> exit 2 / block
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
An internal failure in this pre-action path also remains fail-closed.
|
|
23
|
+
|
|
24
|
+
### PermissionRequest
|
|
25
|
+
|
|
26
|
+
Codex exposes a separate `PermissionRequest` lifecycle event when Codex is already going to request approval. A PermissionRequest hook can refine that request, but the event is not a mechanism for manufacturing an approval prompt for an otherwise permitted command.
|
|
27
|
+
|
|
28
|
+
Harness therefore registers `codex-permission-request.js` as an observer only:
|
|
29
|
+
|
|
30
|
+
- it records that Codex entered its native approval flow;
|
|
31
|
+
- it records the Harness rule match when the current rule table recognizes the tool input;
|
|
32
|
+
- it stores a SHA-256 of the exact tool input rather than the raw command/tool arguments;
|
|
33
|
+
- it returns no `allow`, `deny`, or `ask` decision;
|
|
34
|
+
- an observer/audit failure returns no authorization decision, so Codex's already-active native approval flow remains authoritative.
|
|
35
|
+
|
|
36
|
+
The current documented PermissionRequest payload does not provide the same `tool_use_id` correlation used by PreToolUse/PostToolUse. PermissionRequest evidence therefore explicitly records `exactToolCallCorrelation: false` and uses only the turn id plus the payload hash for limited correlation.
|
|
37
|
+
|
|
38
|
+
### PostToolUse
|
|
39
|
+
|
|
40
|
+
Codex uses the documented `PostToolUse` lifecycle for completion evidence, including non-zero Bash outcomes. Harness does not declare a Codex `PostToolUseFailure` hook.
|
|
41
|
+
|
|
42
|
+
`codex-action-gate-post.js` is kept in canonical `hooks/scripts/` and copied into the plugin package by `sync-openai-plugin.js`. This prevents a package sync from deleting the Codex-only adapter.
|
|
43
|
+
|
|
44
|
+
## Native execpolicy prompt rules
|
|
45
|
+
|
|
46
|
+
Codex rules/execpolicy can express native `prompt` decisions for command prefixes. That is a promising native confirmation path, but Harness does not automatically install or bundle those rules yet.
|
|
47
|
+
|
|
48
|
+
Reasons:
|
|
49
|
+
|
|
50
|
+
1. the plugin packaging contract does not establish a plugin-owned execpolicy-rules component that Harness can assume is installed and active;
|
|
51
|
+
2. execpolicy is an approval/sandbox layer and is not proven to cover every destructive operation that may already be allowed in the active sandbox;
|
|
52
|
+
3. rule activation and approval behavior must be verified in the real Codex host before a hard safety guard is weakened.
|
|
53
|
+
|
|
54
|
+
Until live evidence establishes a guaranteed native approval path for a matched action, a Harness destructive match remains a hard PreToolUse block.
|
|
55
|
+
|
|
56
|
+
## Required live verification before native-prompt delegation
|
|
57
|
+
|
|
58
|
+
A future change may delegate selected destructive rules to Codex-native prompting only after preserving evidence for at least:
|
|
59
|
+
|
|
60
|
+
- `approval_policy = "on-request"`;
|
|
61
|
+
- `approval_policy = "never"`;
|
|
62
|
+
- granular approvals with rules enabled;
|
|
63
|
+
- `approvals_reviewer = "user"`;
|
|
64
|
+
- `approvals_reviewer = "auto_review"`;
|
|
65
|
+
- plugin-hook trust/review enabled in the actual host;
|
|
66
|
+
- Windows and one Unix-like environment where practical.
|
|
67
|
+
|
|
68
|
+
The evidence must distinguish:
|
|
69
|
+
|
|
70
|
+
```text
|
|
71
|
+
Harness PreToolUse block
|
|
72
|
+
Codex execpolicy/native prompt
|
|
73
|
+
PermissionRequest defer/allow/deny
|
|
74
|
+
PostToolUse execution result
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Package and mechanism tests must not be promoted to `Live verified` in the compatibility matrix.
|
|
78
|
+
|
|
79
|
+
## Tool coverage
|
|
80
|
+
|
|
81
|
+
Current Harness PreToolUse enforcement remains wired to the tool set explicitly covered by the package (`Bash|apply_patch`). Codex hooks can cover additional local function/MCP tools, but those tools must not be claimed as action-gated until Harness has a safe classifier/rule representation for their structured inputs. Hosted or specialized tool paths may have different hook behavior and remain outside this guarantee unless separately verified.
|
|
82
|
+
|
|
83
|
+
## References
|
|
84
|
+
|
|
85
|
+
- Codex Hooks: https://developers.openai.com/codex/hooks
|
|
86
|
+
- Codex Rules: https://developers.openai.com/codex/rules
|
|
87
|
+
- Codex agent approvals and security: https://developers.openai.com/codex/agent-approvals-security
|
|
88
|
+
- Codex configuration reference: https://developers.openai.com/codex/config-reference
|
|
89
|
+
|
|
90
|
+
Related tracking: #82, #85, #98, #100, #109, #110.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Issue #20 umbrella rollup
|
|
2
|
+
|
|
3
|
+
This rollup turns issue #20's historical checklist into bounded acceptance decisions. The review snapshot is `main` at `ba7853b`, checked on 2026-09-07. The umbrella owns coordination and release evidence; implementation details remain with the linked issue.
|
|
4
|
+
|
|
5
|
+
## Closure status
|
|
6
|
+
|
|
7
|
+
Issue #20 remains open after this PR. This branch adds deterministic parser,
|
|
8
|
+
collision-gate, plugin-runner, and negative-control coverage, but it does not
|
|
9
|
+
provide live opencode session evidence, a paid behavioral rerun, the proposed
|
|
10
|
+
100-case study, or the three skill-specific improvements below. Those gaps are
|
|
11
|
+
recorded explicitly rather than treated as closed by the deterministic gates.
|
|
12
|
+
|
|
13
|
+
## Workstream decisions
|
|
14
|
+
|
|
15
|
+
| Workstream | Evidence in the review snapshot | Acceptance state | Remaining claim boundary |
|
|
16
|
+
| --- | --- | --- | --- |
|
|
17
|
+
| #37 opencode enforcement | The real plugin API is implemented and mechanism coverage exists in `ci/mechanism-2n-opencode-plugin.test.js`; the integrated reflection lifecycle is `d790e7f` on the integration branch. | Integrated, pending final integration gates | No live opencode loading or session smoke evidence was available; do not claim live enforcement. |
|
|
18
|
+
| #52 behavioral transcript parsing | Claude stream fixtures and parser coverage are integrated in `77efb06`; deterministic parser tests are available. | Integrated, pending final integration gates | No new paid behavioral rerun is part of this umbrella. Parse failures and missing tool visibility remain evidence states, not skill failures. |
|
|
19
|
+
| Systematic gate hardening | The local gates now execute real negative controls; YAML frontmatter parsing fails closed in `ci/description-collision.js`, `ci/routing-check.js` skips clearly when waza is absent, and `ci/mechanism-2t-issue-20.test.js` covers CRLF behavioral parsing and plugin validation. | Deterministic hardening integrated | Live platform loading and model behavior remain unverified. |
|
|
20
|
+
| #42 workspace identity and migration | Workspace-keyed state and migration coverage are integrated in `9dd2e3d`; the issue review identified session and migration acceptance as the relevant boundary. | Integrated, pending final integration gates | Do not claim every host/session migration path is live verified until the integration gate and targeted fixtures pass. |
|
|
21
|
+
| #43 multi-agent workspace layout | The reviewed plan makes its root contract depend on #42. No implementation commit was present in the issue #20 review snapshot. | Deferred to its owner | Do not duplicate its layout or runtime work in this umbrella. |
|
|
22
|
+
| #44 progressive disclosure | Documentation and reference checks are integrated in `b33f598`; `ci/disclosure-check.js` and its mechanism coverage are present. | Integrated, pending final integration gates | This records repository documentation evidence; it does not establish behavioral-model effectiveness. |
|
|
23
|
+
| #56 behavioral rubric triage | The review found grader/source-boundary issues, including `command_exit_0` proving the grader command rather than the agent action. | Deferred to its owner | Do not reuse the historical pass-rate claims until the rubric and trace evidence are corrected. |
|
|
24
|
+
| Extended 100-case study | The original issue proposed 100 cases, but the review did not establish a decision threshold or clean attribution between skill text, plugin enforcement, and grader behavior. | Explicitly deferred | No uncontrolled 100-case run is authorized by this rollup. Define paired design, failure taxonomy, and decision rule first. |
|
|
25
|
+
| Five historical baseline failures | The committed 2026-08-27 records show one failed expectation in each of `baseline-debugging` (edge-case consideration), `baseline-performance` (binary-search suggestion), `baseline-security-review` (MD5 identification), `baseline-simple-bugfix` (function understanding), and `verify-before-done` (verification actually run). | Explicitly deferred | These are historical observations, not a new causal investigation or rerun; do not claim they were fixed by this PR. |
|
|
26
|
+
| Skill-specific improvements | No targeted changes were made to `verification-loop`, `zoom-out`, or `security-review` in this branch. | Explicitly deferred | Strengthening verification enforcement, reflection-report detection, and vulnerability detection requires separate design and behavioral evidence. |
|
|
27
|
+
| Release | The tag-triggered full gate and provenance publish workflow already exists. Historical `0.3.6` evidence is recorded in [docs/release-evidence.md](release-evidence.md). | Documentation complete; next release unproven | A prepared `0.3.7-beta` version is not evidence of a tag, npm publication, or GitHub release. |
|
|
28
|
+
|
|
29
|
+
## Focused acceptance checks
|
|
30
|
+
|
|
31
|
+
The integration owner should run these checks after incorporating the issue branches:
|
|
32
|
+
|
|
33
|
+
```text
|
|
34
|
+
npm test
|
|
35
|
+
npm run test:mechanism
|
|
36
|
+
npm run test:consistency
|
|
37
|
+
npm run test:references
|
|
38
|
+
npm run test:release
|
|
39
|
+
npm run test:collision
|
|
40
|
+
npm run test:routing:skills
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
The checks establish deterministic repository and routing health. They do not substitute for live opencode loading, a paid behavioral run, or npm/GitHub release evidence. Those claims require the artifacts named in [RELEASING.md](../RELEASING.md) and [docs/release-evidence.md](release-evidence.md).
|
|
44
|
+
|
|
45
|
+
## Historical evidence boundary
|
|
46
|
+
|
|
47
|
+
The local review artifact `.git/issue-review-20260907/release-evidence.md` records a successful `0.3.6` workflow (`33466676632`) and npm observation. It explicitly says that this is not proof of a `0.3.7-beta` release. This rollup preserves that distinction so an old successful release cannot be mistaken for current publication evidence.
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Harness is not intended to be a rigid workflow engine. Its skills should remain independently useful, while still coordinating through lightweight mechanisms that help the model and the software engineer work together with less drift, fewer loops, and better evidence.
|
|
4
4
|
|
|
5
|
-
The preferred shape is a **mechanism-first skill mesh**: skills are autonomous, coordination is explicit, and enforcement happens at narrow decision points through scripts, hooks, exit codes, and compact return values.
|
|
5
|
+
The preferred shape is a **mechanism-first skill mesh**: skills are autonomous, coordination is explicit, and enforcement happens at narrow decision points through scripts, hooks, exit codes, and compact return values. Platform-specific enforcement claims follow [platform-capabilities.md](platform-capabilities.md).
|
|
6
6
|
|
|
7
7
|
---
|
|
8
8
|
|
|
@@ -50,8 +50,8 @@ Example shape:
|
|
|
50
50
|
```text
|
|
51
51
|
HARNESS_DECISION
|
|
52
52
|
tier: 2
|
|
53
|
-
|
|
54
|
-
|
|
53
|
+
required: route-before-execution, verify-before-claim, re-plan-after-repeat-failure
|
|
54
|
+
suggest: todo-driven-workflow, verification-loop
|
|
55
55
|
reason: user requested code changes
|
|
56
56
|
```
|
|
57
57
|
|
|
@@ -81,57 +81,59 @@ Harness should coordinate skills through small, composable patterns.
|
|
|
81
81
|
|
|
82
82
|
### Router Decision
|
|
83
83
|
|
|
84
|
-
`harness-everything/scripts/
|
|
84
|
+
`harness-everything/scripts/kernel-router.js` is the public invariant-first runtime entry point. It delegates classification and guide discovery to `tier-router.js`, then returns the recommended tier, rationale, required invariants, and advisory skill suggestions. It intentionally does not emit a mandatory global execution pipeline.
|
|
85
85
|
|
|
86
86
|
### Checklist State
|
|
87
87
|
|
|
88
|
-
`todo-driven-workflow`
|
|
88
|
+
`todo-driven-workflow` may own explicit task decomposition state when that helps a Tier 2 or Tier 3 task. It is a suggested tactic, not a universal prerequisite, and the model may use native host TODO tracking or a Markdown checklist instead.
|
|
89
89
|
|
|
90
90
|
### Verification Gate
|
|
91
91
|
|
|
92
|
-
`verification-loop` and `verify-gate.js`
|
|
92
|
+
`verification-loop` and `verify-gate.js` provide reusable verification mechanisms. Hosts with a compatible stop/completion hook can invoke verification mechanically; instruction-only hosts can call the same verifier explicitly before final delivery.
|
|
93
93
|
|
|
94
94
|
### Recovery Gate
|
|
95
95
|
|
|
96
|
-
`rule-of-3` and `zoom-out`
|
|
96
|
+
`rule-of-3` and `zoom-out` coordinate through failure signatures and recovery instructions where the host exposes the required lifecycle/tool hooks. On hosts without those hooks, `zoom-out` remains usable as an explicit recovery discipline, but the automatic failure counter must not be implied.
|
|
97
97
|
|
|
98
98
|
### Skill Suggestion
|
|
99
99
|
|
|
100
|
-
`self-evolve` and generated skills
|
|
100
|
+
`self-evolve` and generated skills feed the router through manifest metadata. The router should surface likely skills, but the model keeps final responsibility for selecting what actually applies.
|
|
101
101
|
|
|
102
102
|
---
|
|
103
103
|
|
|
104
104
|
## Platform Strategy
|
|
105
105
|
|
|
106
|
-
Harness should use the strongest mechanism each
|
|
106
|
+
Harness should use the strongest mechanism each installation surface actually provides.
|
|
107
107
|
|
|
108
108
|
| Platform class | Preferred coordination style | Enforcement level |
|
|
109
109
|
| --- | --- | --- |
|
|
110
|
-
| Hook-capable
|
|
111
|
-
| CLI/tool-capable but hook-less
|
|
112
|
-
| Prompt-only
|
|
110
|
+
| Hook/plugin-capable surfaces | Hooks/plugins run routers, guards, trackers, or completion gates automatically | Mechanical injection/blocking is possible for the mechanisms actually packaged |
|
|
111
|
+
| CLI/tool-capable but hook-less surfaces | The model explicitly calls Harness scripts at decision points | Mechanism-guided, not automatically enforced |
|
|
112
|
+
| Prompt-only surfaces | Advisory instructions include command snippets and decision rules | Self-regulated by the model |
|
|
113
113
|
|
|
114
|
-
|
|
114
|
+
One host may expose more than one surface. Codex is the important example: the general `--codex` installer path is instruction/advisory oriented, while the **local OpenAI plugin** packages session, prompt, supported-tool, subagent, and stop hooks for mechanism-tested runtime enforcement. The public OpenAI **Skills-only** submission does not include those local lifecycle hooks.
|
|
115
|
+
|
|
116
|
+
OpenCode is plugin-capable and has mechanism coverage, but live plugin loading remains unverified. Claude Code has the broadest currently verified Harness lifecycle-hook surface.
|
|
115
117
|
|
|
116
118
|
---
|
|
117
119
|
|
|
118
|
-
## Guidance for Hook-Less
|
|
120
|
+
## Guidance for Instruction-Only / Hook-Less Paths
|
|
119
121
|
|
|
120
122
|
Do not recreate hooks with a rigid universal handoff template. Instead, make mechanisms easy to call.
|
|
121
123
|
|
|
122
124
|
Implemented today, both wired into `bin/cli.js` and referenced by name in the advisory text each platform's installer writes (`scripts/lib/advisory-text.js`):
|
|
123
125
|
|
|
124
|
-
- `npx github:dyphn1/Harness-everything next "<prompt>"`:
|
|
125
|
-
- `npx github:dyphn1/Harness-everything verify`: wraps `verify-gate.js`, running the target project's own lint/test scripts and exiting non-zero on failure. This is the explicit
|
|
126
|
+
- `npx github:dyphn1/Harness-everything next "<prompt>"`: runs the Harness routing path and prints the recommended tier, invariant contract, and matching knowledge/skill suggestions.
|
|
127
|
+
- `npx github:dyphn1/Harness-everything verify`: wraps `verify-gate.js`, running the target project's own lint/test scripts and exiting non-zero on failure. This is the explicit hook-less verification path; it is not equivalent to claiming that a host automatically runs a `Stop` hook.
|
|
126
128
|
|
|
127
|
-
Both resolve their target scripts relative to the CLI's own package install, not the caller's cwd, so they work the same way regardless of which platform-specific directory (`.codex/skills/`, `.cursor/skills/`, `.github/skills/`, ...) a copy of the `harness-everything` skill also happens to be sitting in
|
|
129
|
+
Both resolve their target scripts relative to the CLI's own package install, not the caller's cwd, so they work the same way regardless of which platform-specific directory (`.codex/skills/`, `.cursor/skills/`, `.github/skills/`, ...) a copy of the `harness-everything` skill also happens to be sitting in — a fixed relative path like `harness-everything/scripts/tier-router.js` would silently point at nothing depending on which platform installed it.
|
|
128
130
|
|
|
129
|
-
Not yet implemented
|
|
131
|
+
Not yet implemented — still aspirational, do not reference these as if they exist until they're built:
|
|
130
132
|
|
|
131
|
-
- `harness recover`: summarize repeated failure signatures and recommend `zoom-out` when needed.
|
|
133
|
+
- `harness recover`: summarize repeated failure signatures and recommend `zoom-out` when needed. Hook-less paths do not collect failure signatures the way the Claude Code `PostToolUse` mechanism does, so there is no reliable state to summarize yet.
|
|
132
134
|
- `harness skills`: list installed, generated, and matching skills for the current workspace.
|
|
133
135
|
|
|
134
|
-
These commands let Codex, Cursor, Copilot, Continue, and Hermes receive high-signal mechanical feedback without forcing them into a single scripted conversation shape.
|
|
136
|
+
These commands let instruction-only Codex installs, Cursor, Copilot, Continue, and Hermes receive high-signal mechanical feedback without forcing them into a single scripted conversation shape.
|
|
135
137
|
|
|
136
138
|
---
|
|
137
139
|
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# OpenAI / ChatGPT Plugin Packaging
|
|
2
|
+
|
|
3
|
+
Harness Everything ships an Agent Plugin package under `plugins/harness-everything/`. The root skill directories remain canonical; packaged copies are synchronized from them.
|
|
4
|
+
|
|
5
|
+
## Architecture
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart TD
|
|
9
|
+
U([User prompt]) --> H[UserPromptSubmit hook]
|
|
10
|
+
H --> K[Harness Kernel Router]
|
|
11
|
+
K --> I[Required invariants]
|
|
12
|
+
I --> A[Agent chooses useful domain skills]
|
|
13
|
+
A --> E[Execute / inspect / edit]
|
|
14
|
+
E --> P[PreToolUse guards]
|
|
15
|
+
P --> T[Supported local tool]
|
|
16
|
+
T --> Q[PostToolUse state and evidence]
|
|
17
|
+
Q --> V{Objective evidence?}
|
|
18
|
+
V -->|yes| D[Evidence-backed completion]
|
|
19
|
+
V -->|no| R[Diagnose and retry]
|
|
20
|
+
R --> F{Same-signature failure x3?}
|
|
21
|
+
F -->|no| E
|
|
22
|
+
F -->|yes| Z[Zoom out / re-plan]
|
|
23
|
+
Z --> E
|
|
24
|
+
|
|
25
|
+
S[SessionStart hook] --> C[Cognitive OS policy]
|
|
26
|
+
C --> K
|
|
27
|
+
G[SubagentStart / Stop] --> A
|
|
28
|
+
X[Stop hook] --> V
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The local OpenAI adapter enforces cross-cutting invariants through the lifecycle events that it packages. Tier-specific skills remain advisory; the model may choose, combine, reorder, or skip them. Tool enforcement is limited to the host tool names declared in `hooks/hooks.json`.
|
|
32
|
+
|
|
33
|
+
## Package layout
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
plugins/harness-everything/
|
|
37
|
+
├── plugin.json # portable Agent Plugin manifest
|
|
38
|
+
├── .codex-plugin/plugin.json # Codex compatibility manifest
|
|
39
|
+
├── hooks/
|
|
40
|
+
│ ├── hooks.json
|
|
41
|
+
│ ├── session-start.js
|
|
42
|
+
│ └── scripts/ # shared runtime hook dependencies
|
|
43
|
+
└── skills/ # 26 synchronized canonical skills
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
The portable manifest follows the Agent Plugins schema and uses a root `skills/` directory. OpenAI-specific hook and interface metadata lives under `extensions.com.openai`; the compatibility manifest keeps the legacy `.codex-plugin` shape for clients that still expect it. These are two distribution manifests for the same package, not two different products.
|
|
47
|
+
|
|
48
|
+
The package is intentionally skill-only: it does not declare `mcpServers` or `apps`. Hooks are a local package capability and require the host’s trust/review flow; they are not a claim that every OpenAI surface provides the same lifecycle enforcement.
|
|
49
|
+
|
|
50
|
+
Run:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npm run plugin:sync
|
|
54
|
+
npm run test:plugin:openai
|
|
55
|
+
npm run test:plugin:submission
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
`plugin:sync` refreshes package copies from canonical skill directories and copies the shared hook runtime/dependencies into the package. Text parity is normalized to LF so the same package is reproducible from Windows and Unix checkouts. The checks compare both skill trees byte-for-byte, validate both manifest forms, execute local hook entry points, and verify deterministic routing/runtime behavior.
|
|
59
|
+
|
|
60
|
+
## Managed ChatGPT workspace import
|
|
61
|
+
|
|
62
|
+
OpenAI documents GitHub marketplace import for workspace administrators. The repository provides both `.agents/plugins/marketplace.json` and `.claude-plugin/marketplace.json`; the source entry resolves to `plugins/harness-everything/`.
|
|
63
|
+
|
|
64
|
+
Use the repository URL as the source, leave Path empty so the marketplace is read from the repository root, and select the intended branch (normally `main`). In ChatGPT, use **Workspace settings > Plugins > Add > Import marketplace**, review the import results, and set the workspace installation policy. OpenAI documents automatic daily synchronization and a **Sync now** action for later refreshes.
|
|
65
|
+
|
|
66
|
+
This repository contains the marketplace/package artifacts and deterministic import-shape checks. It does not contain a managed-workspace import report or a fresh host smoke-test artifact, so those operations remain `Unknown` in the compatibility matrix.
|
|
67
|
+
|
|
68
|
+
Official documentation: [Importing and syncing plugin marketplaces from GitHub](https://help.openai.com/en/articles/20001504-importing-and-syncing-plugin-marketplaces-from-github) and [Plugins in Codex](https://help.openai.com/en/articles/20001256-plugins-in-codex).
|
|
69
|
+
|
|
70
|
+
## Local Codex / OpenAI package
|
|
71
|
+
|
|
72
|
+
The local package path is distinct from the general `--codex` installer. The general installer writes advisory `AGENTS.md` and repo-scoped `.agents/skills/`; the plugin package additionally contains local session, prompt, supported-tool, subagent, and stop hooks.
|
|
73
|
+
|
|
74
|
+
The package tests prove the following repository-side contract:
|
|
75
|
+
|
|
76
|
+
- the portable root manifest and `.codex-plugin` compatibility manifest agree on identity, version, interface, and hook entry point;
|
|
77
|
+
- the root `skills/` tree contains exactly the 26 canonical skills;
|
|
78
|
+
- Windows and POSIX hook commands resolve through `PLUGIN_ROOT`;
|
|
79
|
+
- session start emits the compact policy and prompt submission invokes invariant-first routing;
|
|
80
|
+
- supported local `Bash` and `apply_patch` calls receive the packaged guards and state tracking;
|
|
81
|
+
- subagent lifecycle and stop verification contracts are covered;
|
|
82
|
+
- identical routing input produces identical output.
|
|
83
|
+
|
|
84
|
+
These are package/mechanism results. They do not prove that a live Codex or ChatGPT host loaded the package; a preserved host/session artifact is required for that claim. OpenAI’s current event/tool contract is documented in the [Codex/ChatGPT hooks reference](https://learn.chatgpt.com/docs/hooks).
|
|
85
|
+
|
|
86
|
+
## Public Skills-only submission
|
|
87
|
+
|
|
88
|
+
OpenAI’s public submission flow supports a **Skills only** plugin. This is separate from managed-workspace marketplace import and from the local hook-capable package.
|
|
89
|
+
|
|
90
|
+
Build the final upload from the synchronized package:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
npm run plugin:submission:build
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
The generated ZIP contains only `skills/<name>/...` files from `plugins/harness-everything/skills/`. It does **not** include the local `.codex-plugin` lifecycle hooks, MCP definitions, or a fake app wrapper. The generated manifest records every path, byte count, file hash, and final bundle hash; repeated builds from the same revision must match.
|
|
97
|
+
|
|
98
|
+
The submission source files in [`../submission/openai/`](../submission/openai/) model the official review inputs: listing metadata, starter prompts, release notes, exactly five positive cases, and exactly three negative cases. Positive cases specify the prompt, expected skill/workflow behavior, expected result shape, and fixture/account data. Negative cases specify the prompt/scenario, expected refusal/clarification/safe fallback, and why completion is not appropriate.
|
|
99
|
+
|
|
100
|
+
The official portal flow is: **Create plugin → Skills only → upload the final skill bundle → provide review cases and listing metadata → complete identity, availability, policy, and access requirements → submit for review → publish after approval**. The repository cannot complete verified identity, logo upload, availability selection, Apps Management write access, review, approval, or public-directory publication.
|
|
101
|
+
|
|
102
|
+
Official documentation: [Submit a plugin](https://developers.openai.com/plugins/deploy/submission) and [Build plugins](https://developers.openai.com/plugins/build/plugins).
|
|
103
|
+
|
|
104
|
+
## Evidence boundary
|
|
105
|
+
|
|
106
|
+
The local plugin and public Skills-only artifact must not be conflated:
|
|
107
|
+
|
|
108
|
+
| Artifact | Repository evidence | Claim that is safe today |
|
|
109
|
+
| --- | --- | --- |
|
|
110
|
+
| Portable/root plugin package | Manifest, tree, hook, and routing mechanism tests | Package shape and local mechanism are verified |
|
|
111
|
+
| Managed workspace marketplace | Marketplace JSON and source-path checks | Import-ready repository artifact; live import remains unverified |
|
|
112
|
+
| Public Skills-only ZIP | Deterministic builder and 5+3 review-input checks | Reproducible skills bundle; approval and live public behavior remain unverified |
|
|
113
|
+
|
|
114
|
+
See the [platform capability matrix](platform-capabilities.md) for the complete ten-dimension status table and official source URLs.
|
|
115
|
+
|
|
116
|
+
## Publication smoke checklist
|
|
117
|
+
|
|
118
|
+
Before public submission, run `plugin:sync`, both OpenAI package checks, and the platform compatibility check. Confirm all 26 skills are content-identical to canonical sources after EOL normalization, the ZIP is reproducible, public links are current, the publisher identity and Apps Management access are ready, and final reviewer-facing cases run against the exact submitted skills tree.
|
package/docs/philosophy.md
CHANGED
|
@@ -1,47 +1,58 @@
|
|
|
1
1
|
# Harness Philosophy
|
|
2
2
|
|
|
3
|
-
Harness is built on a single, core belief: **AI coding models are highly capable, but they struggle with self-regulation, attention drift, and environment awareness.**
|
|
3
|
+
Harness is built on a single, core belief: **AI coding models are highly capable, but they struggle with self-regulation, attention drift, and environment awareness.**
|
|
4
4
|
|
|
5
|
-
Instead of replacing the model or restricting it with heavy, opinionated frameworks that stifle creativity, Harness acts as a **behavioral
|
|
5
|
+
Instead of replacing the model or restricting it with heavy, opinionated frameworks that stifle creativity, Harness acts as a **behavioral layer** — reusable skills plus the strongest lightweight runtime mechanisms each supported installation surface actually provides.
|
|
6
|
+
|
|
7
|
+
The current mechanism/enforcement boundary is platform-specific; see [Platform Capability Matrix](platform-capabilities.md). A skill-only or instruction-only install must not be described as if it automatically runs lifecycle hooks.
|
|
6
8
|
|
|
7
9
|
---
|
|
8
10
|
|
|
9
11
|
## The Core Philosophy: Intervene Only When Necessary
|
|
10
12
|
|
|
11
|
-
Many AI development frameworks attempt to fully control the model's execution flow. They use rigid DAGs, forced chains, or complex state machines. While this works for trivial, repetitive tasks, it
|
|
13
|
+
Many AI development frameworks attempt to fully control the model's execution flow. They use rigid DAGs, forced chains, or complex state machines. While this works for trivial, repetitive tasks, it can suppress useful model judgment on complex, novel engineering work.
|
|
12
14
|
|
|
13
15
|
Harness takes the opposite approach:
|
|
14
16
|
|
|
15
|
-
1. **Model Autonomy is Paramount:** The model should have
|
|
16
|
-
2. **Context-Driven Guidance:** Harness
|
|
17
|
-
3. **Reactive Guardrails:**
|
|
18
|
-
4. **Behavior-First, Not Prompt-First:** Harness is not a collection of
|
|
17
|
+
1. **Model Autonomy is Paramount:** The model should have freedom to explore, choose useful skills/tools, and design its solution inside a small set of cross-cutting invariants.
|
|
18
|
+
2. **Context-Driven Guidance:** Harness uses high-context, low-friction signals — skills, rules, local checks, and hooks/plugins where the host supports and packages them — rather than one rigid global execution path.
|
|
19
|
+
3. **Reactive Guardrails:** On integration surfaces that expose the required lifecycle/tool hooks, Harness can stay quiet until a boundary is violated or a repeated failure is detected. On advisory surfaces, the same principles remain guidance rather than automatic blocking.
|
|
20
|
+
4. **Behavior-First, Not Prompt-First:** Harness is not a collection of magic prompts. Its skills remain independently useful, while supported runtime adapters can react to real tool/session outcomes and workspace conditions.
|
|
19
21
|
|
|
20
22
|
---
|
|
21
23
|
|
|
22
24
|
## The 4 Pillars of AI Model Guidance
|
|
23
25
|
|
|
24
|
-
To keep agents aligned and productive, Harness operates across four distinct domains
|
|
26
|
+
To keep agents aligned and productive, Harness operates across four distinct domains. The **principle** applies everywhere; whether it is injected or enforced automatically depends on the selected host surface.
|
|
25
27
|
|
|
26
28
|
### 1. Environment Alignment (Discovery over Assumption)
|
|
27
|
-
|
|
28
|
-
|
|
29
|
+
|
|
30
|
+
AI models often hallucinate terminal environments or make incorrect assumptions about the host operating system, shell, or package manager — especially on Windows or mixed-shell setups.
|
|
31
|
+
|
|
32
|
+
Harness requires environment assumptions to be discovered rather than guessed. On surfaces that package a session-start/preflight hook, that context can be injected automatically. On instruction/skill-only paths, the agent must discover it explicitly before relying on shell-specific commands.
|
|
29
33
|
|
|
30
34
|
### 2. Guardrails (The Circuit Breakers)
|
|
31
|
-
|
|
32
|
-
|
|
35
|
+
|
|
36
|
+
When a model gets stuck on a subtle bug or compiler error, its natural tendency is to make micro-adjustments repeatedly.
|
|
37
|
+
|
|
38
|
+
The Harness recovery invariant is to stop same-signature micro-retries and re-plan after repeated failure. On hosts where the Rule-of-3 mechanism is actually packaged, a circuit breaker can track failures and block further mutation until a `zoom-out` diagnosis is performed. On other surfaces, `zoom-out` remains a reusable recovery discipline but there is no claim that an automatic counter/blocker exists.
|
|
33
39
|
|
|
34
40
|
### 3. Context Preservation (Anti-Bloat Protection)
|
|
35
|
-
|
|
36
|
-
|
|
41
|
+
|
|
42
|
+
As sessions grow, models can read too broadly or produce oversized output, degrading active context.
|
|
43
|
+
|
|
44
|
+
Harness uses narrow reads, explicit scope, compact evidence, and — where a compatible runtime adapter exists — boundary mechanisms that can react mechanically. Skill-only surfaces still receive the discipline, but not an imaginary background daemon or hook.
|
|
37
45
|
|
|
38
46
|
### 4. Self-Evolution (Continuous Workspace Memory)
|
|
39
|
-
|
|
47
|
+
|
|
48
|
+
When a complex issue is resolved, Harness can turn the verified lesson into durable workspace knowledge through `self-evolve`: a concise rule or a reusable generated skill, when persistence is actually justified.
|
|
49
|
+
|
|
50
|
+
This long-term learning path is distinct from **runtime session state** such as WAL/checkpoint files. Runtime state exists only on integrations that package the corresponding state hooks; reusable rules/skills are project artifacts and can outlive a single host session.
|
|
40
51
|
|
|
41
52
|
---
|
|
42
53
|
|
|
43
54
|
## A Non-Intrusive Cognitive Amplifier
|
|
44
55
|
|
|
45
|
-
Harness does not
|
|
56
|
+
Harness does not require a heavy daemon or a proprietary service. The repository's mechanisms and skill content operate locally in the developer/host environment, while each platform gets only the enforcement strength its real integration supports.
|
|
46
57
|
|
|
47
|
-
For
|
|
58
|
+
For skill-level coordination, see [Mechanism-First Skill Mesh](mechanism-first-skill-mesh.md). For the exact current platform/install/evidence boundary, see [Platform Capability Matrix](platform-capabilities.md).
|