harness-everything 0.5.2 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. package/.claude-plugin/marketplace.json +1 -1
  2. package/.claude-plugin/plugin.json +1 -1
  3. package/AGENTS.md +5 -1
  4. package/BENCHMARK_SOP.md +1 -1
  5. package/CHANGELOG.md +12 -0
  6. package/README.md +3 -3
  7. package/VERIFICATION.md +5 -5
  8. package/behavioral-evals/PAIRED-BENCHMARK.md +186 -0
  9. package/behavioral-evals/README.md +46 -0
  10. package/behavioral-evals/opencode-hardlock-live.js +301 -0
  11. package/behavioral-evals/paired-benchmark.js +781 -0
  12. package/behavioral-evals/run-with-plugin.js +1 -1
  13. package/benchmarks/results/live-host/opencode-2026-09-16/README.md +133 -0
  14. package/benchmarks/results/live-host/opencode-2026-09-16/probe-control.md +88 -0
  15. package/benchmarks/results/live-host/opencode-2026-09-16/run-commands.md +100 -0
  16. package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/circuit-breaker.json +16 -0
  17. package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/compliance.json +7 -0
  18. package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/edit-state.json +10 -0
  19. package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/zoom-out-report.md +29 -0
  20. package/ci/doc-capability-consistency.test.js +56 -5
  21. package/ci/mechanism-2w-platform-compatibility.test.js +20 -7
  22. package/ci/mechanism-30-opencode-plugin-loadability.test.js +86 -0
  23. package/ci/mechanism-31-opencode-hardlock-evidence.test.js +138 -0
  24. package/ci/mechanism-32-paired-benchmark.test.js +254 -0
  25. package/docs/architecture.md +2 -2
  26. package/docs/platform-capabilities.md +3 -3
  27. package/docs/platform-compatibility.json +4 -4
  28. package/opencode-plugin/README.md +32 -11
  29. package/opencode-plugin/plugin.json +1 -1
  30. package/package.json +5 -1
  31. package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
  32. package/plugins/harness-everything/plugin.json +1 -1
@@ -10,7 +10,7 @@
10
10
  "name": "harness-everything",
11
11
  "source": "./",
12
12
  "description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
13
- "version": "0.5.2",
13
+ "version": "0.6.0",
14
14
  "author": {
15
15
  "name": "Miya Daniel"
16
16
  },
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "harness-everything",
3
3
  "description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
4
- "version": "0.5.2",
4
+ "version": "0.6.0",
5
5
  "author": {
6
6
  "name": "Miya Daniel"
7
7
  },
package/AGENTS.md CHANGED
@@ -13,7 +13,7 @@ You are modifying **Harness**: an orchestrated agent operating system (skills +
13
13
  | `<skill-name>/SKILL.md` | One canonical skill per top-level directory. Frontmatter: `name`, `description`, `metadata.author`, `metadata.version`. |
14
14
  | `hooks/` | Claude Code lifecycle hook scripts and shared mechanisms such as circuit breaker, scope guard, and stop gate. Do not infer that every host packages every hook. |
15
15
  | `harness-everything/scripts/` | Runtime: kernel/tier routing, verification, bootstrap, self-heal. |
16
- | `opencode-plugin/` | OpenCode native plugin implementation. Mechanism-tested; live plugin loading remains unverified until real host evidence exists. |
16
+ | `opencode-plugin/` | OpenCode native plugin implementation. Mechanism-tested, with partial live-host evidence for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS); see the evidence boundary below. |
17
17
  | `.agents/plugins/` + `plugins/harness-everything/` | Local OpenAI/Codex marketplace + `.codex-plugin` package with canonical skill copies and packaged session/prompt hooks. |
18
18
  | `submission/openai/` | Public OpenAI Skills-only listing/test inputs. The public artifact does not include local `.codex-plugin` lifecycle hooks. |
19
19
  | `evals/<skill>/` | Trigger/routing eval per skill (waza format). Required for every canonical routed skill. |
@@ -25,6 +25,10 @@ You are modifying **Harness**: an orchestrated agent operating system (skills +
25
25
  | `.nvmrc` | Primary development/CI Node runtime. `package.json#engines.node` remains the minimum supported runtime. |
26
26
  | `docs/`, `references/` | Philosophy, architecture, workflow documentation, shared checklists. |
27
27
 
28
+ ## OpenCode evidence boundary
29
+
30
+ The retained [OpenCode evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md) supports only the scoped loading and state effects above. The final snapshot is post-reset (`hardLock: false`, `count: 1`). Hard lock is only an interactive observation with no retained blocked-tool trace; reflection was operator-seeded, then agent-rewritten. Agent-controlled state deletion resets the breaker, so do not claim durable hard enforcement or behavioral effectiveness. `.mjs` auto-discovery is broken on this host version. Global scope, npm-package installation, and other host versions remain unverified.
31
+
28
32
  ## Non-Negotiable Change Rules
29
33
 
30
34
  1. **Every canonical SKILL.md keeps ≤ 500 tokens** (hard CI gate) and must contain `## USE FOR:` and `## DO NOT USE FOR:` sections. Deep detail goes in the skill's `references/` or `guides/`.
package/BENCHMARK_SOP.md CHANGED
@@ -4,7 +4,7 @@ This guide helps you objectively test the performance difference between an AI A
4
4
 
5
5
  **What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether an underlying hook/plugin mechanism actually loaded, fired, or blocked anything. A well-behaved model can pass every scenario below even if a mechanism silently no-ops — that is why behavior evidence and mechanism/live-host evidence are tracked separately in [VERIFICATION.md](VERIFICATION.md).
6
6
 
7
- Do not classify mechanism support by product name alone. A host can expose more than one Harness surface. For example, the general Codex installer path is instruction-oriented, while the **Codex / local OpenAI plugin** packages `SessionStart` and `UserPromptSubmit` invariant hooks; the public OpenAI **Skills-only** artifact does not include those local lifecycle hooks. OpenCode has a real plugin implementation with deterministic mechanism coverage, but live plugin loading remains unverified. Use [docs/platform-capabilities.md](docs/platform-capabilities.md) as the current source of truth.
7
+ Do not classify mechanism support by product name alone. A host can expose more than one Harness surface. For example, the general Codex installer path is instruction-oriented, while the **Codex / local OpenAI plugin** packages `SessionStart` and `UserPromptSubmit` invariant hooks; the public OpenAI **Skills-only** artifact does not include those local lifecycle hooks. OpenCode has deterministic mechanism coverage and partial live-host evidence for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS). The [retained evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md) is a post-reset snapshot, not a full enforcement history. Hard lock is only an interactive observation with no retained blocked-tool trace; reflection was operator-seeded, then agent-rewritten. Agent-controlled state deletion resets the breaker; neither durable hard enforcement nor behavioral effectiveness is proven. Global scope, npm-package installation, and other host versions remain unverified. Use [docs/platform-capabilities.md](docs/platform-capabilities.md) as the current source of truth.
8
8
 
9
9
  This SOP is therefore a **behavior** benchmark. Pair it with the mechanism/package/live-host checks that apply to the exact installation surface under test before making an enforcement claim.
10
10
 
package/CHANGELOG.md CHANGED
@@ -2,6 +2,18 @@
2
2
 
3
3
  All notable changes to this project are documented in this file.
4
4
 
5
+ ## [0.6.0](https://github.com/dyphn1/Harness-everything/compare/v0.5.3...v0.6.0) (2026-09-17)
6
+
7
+ ### Features
8
+
9
+ * **behavioral:** add paired benchmark contract ([42d52b3](https://github.com/dyphn1/Harness-everything/commit/42d52b3a6db89fa27cd2239726a22fe119299932))
10
+
11
+ ## [0.5.3](https://github.com/dyphn1/Harness-everything/compare/v0.5.2...v0.5.3) (2026-09-17)
12
+
13
+ ### Bug Fixes
14
+
15
+ * **opencode:** Install discoverable plugin and guard loadability ([5645c08](https://github.com/dyphn1/Harness-everything/commit/5645c080c794b82afd50a0206cf35d2dd3fcd54c))
16
+
5
17
  ## [0.5.2](https://github.com/dyphn1/Harness-everything/compare/v0.5.1...v0.5.2) (2026-09-16)
6
18
 
7
19
  ### Bug Fixes
package/README.md CHANGED
@@ -31,7 +31,7 @@ Harness deliberately follows a **minimal rails, maximum freedom** design: the ru
31
31
  | **System Audit** | Blindly assumes shell syntax | Requires manual shell check | Uses environment detection and explicit verification paths instead of assuming one shell/runtime |
32
32
  | **Memory** | Resets on every new chat session | Static text rules | Stateful hook/plugin integrations can persist bounded runtime state; instruction-only integrations cannot |
33
33
 
34
- The exact "Harness" behavior depends on the installation surface. Claude Code has the broadest currently verified lifecycle-hook coverage. OpenCode has a real plugin implementation with deterministic mechanism tests, but **live loading remains unverified**. Codex has both an advisory `--codex` installer path and a **local OpenAI plugin** path with mechanism-tested session, prompt, supported-tool, subagent, and stop hooks. The public OpenAI **Skills-only** submission is narrower and does not include those local lifecycle hooks. See [Supported AI IDEs & Tools](#supported-ai-ides--tools) and [docs/platform-capabilities.md](docs/platform-capabilities.md).
34
+ The exact "Harness" behavior depends on the installation surface. Claude Code has the broadest currently verified lifecycle-hook coverage. OpenCode has deterministic mechanism tests and **partial live-host evidence** for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS), not durable hard enforcement; global scope and npm install remain unverified. Codex has both an advisory `--codex` installer path and a **local OpenAI plugin** path with mechanism-tested session, prompt, supported-tool, subagent, and stop hooks. The public OpenAI **Skills-only** submission is narrower and does not include those local lifecycle hooks. See [Supported AI IDEs & Tools](#supported-ai-ides--tools) and [docs/platform-capabilities.md](docs/platform-capabilities.md).
35
35
 
36
36
  ### When should I use Harness?
37
37
  * You regularly use agentic coding tools (like Claude Code, Cursor, or Copilot) on medium-to-large codebases.
@@ -158,12 +158,12 @@ Harness operates through six core cognitive concepts:
158
158
 
159
159
  The authoritative current matrix is [docs/platform-capabilities.md](docs/platform-capabilities.md). The important distinction is that **one host can have multiple Harness installation surfaces**.
160
160
 
161
- [`opencode-plugin/`](opencode-plugin/) (`index.mjs`) implements a verification gate and a Rule of 3 breaker against OpenCode's real, source-verified plugin API (`tool.execute.before`/`.after`, the `session.idle` event). `ci/mechanism-2n-opencode-plugin.test.js` drives the exported hooks directly against a mock context matching that API. What's *not* yet done: a live OpenCode session actually loading and firing the plugin, so the current claim remains **hard-capable / mechanism-tested, unverified live**.
161
+ [`opencode-plugin/`](opencode-plugin/) (`index.mjs`) implements a verification gate and a Rule of 3 breaker against OpenCode's real, source-verified plugin API (`tool.execute.before`/`.after`, the `session.idle` event). `ci/mechanism-2n-opencode-plugin.test.js` drives the exported hooks directly against a mock context matching that API. Retained evidence supports project-scope `.js` plugin loading and edit/verification state on OpenCode 1.18.31 (macOS): [live-host evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md). The final snapshot is post-reset (`hardLock: false`, `count: 1`); hard lock is only an interactive observation with no retained blocked-tool trace. Reflection was operator-seeded, then agent-rewritten, not unaided. Agent-controlled state deletion resets the breaker, so this does not prove durable hard enforcement or behavioral effectiveness. Install with a **`.js`** destination name: this host silently ignores `.mjs` (issue #127, guarded by `test:opencode:loadability`). Global scope, npm-package installation, and other host versions remain unverified.
162
162
 
163
163
  | AI Agent Tool / Surface | Integration Method | Local Target Location | Enforcement claim |
164
164
  |---|---|---|---|
165
165
  | **Claude Code** | Native lifecycle hooks (`PreToolUse`, `PostToolUse`, `SessionStart`, `UserPromptSubmit`, `Stop`) | `.claude/settings.json`, `.claude/skills/`, `.claude/agents/` | **Hard** for the supported verified hook gates |
166
- | **OpenCode** | Native plugin module ([`opencode-plugin/index.mjs`](opencode-plugin/index.mjs)) | `.opencode/plugins/` | **Hard-capable, unverified live** — mechanism/source coverage exists; live host loading is not yet evidence-backed |
166
+ | **OpenCode** | Native plugin module ([`opencode-plugin/index.mjs`](opencode-plugin/index.mjs), install as `harness-enforcement.js`) | `.opencode/plugins/` | **Partial live-host evidence** — project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS); hard lock reported only, no retained blocked-tool trace; not durable hard enforcement |
167
167
  | **Codex — general installer path** | Skills + `AGENTS.md` guidance | `AGENTS.md` + repo-scoped `.agents/skills/` | **Advisory/instruction-oriented** where the host only consumes instructions |
168
168
  | **Codex / local OpenAI plugin** | `.codex-plugin` package with session, prompt, supported-tool, subagent, and stop hooks plus 26 canonical skills | `.agents/plugins/marketplace.json` → `plugins/harness-everything/` | **Mechanism-tested local enforcement** for the packaged mappings; live host loading remains unverified |
169
169
  | **Public OpenAI Skills-only plugin** | Public Skills-only bundle | Generated submission ZIP from `plugins/harness-everything/skills/` | **Skill/workflow behavior only**; no local `.codex-plugin` lifecycle hooks in the public artifact |
package/VERIFICATION.md CHANGED
@@ -22,7 +22,7 @@ First identify **which Harness surface you are testing**. One host can have more
22
22
  | Surface | Artifact / location | What artifact presence proves |
23
23
  |---|---|---|
24
24
  | Claude Code | `.claude/settings.json`, `.claude/skills/`, `.claude/agents/` | The Claude installer/plugin wrote its lifecycle-hook configuration and content |
25
- | OpenCode | `.opencode/plugins/` / `opencode-plugin/index.mjs` | The native plugin module exists; this alone does not prove live loading |
25
+ | OpenCode | `.opencode/plugins/harness-enforcement.js` (copied from `opencode-plugin/index.mjs`) | The loadable plugin module exists under a `.js` name; this alone does not prove live loading |
26
26
  | Codex — general installer | `AGENTS.md`, repo-scoped `.agents/skills/` | Advisory/instruction integration and project Agent Skills are installed |
27
27
  | **Codex / local OpenAI plugin** | `.agents/plugins/marketplace.json`, `plugins/harness-everything/.codex-plugin/plugin.json`, packaged hooks/skills | The local OpenAI plugin package is structurally present |
28
28
  | **Public OpenAI Skills-only** | generated `dist/openai-submission/harness-everything-skills.zip` | The public-review bundle was generated; local `.codex-plugin` lifecycle hooks are intentionally not part of this artifact |
@@ -127,9 +127,9 @@ The repository implements the real OpenCode plugin API and has deterministic mec
127
127
  npm run test:mechanism
128
128
  ```
129
129
 
130
- and inspect the OpenCode-specific mechanism suite (`ci/mechanism-2n-opencode-plugin.test.js`).
130
+ and inspect the OpenCode-specific mechanism suites (`ci/mechanism-2n-opencode-plugin.test.js` for hook behavior, `ci/mechanism-30-opencode-plugin-loadability.test.js` for install-filename loadability).
131
131
 
132
- **Current evidence boundary:** implementation/mechanism-tested, **live plugin loading remains unverified** until a real OpenCode session artifact demonstrates that the host loaded and fired the plugin.
132
+ **Current evidence boundary:** deterministic mechanism coverage plus partial live-host evidence for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS). See [retained evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md). The final snapshot is post-reset (`hardLock: false`, `count: 1`), not a history of the enforcement sequence. Hard lock is only an interactive observation with no retained blocked-tool trace. Reflection was operator-seeded, then agent-rewritten; follow-up delivery is reported/state-consistent, without a retained transcript. Agent-controlled state deletion resets the breaker, so neither durable hard enforcement nor behavioral effectiveness is proven. `.mjs` auto-discovery is broken on this host version (issue #127). Global scope, npm-package installation, and other host versions remain unverified.
133
133
 
134
134
  ---
135
135
 
@@ -142,7 +142,7 @@ npm run test:docs:capabilities
142
142
  npm run test:consistency
143
143
  ```
144
144
 
145
- The capability test checks the current-state documentation surfaces for stale platform paths and pre-plugin claims, requires the Codex local-plugin vs public Skills-only distinction, preserves the OpenCode live-unverified qualifier, and records the Hermes native user skill target plus the Continue installer candidate boundary.
145
+ The capability test checks the current-state documentation surfaces for stale platform paths and pre-plugin claims, requires the Codex local-plugin vs public Skills-only distinction, requires the OpenCode retained artifact path, host/version/OS and project `.js` scope, and explicit partial-evidence limitations, and records the Hermes native user skill target plus the Continue installer candidate boundary.
146
146
 
147
147
  When changing platform integration behavior, update [docs/platform-capabilities.md](docs/platform-capabilities.md) and all affected current-state docs in the same PR.
148
148
 
@@ -237,7 +237,7 @@ Fill in one row **per installation surface**, not merely per brand name.
237
237
  |---|---|---|---|---|---|---|
238
238
  | Package/artifact present | | | | | | |
239
239
  | Deterministic mechanism tests | | | N/A/explicit CLI only | | N/A (skill bundle) | N/A |
240
- | Live host loaded mechanism | | **unverified until evidenced** | N/A | | N/A | N/A |
240
+ | Live host loaded mechanism | | **loading verified; lifecycle evidence partial** (project scope, OpenCode 1.18.31, macOS, `.js`) — [evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md) | N/A | | N/A | N/A |
241
241
  | Routing/invariant behavior | | | | | | |
242
242
  | Objective verification discipline | | | | | | |
243
243
  | Failure recovery discipline | | | | | | |
@@ -0,0 +1,186 @@
1
+ # Paired Behavioral Benchmark Protocol
2
+
3
+ This protocol is the measurement surface for issue #71. It is intentionally
4
+ separate from mechanism tests: mechanism tests prove deterministic hook and
5
+ state-machine behavior, while this benchmark measures agent behavior under a
6
+ real model session.
7
+
8
+ ## Separate causal questions
9
+
10
+ Never combine these interventions into one headline number.
11
+
12
+ ### `skill-text`
13
+
14
+ Measures the incremental effect of Harness skill guidance only.
15
+
16
+ - baseline: no Harness skill text, no Harness hooks/plugin enforcement;
17
+ - treatment: the case's named Harness skill text only;
18
+ - same engine, explicit model, prompt, fixture, rubric, CLI policy, and tool
19
+ availability in both arms.
20
+
21
+ The treatment does **not** install `hooks/hooks.json` or the OpenCode
22
+ `harness-enforcement.js` plugin. This removes the old confound where a
23
+ "skill" treatment also changed enforcement behavior.
24
+
25
+ ### `plugin-enforcement`
26
+
27
+ Measures the incremental effect of OpenCode enforcement while holding skill
28
+ text constant.
29
+
30
+ - baseline: named Harness skill text, no OpenCode enforcement plugin;
31
+ - treatment: the same named Harness skill text plus the canonical OpenCode
32
+ enforcement module installed as `.opencode/plugins/harness-enforcement.js`.
33
+
34
+ This experiment currently supports OpenCode only. It requires a retained
35
+ hard-lock preflight produced by `opencode-hardlock-live.js`. The preflight must
36
+ still pass verification and its plugin SHA-256 must equal the current canonical
37
+ `opencode-plugin/index.mjs`; changing the plugin invalidates the old preflight.
38
+
39
+ Each treatment case must also leave retained plugin state proving that the
40
+ plugin fired during that case. Missing attribution is classified as
41
+ infrastructure/inconclusive and is excluded from behavioral pass rates.
42
+
43
+ ### Cost and execution overhead
44
+
45
+ Cost is reported separately from correctness:
46
+
47
+ - provider cost, where exposed by the host;
48
+ - total tokens, where exposed;
49
+ - attempted tool calls;
50
+ - elapsed duration.
51
+
52
+ The report gives treatment-minus-baseline deltas and does not fold them into a
53
+ correctness score.
54
+
55
+ ## Predeclare the decision threshold
56
+
57
+ There is deliberately **no default minimum effect**. Before model calls start,
58
+ the caller must supply `--min-effect-pp <N>` to state the smallest paired pass
59
+ rate improvement that would be practically meaningful for that study. The
60
+ experiment manifest is written before either arm runs, so the threshold cannot
61
+ be selected after seeing the result.
62
+
63
+ Example only (not a project-wide default):
64
+
65
+ ```bash
66
+ --min-effect-pp 10
67
+ ```
68
+
69
+ A study reports `meets-predeclared-threshold` only when the lower bound of the
70
+ paired bootstrap 95% confidence interval is at or above that predeclared
71
+ minimum. It reports `below-predeclared-threshold` when the interval lies wholly
72
+ below it, otherwise `uncertain-at-predeclared-threshold`. Fewer than two
73
+ included pairs is `insufficient-data`.
74
+
75
+ The report also includes an exact McNemar p-value for the discordant pairs. The
76
+ p-value is evidence about symmetry of fail->pass vs pass->fail transitions; it
77
+ is not a substitute for the predeclared practical-effect threshold.
78
+
79
+ ## Evidence and exclusions
80
+
81
+ A pair enters the behavioral denominator only when both arms are definitive
82
+ `pass` or `fail` and the pair contract validates.
83
+
84
+ Excluded reasons include:
85
+
86
+ - CLI/session error;
87
+ - grader error;
88
+ - incomplete or inconclusive transcript evidence;
89
+ - mismatched actual model when both hosts report it;
90
+ - pair-contract hash mismatch;
91
+ - intervention shape mismatch;
92
+ - missing OpenCode plugin attribution/preflight/hash evidence.
93
+
94
+ `command_exit_0` remains a post-hoc workspace correctness check. It is tagged
95
+ as such in the result and is never presented as proof that the agent itself
96
+ executed the same command. Agent execution claims require structured tool
97
+ transcript evidence.
98
+
99
+ ## Retained artifacts
100
+
101
+ Each experiment receives a unique directory under
102
+ `behavioral-evals/results/paired/` containing:
103
+
104
+ - `manifest.json` — pre-run engine/model/effect/threshold/repository/host
105
+ contract;
106
+ - `pairs/*.json` — one immutable record per case/repeat;
107
+ - `<pair-id>/<arm>.transcript.jsonl` and stderr;
108
+ - `<pair-id>/<arm>.workspace/` — final synthetic fixture snapshot excluding
109
+ Harness install artifacts and `.git`;
110
+ - `<pair-id>/<arm>.state/` — isolated Harness state, when any;
111
+ - `summary.json` — included/excluded counts, transitions, paired effect and CI,
112
+ repeat variance, McNemar statistic, and overhead deltas.
113
+
114
+ Pair IDs include a UUID, so repeated runs of the same case never overwrite one
115
+ another.
116
+
117
+ ## Run skill-text pairs
118
+
119
+ Use an explicit model. The same model argument is sent to both arms.
120
+
121
+ ```bash
122
+ npm run eval:paired -- \
123
+ --effect skill-text \
124
+ --engine opencode \
125
+ --model <provider/model> \
126
+ --min-effect-pp <N> \
127
+ --repeats 3
128
+ ```
129
+
130
+ A single case can be targeted with `--case <case-id>`.
131
+
132
+ Claude is also supported for the `skill-text` experiment:
133
+
134
+ ```bash
135
+ npm run eval:paired -- \
136
+ --effect skill-text \
137
+ --engine claude \
138
+ --model <model> \
139
+ --min-effect-pp <N> \
140
+ --case baseline-debugging \
141
+ --repeats 3
142
+ ```
143
+
144
+ Both Claude arms use the same permission/configuration invocation. The old
145
+ runner's baseline-only `--safe-mode` difference is not used in this protocol.
146
+
147
+ ## Run OpenCode plugin-enforcement pairs
148
+
149
+ First produce the #37 attribution evidence from the exact plugin revision being
150
+ measured:
151
+
152
+ ```bash
153
+ npm run eval:opencode:hardlock-live
154
+ ```
155
+
156
+ Then pass that evidence directory into the paired study:
157
+
158
+ ```bash
159
+ npm run eval:paired -- \
160
+ --effect plugin-enforcement \
161
+ --engine opencode \
162
+ --model <provider/model> \
163
+ --min-effect-pp <N> \
164
+ --opencode-preflight benchmarks/results/live-host/opencode-hardlock-<timestamp> \
165
+ --repeats 3
166
+ ```
167
+
168
+ If the preflight fails, is missing, or belongs to a different plugin SHA, the
169
+ study refuses to start rather than silently counting an unattributed plugin
170
+ arm.
171
+
172
+ ## Statistical output
173
+
174
+ For included pairs the summary reports:
175
+
176
+ - baseline and treatment pass/fail counts;
177
+ - `fail->pass`, `pass->fail`, `pass->pass`, and `fail->fail` transitions;
178
+ - paired effect in percentage points: `(treatment passes - baseline passes) / n`;
179
+ - deterministic paired-bootstrap 95% confidence interval;
180
+ - exact McNemar p-value on discordant pairs;
181
+ - per-case repeat count, mean paired difference, and sample variance;
182
+ - excluded pairs with explicit reasons;
183
+ - cost/token/tool-call/duration deltas in a separate section.
184
+
185
+ Historical `opencode-with-plugin` result files pre-dating the live attribution
186
+ protocol are descriptive only and must not be mixed into this paired dataset.
@@ -104,3 +104,49 @@ machine-specific workspace or transcript paths. Each archive includes the case
104
104
  fixture, replay command, fixture/prompt/rubric/code hashes, and engine/model
105
105
  provenance. Historical failures remain unchanged; archived replays are marked
106
106
  pending until a live paired rerun records current execution evidence.
107
+
108
+ ## OpenCode hard-lock live evidence
109
+
110
+ Issue #37 needs a stronger proof than the hermetic state-machine test: a real
111
+ OpenCode host must show an attributed edit attempt being blocked after the same
112
+ verification failure returns post-reflection. Run the dedicated probe on a
113
+ machine with an authenticated `opencode` CLI:
114
+
115
+ ```bash
116
+ npm run eval:opencode:hardlock-live
117
+
118
+ # Optional explicit output/model
119
+ node behavioral-evals/opencode-hardlock-live.js run \
120
+ --out benchmarks/results/live-host/opencode-hardlock-local \
121
+ --model opencode/union-alpha
122
+
123
+ # Re-check a retained evidence directory without making a model call
124
+ node behavioral-evals/opencode-hardlock-live.js verify \
125
+ benchmarks/results/live-host/opencode-hardlock-local
126
+ ```
127
+
128
+ The runner uses an isolated temporary `HARNESS_STATE_HOME`, installs the
129
+ canonical `opencode-plugin/index.mjs` as the auto-discoverable
130
+ `.opencode/plugins/harness-enforcement.js`, and keeps the verification failure
131
+ constant. It does **not** seed breaker/reflection state. The live agent must
132
+ produce the reflection artifact through the host/plugin flow.
133
+
134
+ A PASS requires all of the following evidence to agree:
135
+
136
+ - a structured pre-lock edit attempt containing `PRELOCK_OK`, and that marker
137
+ must reach `probe.txt` (negative control showing edits were possible);
138
+ - a preserved reflection artifact with the current token and `RESUME:` decision;
139
+ - a post-reflection retry containing `POST_REFLECTION_RETRY`, which must reach
140
+ the file and drive the same failure signature to at least count 4;
141
+ - a pre-reset `circuit-breaker.json` snapshot with `hardLock: true`;
142
+ - a structured final edit attempt containing `SHOULD_NOT_LAND` plus the live
143
+ Harness hard-lock error in host output;
144
+ - `SHOULD_NOT_LAND` must **not** appear in the final file, proving the attempted
145
+ edit was blocked before filesystem mutation.
146
+
147
+ The runner archives raw transcript/stderr, sanitized host/model/plugin metadata,
148
+ the final fixture files, the selected pre-reset state, the reflection artifact,
149
+ and the complete isolated raw-state tree before deleting its temporary working
150
+ directory. A failed/incomplete live run is retained as failed evidence rather
151
+ than being counted as behavioral failure. This probe is intended to become the
152
+ OpenCode plugin-arm preflight for the paired benchmark tracked by #71.