harness-everything 0.5.2 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/AGENTS.md +5 -1
- package/BENCHMARK_SOP.md +1 -1
- package/CHANGELOG.md +12 -0
- package/README.md +3 -3
- package/VERIFICATION.md +5 -5
- package/behavioral-evals/PAIRED-BENCHMARK.md +186 -0
- package/behavioral-evals/README.md +46 -0
- package/behavioral-evals/opencode-hardlock-live.js +301 -0
- package/behavioral-evals/paired-benchmark.js +781 -0
- package/behavioral-evals/run-with-plugin.js +1 -1
- package/benchmarks/results/live-host/opencode-2026-09-16/README.md +133 -0
- package/benchmarks/results/live-host/opencode-2026-09-16/probe-control.md +88 -0
- package/benchmarks/results/live-host/opencode-2026-09-16/run-commands.md +100 -0
- package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/circuit-breaker.json +16 -0
- package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/compliance.json +7 -0
- package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/edit-state.json +10 -0
- package/benchmarks/results/live-host/opencode-2026-09-16/state/sessions/ses_f5520f1d9ffeB8KLTIUKgA5ng8/zoom-out-report.md +29 -0
- package/ci/doc-capability-consistency.test.js +56 -5
- package/ci/mechanism-2w-platform-compatibility.test.js +20 -7
- package/ci/mechanism-30-opencode-plugin-loadability.test.js +86 -0
- package/ci/mechanism-31-opencode-hardlock-evidence.test.js +138 -0
- package/ci/mechanism-32-paired-benchmark.test.js +254 -0
- package/docs/architecture.md +2 -2
- package/docs/platform-capabilities.md +3 -3
- package/docs/platform-compatibility.json +4 -4
- package/opencode-plugin/README.md +32 -11
- package/opencode-plugin/plugin.json +1 -1
- package/package.json +5 -1
- package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
- package/plugins/harness-everything/plugin.json +1 -1
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
"name": "harness-everything",
|
|
11
11
|
"source": "./",
|
|
12
12
|
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
13
|
-
"version": "0.
|
|
13
|
+
"version": "0.6.0",
|
|
14
14
|
"author": {
|
|
15
15
|
"name": "Miya Daniel"
|
|
16
16
|
},
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "harness-everything",
|
|
3
3
|
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.6.0",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Miya Daniel"
|
|
7
7
|
},
|
package/AGENTS.md
CHANGED
|
@@ -13,7 +13,7 @@ You are modifying **Harness**: an orchestrated agent operating system (skills +
|
|
|
13
13
|
| `<skill-name>/SKILL.md` | One canonical skill per top-level directory. Frontmatter: `name`, `description`, `metadata.author`, `metadata.version`. |
|
|
14
14
|
| `hooks/` | Claude Code lifecycle hook scripts and shared mechanisms such as circuit breaker, scope guard, and stop gate. Do not infer that every host packages every hook. |
|
|
15
15
|
| `harness-everything/scripts/` | Runtime: kernel/tier routing, verification, bootstrap, self-heal. |
|
|
16
|
-
| `opencode-plugin/` | OpenCode native plugin implementation. Mechanism-tested
|
|
16
|
+
| `opencode-plugin/` | OpenCode native plugin implementation. Mechanism-tested, with partial live-host evidence for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS); see the evidence boundary below. |
|
|
17
17
|
| `.agents/plugins/` + `plugins/harness-everything/` | Local OpenAI/Codex marketplace + `.codex-plugin` package with canonical skill copies and packaged session/prompt hooks. |
|
|
18
18
|
| `submission/openai/` | Public OpenAI Skills-only listing/test inputs. The public artifact does not include local `.codex-plugin` lifecycle hooks. |
|
|
19
19
|
| `evals/<skill>/` | Trigger/routing eval per skill (waza format). Required for every canonical routed skill. |
|
|
@@ -25,6 +25,10 @@ You are modifying **Harness**: an orchestrated agent operating system (skills +
|
|
|
25
25
|
| `.nvmrc` | Primary development/CI Node runtime. `package.json#engines.node` remains the minimum supported runtime. |
|
|
26
26
|
| `docs/`, `references/` | Philosophy, architecture, workflow documentation, shared checklists. |
|
|
27
27
|
|
|
28
|
+
## OpenCode evidence boundary
|
|
29
|
+
|
|
30
|
+
The retained [OpenCode evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md) supports only the scoped loading and state effects above. The final snapshot is post-reset (`hardLock: false`, `count: 1`). Hard lock is only an interactive observation with no retained blocked-tool trace; reflection was operator-seeded, then agent-rewritten. Agent-controlled state deletion resets the breaker, so do not claim durable hard enforcement or behavioral effectiveness. `.mjs` auto-discovery is broken on this host version. Global scope, npm-package installation, and other host versions remain unverified.
|
|
31
|
+
|
|
28
32
|
## Non-Negotiable Change Rules
|
|
29
33
|
|
|
30
34
|
1. **Every canonical SKILL.md keeps ≤ 500 tokens** (hard CI gate) and must contain `## USE FOR:` and `## DO NOT USE FOR:` sections. Deep detail goes in the skill's `references/` or `guides/`.
|
package/BENCHMARK_SOP.md
CHANGED
|
@@ -4,7 +4,7 @@ This guide helps you objectively test the performance difference between an AI A
|
|
|
4
4
|
|
|
5
5
|
**What this SOP does NOT test:** it compares emergent behavior (did the agent avoid the loop, did it stay in scope), not whether an underlying hook/plugin mechanism actually loaded, fired, or blocked anything. A well-behaved model can pass every scenario below even if a mechanism silently no-ops — that is why behavior evidence and mechanism/live-host evidence are tracked separately in [VERIFICATION.md](VERIFICATION.md).
|
|
6
6
|
|
|
7
|
-
Do not classify mechanism support by product name alone. A host can expose more than one Harness surface. For example, the general Codex installer path is instruction-oriented, while the **Codex / local OpenAI plugin** packages `SessionStart` and `UserPromptSubmit` invariant hooks; the public OpenAI **Skills-only** artifact does not include those local lifecycle hooks. OpenCode has a
|
|
7
|
+
Do not classify mechanism support by product name alone. A host can expose more than one Harness surface. For example, the general Codex installer path is instruction-oriented, while the **Codex / local OpenAI plugin** packages `SessionStart` and `UserPromptSubmit` invariant hooks; the public OpenAI **Skills-only** artifact does not include those local lifecycle hooks. OpenCode has deterministic mechanism coverage and partial live-host evidence for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS). The [retained evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md) is a post-reset snapshot, not a full enforcement history. Hard lock is only an interactive observation with no retained blocked-tool trace; reflection was operator-seeded, then agent-rewritten. Agent-controlled state deletion resets the breaker; neither durable hard enforcement nor behavioral effectiveness is proven. Global scope, npm-package installation, and other host versions remain unverified. Use [docs/platform-capabilities.md](docs/platform-capabilities.md) as the current source of truth.
|
|
8
8
|
|
|
9
9
|
This SOP is therefore a **behavior** benchmark. Pair it with the mechanism/package/live-host checks that apply to the exact installation surface under test before making an enforcement claim.
|
|
10
10
|
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,18 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project are documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.6.0](https://github.com/dyphn1/Harness-everything/compare/v0.5.3...v0.6.0) (2026-09-17)
|
|
6
|
+
|
|
7
|
+
### Features
|
|
8
|
+
|
|
9
|
+
* **behavioral:** add paired benchmark contract ([42d52b3](https://github.com/dyphn1/Harness-everything/commit/42d52b3a6db89fa27cd2239726a22fe119299932))
|
|
10
|
+
|
|
11
|
+
## [0.5.3](https://github.com/dyphn1/Harness-everything/compare/v0.5.2...v0.5.3) (2026-09-17)
|
|
12
|
+
|
|
13
|
+
### Bug Fixes
|
|
14
|
+
|
|
15
|
+
* **opencode:** Install discoverable plugin and guard loadability ([5645c08](https://github.com/dyphn1/Harness-everything/commit/5645c080c794b82afd50a0206cf35d2dd3fcd54c))
|
|
16
|
+
|
|
5
17
|
## [0.5.2](https://github.com/dyphn1/Harness-everything/compare/v0.5.1...v0.5.2) (2026-09-16)
|
|
6
18
|
|
|
7
19
|
### Bug Fixes
|
package/README.md
CHANGED
|
@@ -31,7 +31,7 @@ Harness deliberately follows a **minimal rails, maximum freedom** design: the ru
|
|
|
31
31
|
| **System Audit** | Blindly assumes shell syntax | Requires manual shell check | Uses environment detection and explicit verification paths instead of assuming one shell/runtime |
|
|
32
32
|
| **Memory** | Resets on every new chat session | Static text rules | Stateful hook/plugin integrations can persist bounded runtime state; instruction-only integrations cannot |
|
|
33
33
|
|
|
34
|
-
The exact "Harness" behavior depends on the installation surface. Claude Code has the broadest currently verified lifecycle-hook coverage. OpenCode has
|
|
34
|
+
The exact "Harness" behavior depends on the installation surface. Claude Code has the broadest currently verified lifecycle-hook coverage. OpenCode has deterministic mechanism tests and **partial live-host evidence** for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS), not durable hard enforcement; global scope and npm install remain unverified. Codex has both an advisory `--codex` installer path and a **local OpenAI plugin** path with mechanism-tested session, prompt, supported-tool, subagent, and stop hooks. The public OpenAI **Skills-only** submission is narrower and does not include those local lifecycle hooks. See [Supported AI IDEs & Tools](#supported-ai-ides--tools) and [docs/platform-capabilities.md](docs/platform-capabilities.md).
|
|
35
35
|
|
|
36
36
|
### When should I use Harness?
|
|
37
37
|
* You regularly use agentic coding tools (like Claude Code, Cursor, or Copilot) on medium-to-large codebases.
|
|
@@ -158,12 +158,12 @@ Harness operates through six core cognitive concepts:
|
|
|
158
158
|
|
|
159
159
|
The authoritative current matrix is [docs/platform-capabilities.md](docs/platform-capabilities.md). The important distinction is that **one host can have multiple Harness installation surfaces**.
|
|
160
160
|
|
|
161
|
-
[`opencode-plugin/`](opencode-plugin/) (`index.mjs`) implements a verification gate and a Rule of 3 breaker against OpenCode's real, source-verified plugin API (`tool.execute.before`/`.after`, the `session.idle` event). `ci/mechanism-2n-opencode-plugin.test.js` drives the exported hooks directly against a mock context matching that API.
|
|
161
|
+
[`opencode-plugin/`](opencode-plugin/) (`index.mjs`) implements a verification gate and a Rule of 3 breaker against OpenCode's real, source-verified plugin API (`tool.execute.before`/`.after`, the `session.idle` event). `ci/mechanism-2n-opencode-plugin.test.js` drives the exported hooks directly against a mock context matching that API. Retained evidence supports project-scope `.js` plugin loading and edit/verification state on OpenCode 1.18.31 (macOS): [live-host evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md). The final snapshot is post-reset (`hardLock: false`, `count: 1`); hard lock is only an interactive observation with no retained blocked-tool trace. Reflection was operator-seeded, then agent-rewritten, not unaided. Agent-controlled state deletion resets the breaker, so this does not prove durable hard enforcement or behavioral effectiveness. Install with a **`.js`** destination name: this host silently ignores `.mjs` (issue #127, guarded by `test:opencode:loadability`). Global scope, npm-package installation, and other host versions remain unverified.
|
|
162
162
|
|
|
163
163
|
| AI Agent Tool / Surface | Integration Method | Local Target Location | Enforcement claim |
|
|
164
164
|
|---|---|---|---|
|
|
165
165
|
| **Claude Code** | Native lifecycle hooks (`PreToolUse`, `PostToolUse`, `SessionStart`, `UserPromptSubmit`, `Stop`) | `.claude/settings.json`, `.claude/skills/`, `.claude/agents/` | **Hard** for the supported verified hook gates |
|
|
166
|
-
| **OpenCode** | Native plugin module ([`opencode-plugin/index.mjs`](opencode-plugin/index.mjs)) | `.opencode/plugins/` | **
|
|
166
|
+
| **OpenCode** | Native plugin module ([`opencode-plugin/index.mjs`](opencode-plugin/index.mjs), install as `harness-enforcement.js`) | `.opencode/plugins/` | **Partial live-host evidence** — project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS); hard lock reported only, no retained blocked-tool trace; not durable hard enforcement |
|
|
167
167
|
| **Codex — general installer path** | Skills + `AGENTS.md` guidance | `AGENTS.md` + repo-scoped `.agents/skills/` | **Advisory/instruction-oriented** where the host only consumes instructions |
|
|
168
168
|
| **Codex / local OpenAI plugin** | `.codex-plugin` package with session, prompt, supported-tool, subagent, and stop hooks plus 26 canonical skills | `.agents/plugins/marketplace.json` → `plugins/harness-everything/` | **Mechanism-tested local enforcement** for the packaged mappings; live host loading remains unverified |
|
|
169
169
|
| **Public OpenAI Skills-only plugin** | Public Skills-only bundle | Generated submission ZIP from `plugins/harness-everything/skills/` | **Skill/workflow behavior only**; no local `.codex-plugin` lifecycle hooks in the public artifact |
|
package/VERIFICATION.md
CHANGED
|
@@ -22,7 +22,7 @@ First identify **which Harness surface you are testing**. One host can have more
|
|
|
22
22
|
| Surface | Artifact / location | What artifact presence proves |
|
|
23
23
|
|---|---|---|
|
|
24
24
|
| Claude Code | `.claude/settings.json`, `.claude/skills/`, `.claude/agents/` | The Claude installer/plugin wrote its lifecycle-hook configuration and content |
|
|
25
|
-
| OpenCode | `.opencode/plugins
|
|
25
|
+
| OpenCode | `.opencode/plugins/harness-enforcement.js` (copied from `opencode-plugin/index.mjs`) | The loadable plugin module exists under a `.js` name; this alone does not prove live loading |
|
|
26
26
|
| Codex — general installer | `AGENTS.md`, repo-scoped `.agents/skills/` | Advisory/instruction integration and project Agent Skills are installed |
|
|
27
27
|
| **Codex / local OpenAI plugin** | `.agents/plugins/marketplace.json`, `plugins/harness-everything/.codex-plugin/plugin.json`, packaged hooks/skills | The local OpenAI plugin package is structurally present |
|
|
28
28
|
| **Public OpenAI Skills-only** | generated `dist/openai-submission/harness-everything-skills.zip` | The public-review bundle was generated; local `.codex-plugin` lifecycle hooks are intentionally not part of this artifact |
|
|
@@ -127,9 +127,9 @@ The repository implements the real OpenCode plugin API and has deterministic mec
|
|
|
127
127
|
npm run test:mechanism
|
|
128
128
|
```
|
|
129
129
|
|
|
130
|
-
and inspect the OpenCode-specific mechanism
|
|
130
|
+
and inspect the OpenCode-specific mechanism suites (`ci/mechanism-2n-opencode-plugin.test.js` for hook behavior, `ci/mechanism-30-opencode-plugin-loadability.test.js` for install-filename loadability).
|
|
131
131
|
|
|
132
|
-
**Current evidence boundary:**
|
|
132
|
+
**Current evidence boundary:** deterministic mechanism coverage plus partial live-host evidence for project-scope `.js` loading and edit/verification state on OpenCode 1.18.31 (macOS). See [retained evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md). The final snapshot is post-reset (`hardLock: false`, `count: 1`), not a history of the enforcement sequence. Hard lock is only an interactive observation with no retained blocked-tool trace. Reflection was operator-seeded, then agent-rewritten; follow-up delivery is reported/state-consistent, without a retained transcript. Agent-controlled state deletion resets the breaker, so neither durable hard enforcement nor behavioral effectiveness is proven. `.mjs` auto-discovery is broken on this host version (issue #127). Global scope, npm-package installation, and other host versions remain unverified.
|
|
133
133
|
|
|
134
134
|
---
|
|
135
135
|
|
|
@@ -142,7 +142,7 @@ npm run test:docs:capabilities
|
|
|
142
142
|
npm run test:consistency
|
|
143
143
|
```
|
|
144
144
|
|
|
145
|
-
The capability test checks the current-state documentation surfaces for stale platform paths and pre-plugin claims, requires the Codex local-plugin vs public Skills-only distinction,
|
|
145
|
+
The capability test checks the current-state documentation surfaces for stale platform paths and pre-plugin claims, requires the Codex local-plugin vs public Skills-only distinction, requires the OpenCode retained artifact path, host/version/OS and project `.js` scope, and explicit partial-evidence limitations, and records the Hermes native user skill target plus the Continue installer candidate boundary.
|
|
146
146
|
|
|
147
147
|
When changing platform integration behavior, update [docs/platform-capabilities.md](docs/platform-capabilities.md) and all affected current-state docs in the same PR.
|
|
148
148
|
|
|
@@ -237,7 +237,7 @@ Fill in one row **per installation surface**, not merely per brand name.
|
|
|
237
237
|
|---|---|---|---|---|---|---|
|
|
238
238
|
| Package/artifact present | | | | | | |
|
|
239
239
|
| Deterministic mechanism tests | | | N/A/explicit CLI only | | N/A (skill bundle) | N/A |
|
|
240
|
-
| Live host loaded mechanism | | **
|
|
240
|
+
| Live host loaded mechanism | | **loading verified; lifecycle evidence partial** (project scope, OpenCode 1.18.31, macOS, `.js`) — [evidence](benchmarks/results/live-host/opencode-2026-09-16/README.md) | N/A | | N/A | N/A |
|
|
241
241
|
| Routing/invariant behavior | | | | | | |
|
|
242
242
|
| Objective verification discipline | | | | | | |
|
|
243
243
|
| Failure recovery discipline | | | | | | |
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# Paired Behavioral Benchmark Protocol
|
|
2
|
+
|
|
3
|
+
This protocol is the measurement surface for issue #71. It is intentionally
|
|
4
|
+
separate from mechanism tests: mechanism tests prove deterministic hook and
|
|
5
|
+
state-machine behavior, while this benchmark measures agent behavior under a
|
|
6
|
+
real model session.
|
|
7
|
+
|
|
8
|
+
## Separate causal questions
|
|
9
|
+
|
|
10
|
+
Never combine these interventions into one headline number.
|
|
11
|
+
|
|
12
|
+
### `skill-text`
|
|
13
|
+
|
|
14
|
+
Measures the incremental effect of Harness skill guidance only.
|
|
15
|
+
|
|
16
|
+
- baseline: no Harness skill text, no Harness hooks/plugin enforcement;
|
|
17
|
+
- treatment: the case's named Harness skill text only;
|
|
18
|
+
- same engine, explicit model, prompt, fixture, rubric, CLI policy, and tool
|
|
19
|
+
availability in both arms.
|
|
20
|
+
|
|
21
|
+
The treatment does **not** install `hooks/hooks.json` or the OpenCode
|
|
22
|
+
`harness-enforcement.js` plugin. This removes the old confound where a
|
|
23
|
+
"skill" treatment also changed enforcement behavior.
|
|
24
|
+
|
|
25
|
+
### `plugin-enforcement`
|
|
26
|
+
|
|
27
|
+
Measures the incremental effect of OpenCode enforcement while holding skill
|
|
28
|
+
text constant.
|
|
29
|
+
|
|
30
|
+
- baseline: named Harness skill text, no OpenCode enforcement plugin;
|
|
31
|
+
- treatment: the same named Harness skill text plus the canonical OpenCode
|
|
32
|
+
enforcement module installed as `.opencode/plugins/harness-enforcement.js`.
|
|
33
|
+
|
|
34
|
+
This experiment currently supports OpenCode only. It requires a retained
|
|
35
|
+
hard-lock preflight produced by `opencode-hardlock-live.js`. The preflight must
|
|
36
|
+
still pass verification and its plugin SHA-256 must equal the current canonical
|
|
37
|
+
`opencode-plugin/index.mjs`; changing the plugin invalidates the old preflight.
|
|
38
|
+
|
|
39
|
+
Each treatment case must also leave retained plugin state proving that the
|
|
40
|
+
plugin fired during that case. Missing attribution is classified as
|
|
41
|
+
infrastructure/inconclusive and is excluded from behavioral pass rates.
|
|
42
|
+
|
|
43
|
+
### Cost and execution overhead
|
|
44
|
+
|
|
45
|
+
Cost is reported separately from correctness:
|
|
46
|
+
|
|
47
|
+
- provider cost, where exposed by the host;
|
|
48
|
+
- total tokens, where exposed;
|
|
49
|
+
- attempted tool calls;
|
|
50
|
+
- elapsed duration.
|
|
51
|
+
|
|
52
|
+
The report gives treatment-minus-baseline deltas and does not fold them into a
|
|
53
|
+
correctness score.
|
|
54
|
+
|
|
55
|
+
## Predeclare the decision threshold
|
|
56
|
+
|
|
57
|
+
There is deliberately **no default minimum effect**. Before model calls start,
|
|
58
|
+
the caller must supply `--min-effect-pp <N>` to state the smallest paired pass
|
|
59
|
+
rate improvement that would be practically meaningful for that study. The
|
|
60
|
+
experiment manifest is written before either arm runs, so the threshold cannot
|
|
61
|
+
be selected after seeing the result.
|
|
62
|
+
|
|
63
|
+
Example only (not a project-wide default):
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
--min-effect-pp 10
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
A study reports `meets-predeclared-threshold` only when the lower bound of the
|
|
70
|
+
paired bootstrap 95% confidence interval is at or above that predeclared
|
|
71
|
+
minimum. It reports `below-predeclared-threshold` when the interval lies wholly
|
|
72
|
+
below it, otherwise `uncertain-at-predeclared-threshold`. Fewer than two
|
|
73
|
+
included pairs is `insufficient-data`.
|
|
74
|
+
|
|
75
|
+
The report also includes an exact McNemar p-value for the discordant pairs. The
|
|
76
|
+
p-value is evidence about symmetry of fail->pass vs pass->fail transitions; it
|
|
77
|
+
is not a substitute for the predeclared practical-effect threshold.
|
|
78
|
+
|
|
79
|
+
## Evidence and exclusions
|
|
80
|
+
|
|
81
|
+
A pair enters the behavioral denominator only when both arms are definitive
|
|
82
|
+
`pass` or `fail` and the pair contract validates.
|
|
83
|
+
|
|
84
|
+
Excluded reasons include:
|
|
85
|
+
|
|
86
|
+
- CLI/session error;
|
|
87
|
+
- grader error;
|
|
88
|
+
- incomplete or inconclusive transcript evidence;
|
|
89
|
+
- mismatched actual model when both hosts report it;
|
|
90
|
+
- pair-contract hash mismatch;
|
|
91
|
+
- intervention shape mismatch;
|
|
92
|
+
- missing OpenCode plugin attribution/preflight/hash evidence.
|
|
93
|
+
|
|
94
|
+
`command_exit_0` remains a post-hoc workspace correctness check. It is tagged
|
|
95
|
+
as such in the result and is never presented as proof that the agent itself
|
|
96
|
+
executed the same command. Agent execution claims require structured tool
|
|
97
|
+
transcript evidence.
|
|
98
|
+
|
|
99
|
+
## Retained artifacts
|
|
100
|
+
|
|
101
|
+
Each experiment receives a unique directory under
|
|
102
|
+
`behavioral-evals/results/paired/` containing:
|
|
103
|
+
|
|
104
|
+
- `manifest.json` — pre-run engine/model/effect/threshold/repository/host
|
|
105
|
+
contract;
|
|
106
|
+
- `pairs/*.json` — one immutable record per case/repeat;
|
|
107
|
+
- `<pair-id>/<arm>.transcript.jsonl` and stderr;
|
|
108
|
+
- `<pair-id>/<arm>.workspace/` — final synthetic fixture snapshot excluding
|
|
109
|
+
Harness install artifacts and `.git`;
|
|
110
|
+
- `<pair-id>/<arm>.state/` — isolated Harness state, when any;
|
|
111
|
+
- `summary.json` — included/excluded counts, transitions, paired effect and CI,
|
|
112
|
+
repeat variance, McNemar statistic, and overhead deltas.
|
|
113
|
+
|
|
114
|
+
Pair IDs include a UUID, so repeated runs of the same case never overwrite one
|
|
115
|
+
another.
|
|
116
|
+
|
|
117
|
+
## Run skill-text pairs
|
|
118
|
+
|
|
119
|
+
Use an explicit model. The same model argument is sent to both arms.
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
npm run eval:paired -- \
|
|
123
|
+
--effect skill-text \
|
|
124
|
+
--engine opencode \
|
|
125
|
+
--model <provider/model> \
|
|
126
|
+
--min-effect-pp <N> \
|
|
127
|
+
--repeats 3
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
A single case can be targeted with `--case <case-id>`.
|
|
131
|
+
|
|
132
|
+
Claude is also supported for the `skill-text` experiment:
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
npm run eval:paired -- \
|
|
136
|
+
--effect skill-text \
|
|
137
|
+
--engine claude \
|
|
138
|
+
--model <model> \
|
|
139
|
+
--min-effect-pp <N> \
|
|
140
|
+
--case baseline-debugging \
|
|
141
|
+
--repeats 3
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Both Claude arms use the same permission/configuration invocation. The old
|
|
145
|
+
runner's baseline-only `--safe-mode` difference is not used in this protocol.
|
|
146
|
+
|
|
147
|
+
## Run OpenCode plugin-enforcement pairs
|
|
148
|
+
|
|
149
|
+
First produce the #37 attribution evidence from the exact plugin revision being
|
|
150
|
+
measured:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
npm run eval:opencode:hardlock-live
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Then pass that evidence directory into the paired study:
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
npm run eval:paired -- \
|
|
160
|
+
--effect plugin-enforcement \
|
|
161
|
+
--engine opencode \
|
|
162
|
+
--model <provider/model> \
|
|
163
|
+
--min-effect-pp <N> \
|
|
164
|
+
--opencode-preflight benchmarks/results/live-host/opencode-hardlock-<timestamp> \
|
|
165
|
+
--repeats 3
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
If the preflight fails, is missing, or belongs to a different plugin SHA, the
|
|
169
|
+
study refuses to start rather than silently counting an unattributed plugin
|
|
170
|
+
arm.
|
|
171
|
+
|
|
172
|
+
## Statistical output
|
|
173
|
+
|
|
174
|
+
For included pairs the summary reports:
|
|
175
|
+
|
|
176
|
+
- baseline and treatment pass/fail counts;
|
|
177
|
+
- `fail->pass`, `pass->fail`, `pass->pass`, and `fail->fail` transitions;
|
|
178
|
+
- paired effect in percentage points: `(treatment passes - baseline passes) / n`;
|
|
179
|
+
- deterministic paired-bootstrap 95% confidence interval;
|
|
180
|
+
- exact McNemar p-value on discordant pairs;
|
|
181
|
+
- per-case repeat count, mean paired difference, and sample variance;
|
|
182
|
+
- excluded pairs with explicit reasons;
|
|
183
|
+
- cost/token/tool-call/duration deltas in a separate section.
|
|
184
|
+
|
|
185
|
+
Historical `opencode-with-plugin` result files pre-dating the live attribution
|
|
186
|
+
protocol are descriptive only and must not be mixed into this paired dataset.
|
|
@@ -104,3 +104,49 @@ machine-specific workspace or transcript paths. Each archive includes the case
|
|
|
104
104
|
fixture, replay command, fixture/prompt/rubric/code hashes, and engine/model
|
|
105
105
|
provenance. Historical failures remain unchanged; archived replays are marked
|
|
106
106
|
pending until a live paired rerun records current execution evidence.
|
|
107
|
+
|
|
108
|
+
## OpenCode hard-lock live evidence
|
|
109
|
+
|
|
110
|
+
Issue #37 needs a stronger proof than the hermetic state-machine test: a real
|
|
111
|
+
OpenCode host must show an attributed edit attempt being blocked after the same
|
|
112
|
+
verification failure returns post-reflection. Run the dedicated probe on a
|
|
113
|
+
machine with an authenticated `opencode` CLI:
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
npm run eval:opencode:hardlock-live
|
|
117
|
+
|
|
118
|
+
# Optional explicit output/model
|
|
119
|
+
node behavioral-evals/opencode-hardlock-live.js run \
|
|
120
|
+
--out benchmarks/results/live-host/opencode-hardlock-local \
|
|
121
|
+
--model opencode/union-alpha
|
|
122
|
+
|
|
123
|
+
# Re-check a retained evidence directory without making a model call
|
|
124
|
+
node behavioral-evals/opencode-hardlock-live.js verify \
|
|
125
|
+
benchmarks/results/live-host/opencode-hardlock-local
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
The runner uses an isolated temporary `HARNESS_STATE_HOME`, installs the
|
|
129
|
+
canonical `opencode-plugin/index.mjs` as the auto-discoverable
|
|
130
|
+
`.opencode/plugins/harness-enforcement.js`, and keeps the verification failure
|
|
131
|
+
constant. It does **not** seed breaker/reflection state. The live agent must
|
|
132
|
+
produce the reflection artifact through the host/plugin flow.
|
|
133
|
+
|
|
134
|
+
A PASS requires all of the following evidence to agree:
|
|
135
|
+
|
|
136
|
+
- a structured pre-lock edit attempt containing `PRELOCK_OK`, and that marker
|
|
137
|
+
must reach `probe.txt` (negative control showing edits were possible);
|
|
138
|
+
- a preserved reflection artifact with the current token and `RESUME:` decision;
|
|
139
|
+
- a post-reflection retry containing `POST_REFLECTION_RETRY`, which must reach
|
|
140
|
+
the file and drive the same failure signature to at least count 4;
|
|
141
|
+
- a pre-reset `circuit-breaker.json` snapshot with `hardLock: true`;
|
|
142
|
+
- a structured final edit attempt containing `SHOULD_NOT_LAND` plus the live
|
|
143
|
+
Harness hard-lock error in host output;
|
|
144
|
+
- `SHOULD_NOT_LAND` must **not** appear in the final file, proving the attempted
|
|
145
|
+
edit was blocked before filesystem mutation.
|
|
146
|
+
|
|
147
|
+
The runner archives raw transcript/stderr, sanitized host/model/plugin metadata,
|
|
148
|
+
the final fixture files, the selected pre-reset state, the reflection artifact,
|
|
149
|
+
and the complete isolated raw-state tree before deleting its temporary working
|
|
150
|
+
directory. A failed/incomplete live run is retained as failed evidence rather
|
|
151
|
+
than being counted as behavioral failure. This probe is intended to become the
|
|
152
|
+
OpenCode plugin-arm preflight for the paired benchmark tracked by #71.
|