harness-everything 0.11.6 → 0.11.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +13 -0
- package/README.md +7 -3
- package/ci/mechanism-2a-rule-of-3.test.js +71 -0
- package/ci/mechanism-2ah-workflow-lifecycle.test.js +139 -0
- package/ci/mechanism-2c-state-persist.test.js +45 -2
- package/ci/mechanism-2t-issue-20.test.js +0 -8
- package/ci/mechanism-30-opencode-plugin-loadability.test.js +39 -7
- package/docs/architecture.md +17 -0
- package/docs/audit.md +52 -5
- package/docs/release-evidence.md +3 -0
- package/docs/workflow-runtime.md +13 -0
- package/docs/workflows/environment-detection.md +47 -37
- package/docs/workflows/eval-harness.md +45 -38
- package/docs/workflows/fable-discipline.md +45 -33
- package/docs/workflows/git-commit.md +53 -45
- package/docs/workflows/grill-me.md +36 -38
- package/docs/workflows/grill-with-docs.md +36 -46
- package/docs/workflows/harness-everything.md +59 -57
- package/docs/workflows/improve-codebase-architecture.md +36 -44
- package/docs/workflows/install-cognitive-os.md +36 -29
- package/docs/workflows/repo-docs.md +29 -40
- package/docs/workflows/rewrite-commits.md +40 -36
- package/docs/workflows/security-review.md +34 -53
- package/docs/workflows/skill-creator.md +39 -56
- package/docs/workflows/skill-style.md +35 -40
- package/docs/workflows/using-git-worktrees.md +40 -44
- package/docs/workflows/verification-loop.md +43 -48
- package/docs/workflows/verify-before-claim.md +37 -40
- package/docs/workflows/zoom-out.md +39 -51
- package/harness-everything/SKILL.md +1 -1
- package/harness-everything/scripts/kernel-router.js +38 -10
- package/hooks/scripts/lib/workflow-isolation.js +30 -1
- package/hooks/scripts/rule-of-3-tracker.js +17 -9
- package/hooks/scripts/state-persist.js +14 -6
- package/hooks/scripts/workflow-gate.js +5 -1
- package/opencode-plugin/plugin.json +1 -1
- package/package.json +1 -1
- package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
- package/plugins/harness-everything/hooks/scripts/lib/workflow-isolation.js +30 -1
- package/plugins/harness-everything/hooks/scripts/rule-of-3-tracker.js +17 -9
- package/plugins/harness-everything/hooks/scripts/state-persist.js +14 -6
- package/plugins/harness-everything/hooks/scripts/workflow-gate.js +5 -1
- package/plugins/harness-everything/plugin.json +1 -1
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +38 -10
- package/docs/issue-20-rollup.md +0 -47
- package/docs/reviews/pr-142-issue-131.md +0 -89
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
"name": "harness-everything",
|
|
11
11
|
"source": "./",
|
|
12
12
|
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
13
|
-
"version": "0.11.
|
|
13
|
+
"version": "0.11.7",
|
|
14
14
|
"author": {
|
|
15
15
|
"name": "Miya Daniel"
|
|
16
16
|
},
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "harness-everything",
|
|
3
3
|
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
4
|
-
"version": "0.11.
|
|
4
|
+
"version": "0.11.7",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Miya Daniel"
|
|
7
7
|
},
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,19 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project are documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.11.7](https://github.com/dyphn1/Harness-everything/compare/v0.11.6...v0.11.7) (2026-09-18)
|
|
6
|
+
|
|
7
|
+
### Bug Fixes
|
|
8
|
+
|
|
9
|
+
* Claude success path, rule-of-3 category, worktree scan ([fd8c9e7](https://github.com/dyphn1/Harness-everything/commit/fd8c9e79261a18a79883029f7179d94c18ec302e)), closes [#153](https://github.com/dyphn1/Harness-everything/issues/153) [#166](https://github.com/dyphn1/Harness-everything/issues/166) [#167](https://github.com/dyphn1/Harness-everything/issues/167) [#170](https://github.com/dyphn1/Harness-everything/issues/170)
|
|
10
|
+
* fail closed on unbound host notifications ([48faf8c](https://github.com/dyphn1/Harness-everything/commit/48faf8caff38e2cc77d2eae0169d6ba62767d7de))
|
|
11
|
+
* fail closed on unbound host notifications ([fea7201](https://github.com/dyphn1/Harness-everything/commit/fea72018ba74b6dc2cf6d273b4116a3503fdcda2))
|
|
12
|
+
* Host notification routing and out-of-workspace edit accounting ([508a321](https://github.com/dyphn1/Harness-everything/commit/508a32141389612f8fc1c92f89ebe4fde88cee1e)), closes [#172](https://github.com/dyphn1/Harness-everything/issues/172) [#165](https://github.com/dyphn1/Harness-everything/issues/165) [#168](https://github.com/dyphn1/Harness-everything/issues/168)
|
|
13
|
+
* ignore host notifications before routing ([ad355fd](https://github.com/dyphn1/Harness-everything/commit/ad355fdd98f45747b467d034adc4bf6fbcf1404b))
|
|
14
|
+
* ignore host notifications before routing ([ea96577](https://github.com/dyphn1/Harness-everything/commit/ea96577d6f2916261a3a4484bfdb5358c828d475))
|
|
15
|
+
* integrate Claude PostToolUse semantics ([6c3b447](https://github.com/dyphn1/Harness-everything/commit/6c3b447e7112c8a8984ad304de37c8819f17be27))
|
|
16
|
+
* integrate Claude PostToolUse semantics ([a7077e9](https://github.com/dyphn1/Harness-everything/commit/a7077e9240b76a53367198c32a9d0c2645fbbc32))
|
|
17
|
+
|
|
5
18
|
## [0.11.6](https://github.com/dyphn1/Harness-everything/compare/v0.11.5...v0.11.6) (2026-09-18)
|
|
6
19
|
|
|
7
20
|
### Bug Fixes
|
package/README.md
CHANGED
|
@@ -82,6 +82,7 @@ The general installer only writes to your workspace (or, with `--global`, your h
|
|
|
82
82
|
| `AGENTS.md` + `.agents/skills/` | Codex advisory instructions plus repo-scoped Agent Skills; Hermes can also consume trusted project skills from `.agents/skills/` |
|
|
83
83
|
| `.continue/rules/harness.md` + `.continue/skills/` | Continue.dev advisory rule and installer candidate skill path; standalone skill discovery is `Unknown` |
|
|
84
84
|
| `.hermes.md` | Hermes Agent project advisory context |
|
|
85
|
+
| `.opencode/plugins/harness-enforcement.js` (manual copy, not installer-owned) | OpenCode enforcement plugin — the general installer has no `--opencode` path; copy `opencode-plugin/index.mjs` under a `.js` name per [opencode-plugin/README.md](opencode-plugin/README.md) |
|
|
85
86
|
| `.claude/harness-everything/` (or the per-platform equivalent) | Harness installer/runtime bookkeeping owned by that integration |
|
|
86
87
|
|
|
87
88
|
For `--global`, the installer uses each host's supported user-level skill location rather than assuming one shared directory works everywhere: shared Agent Skills remain under `~/.agents/skills/` where natively consumed, Continue uses `~/.continue/skills/`, Hermes uses `~/.hermes/skills/`, and Claude uses `~/.claude/skills/`.
|
|
@@ -212,11 +213,14 @@ This repo uses a flat layout (waza/agentskills.io convention). The table below m
|
|
|
212
213
|
| `plugins/harness-everything` | **Distribution** | Local OpenAI/Codex plugin package plus canonical skill copies |
|
|
213
214
|
| `submission/openai` | **Distribution / Review** | Public OpenAI Skills-only listing/test inputs |
|
|
214
215
|
| `evals` | **Routing Evals** | 26 trigger/routing eval suites (waza format) |
|
|
216
|
+
| `eval-framework` | **Quality Gates** | Negative-control fixtures for consistency/collision gates (not a skill) |
|
|
217
|
+
| `contract-integrity` | **Quality Gates** | ADR→spec→ticket→test→implementation trace audit (Phase 1; not a directly routed skill) |
|
|
218
|
+
| `telemetry` | **Quality Gates** | Local JSONL operational evidence layer with report/benchmark scripts (not a skill) |
|
|
215
219
|
| `behavioral-evals` | **Behavioral Evals** | LLM-level discipline cases plus weekly structural validation workflow |
|
|
216
220
|
| `benchmarks` | **Benchmarks** | BENCHMARK_SOP fixtures and recorded A/B results |
|
|
217
221
|
| `docs` | **Documentation** | Philosophy, architecture, routing, reflection, platform capabilities, generated repository contract, audit |
|
|
218
222
|
| `references` | **Documentation** | Shared checklists (security, performance, definition-of-done) |
|
|
219
|
-
| `multi-agent-workspace` | **Skill (Tier 3)** | Scaffold
|
|
223
|
+
| `multi-agent-workspace` | **Skill (Tier 3)** | Scaffold a verified multi-agent workspace and select bounded specialists from an external catalog without vendoring the full roster |
|
|
220
224
|
| `environment-detection` | **Foundation** | Preflight: detect OS, shell, package manager |
|
|
221
225
|
| `eval-harness` | **Skill (Tier 2)** | Evaluate agent outputs against rubrics |
|
|
222
226
|
| `fable-discipline` | **Skill (Tier 3)** | Fable execution guardrails when Fable is selected |
|
|
@@ -260,7 +264,7 @@ For a deep dive into individual modules and the underlying philosophy, explore o
|
|
|
260
264
|
|
|
261
265
|
Fable model selection is documented in [fable-mode/references/model-matrix.md](fable-mode/references/model-matrix.md); the explicit entrypoints are `fable-haiku`, `fable-sonnet`, and `fable-opus`.
|
|
262
266
|
|
|
263
|
-
Maintainers should follow [RELEASING.md](RELEASING.md) for tag-driven npm releases and record observations in [docs/release-evidence.md](docs/release-evidence.md).
|
|
267
|
+
Maintainers should follow [RELEASING.md](RELEASING.md) for tag-driven npm releases and record observations in [docs/release-evidence.md](docs/release-evidence.md). Issue #20 is closed (2026-09-10); its coordination history lives in git.
|
|
264
268
|
|
|
265
269
|
---
|
|
266
270
|
|
|
@@ -294,7 +298,7 @@ Mechanism tests prove individual packaged mechanisms; only real host/session evi
|
|
|
294
298
|
|
|
295
299
|
Harness audits itself on a dated cycle by running its own test suite and VERIFICATION.md recipes — never by reading the code and assuming it works. The full scorecards, methodology, and per-cycle change log live in [docs/audit.md](docs/audit.md). Audit scorecards are historical snapshots; current platform capability claims live in [docs/platform-capabilities.md](docs/platform-capabilities.md).
|
|
296
300
|
|
|
297
|
-
**Latest local audit baseline — 2026-09-
|
|
301
|
+
**Latest local audit baseline — 2026-09-18** (macOS, Node.js 24; see [docs/audit.md](docs/audit.md)): 26/26 on-disk skills, 26/26 routing-eval directories, 34/34 positive routes, 275/275 invariant checks, and all 26 canonical skills within the 500-token limit. Deterministic gates were green except the pre-existing `mechanism-30` failure caused by leftover `.worktrees` fixtures (reproduced on the clean tree). Waza full-matrix spec verify and live model sessions remain on-demand evidence; run waza on an LF-normalized export as documented by CI.
|
|
298
302
|
|
|
299
303
|
Measure on an LF export, not a Windows working tree — CRLF can inflate waza's token counts and trigger false budget failures.
|
|
300
304
|
|
|
@@ -68,6 +68,77 @@ helper.check(
|
|
|
68
68
|
`Got exit=${successResult.code}, state=${JSON.stringify(resetState)}`
|
|
69
69
|
);
|
|
70
70
|
|
|
71
|
+
helper.runHook('rule-of-3-tracker.js', claudeFailurePayload);
|
|
72
|
+
const claudeNoExitSuccess = helper.runHook('rule-of-3-tracker.js', {
|
|
73
|
+
hook_event_name: 'PostToolUse',
|
|
74
|
+
tool_name: 'Bash',
|
|
75
|
+
tool_input: { command: 'echo recovered' },
|
|
76
|
+
tool_response: {
|
|
77
|
+
stdout: 'recovered',
|
|
78
|
+
stderr: 'warning: successful command emitted an error-like diagnostic',
|
|
79
|
+
},
|
|
80
|
+
session_id: helper.SESSION_ID,
|
|
81
|
+
});
|
|
82
|
+
const noExitResetState = helper.readState('rule-of-3-state.json');
|
|
83
|
+
helper.check(
|
|
84
|
+
'2a-claude-success-no-exit. PostToolUse without exitCode resets even with stderr diagnostics',
|
|
85
|
+
claudeNoExitSuccess.code === 0 &&
|
|
86
|
+
noExitResetState.count === 0 &&
|
|
87
|
+
noExitResetState.zoomOutResolved === true,
|
|
88
|
+
`Got exit=${claudeNoExitSuccess.code}, state=${JSON.stringify(noExitResetState)}`
|
|
89
|
+
);
|
|
90
|
+
|
|
91
|
+
helper.writeState('rule-of-3-state.json', {
|
|
92
|
+
count: 0, lastHash: null, zoomOutResolved: false, zoomOutCycles: 0, lastFailureAt: 0,
|
|
93
|
+
});
|
|
94
|
+
helper.runHook('rule-of-3-tracker.js', {
|
|
95
|
+
hook_event_name: 'PostToolUseFailure',
|
|
96
|
+
tool_name: 'Bash',
|
|
97
|
+
tool_input: { command: 'diff -rq hooks cache-hooks' },
|
|
98
|
+
error: 'Exit code 1\nworkflow-mutation-denied.js\nFiles hooks/a.js and cache-hooks/a.js differ',
|
|
99
|
+
session_id: helper.SESSION_ID,
|
|
100
|
+
});
|
|
101
|
+
const deniedFilenameState = helper.readState('rule-of-3-state.json');
|
|
102
|
+
helper.check(
|
|
103
|
+
'2a-category-negative. File name containing denied stays unknown/threshold 3',
|
|
104
|
+
deniedFilenameState.category === 'unknown' && deniedFilenameState.threshold === 3,
|
|
105
|
+
`State=${JSON.stringify(deniedFilenameState)}`
|
|
106
|
+
);
|
|
107
|
+
|
|
108
|
+
helper.writeState('rule-of-3-state.json', {
|
|
109
|
+
count: 0, lastHash: null, zoomOutResolved: false, zoomOutCycles: 0, lastFailureAt: 0,
|
|
110
|
+
});
|
|
111
|
+
helper.runHook('rule-of-3-tracker.js', {
|
|
112
|
+
hook_event_name: 'PostToolUseFailure',
|
|
113
|
+
tool_name: 'Bash',
|
|
114
|
+
tool_input: { command: 'cat protected.txt' },
|
|
115
|
+
error: 'EACCES: permission denied, open protected.txt',
|
|
116
|
+
session_id: helper.SESSION_ID,
|
|
117
|
+
});
|
|
118
|
+
const permissionState = helper.readState('rule-of-3-state.json');
|
|
119
|
+
helper.check(
|
|
120
|
+
'2a-category-permission. Real EACCES is permission/threshold 2',
|
|
121
|
+
permissionState.category === 'permission' && permissionState.threshold === 2,
|
|
122
|
+
`State=${JSON.stringify(permissionState)}`
|
|
123
|
+
);
|
|
124
|
+
|
|
125
|
+
helper.writeState('rule-of-3-state.json', {
|
|
126
|
+
count: 0, lastHash: null, zoomOutResolved: false, zoomOutCycles: 0, lastFailureAt: 0,
|
|
127
|
+
});
|
|
128
|
+
helper.runHook('rule-of-3-tracker.js', {
|
|
129
|
+
hook_event_name: 'PostToolUseFailure',
|
|
130
|
+
tool_name: 'Bash',
|
|
131
|
+
tool_input: { command: 'node slow.js' },
|
|
132
|
+
error: 'TimeoutError: operation timed out',
|
|
133
|
+
session_id: helper.SESSION_ID,
|
|
134
|
+
});
|
|
135
|
+
const timeoutState = helper.readState('rule-of-3-state.json');
|
|
136
|
+
helper.check(
|
|
137
|
+
'2a-category-timeout. Real timeout is timeout/threshold 2',
|
|
138
|
+
timeoutState.category === 'timeout' && timeoutState.threshold === 2,
|
|
139
|
+
`State=${JSON.stringify(timeoutState)}`
|
|
140
|
+
);
|
|
141
|
+
|
|
71
142
|
helper.writeState('rule-of-3-state.json', { count: 3, lastHash: 'mech-test', zoomOutResolved: false });
|
|
72
143
|
const tripResult = helper.runHook('rule-of-3.js', { session_id: helper.SESSION_ID });
|
|
73
144
|
helper.check(
|
|
@@ -35,6 +35,14 @@ function git(args) {
|
|
|
35
35
|
}
|
|
36
36
|
const route = prompt => node('harness-everything/scripts/kernel-router.js', { ...payload, prompt });
|
|
37
37
|
const gate = (tool, input, cwd = linked) => node('hooks/scripts/workflow-gate.js', { ...payload, cwd, tool_name: tool, tool_input: input });
|
|
38
|
+
const persistDirect = (tool, input, cwd = repo, response = {}) => node('hooks/scripts/state-persist.js', {
|
|
39
|
+
...payload,
|
|
40
|
+
cwd,
|
|
41
|
+
hook_event_name: 'PostToolUse',
|
|
42
|
+
tool_name: tool,
|
|
43
|
+
tool_input: input,
|
|
44
|
+
tool_response: response,
|
|
45
|
+
});
|
|
38
46
|
let toolUseSeq = 0;
|
|
39
47
|
function beginShell(command, cwd = repo) {
|
|
40
48
|
const toolUseId = 'toolu_fixture_' + (++toolUseSeq);
|
|
@@ -87,6 +95,20 @@ try {
|
|
|
87
95
|
git(['init']); fs.writeFileSync(path.join(repo, 'README.md'), 'fixture\n'); git(['add', '.']);
|
|
88
96
|
git(['-c', 'user.name=Harness Test', '-c', 'user.email=harness@example.invalid', 'commit', '-m', 'fixture']);
|
|
89
97
|
git(['worktree', 'add', linked, '-b', 'isolated']);
|
|
98
|
+
const taskNotification = '<task-notification><summary>Background command "Run full test suite" failed with exit code 1</summary></task-notification>';
|
|
99
|
+
const unboundNotification = node('harness-everything/scripts/kernel-router.js', { cwd: repo, prompt: taskNotification });
|
|
100
|
+
check(unboundNotification.status === 0 &&
|
|
101
|
+
unboundNotification.stdout.includes('ignored for routing') &&
|
|
102
|
+
!unboundNotification.stdout.includes('WORKFLOW EXECUTION CONTRACT'),
|
|
103
|
+
'host notification without session binding is a routing no-op');
|
|
104
|
+
const noWorkflowNotification = route(taskNotification);
|
|
105
|
+
check(noWorkflowNotification.status === 0 && !fs.existsSync(file),
|
|
106
|
+
'host notification with no active workflow creates no contract');
|
|
107
|
+
check(noWorkflowNotification.stdout.includes('ignored for routing') &&
|
|
108
|
+
!noWorkflowNotification.stdout.includes('iterative-single') &&
|
|
109
|
+
!noWorkflowNotification.stdout.includes('WORKFLOW EXECUTION CONTRACT'),
|
|
110
|
+
'host notification with no active workflow emits no notification-derived routing contract');
|
|
111
|
+
|
|
90
112
|
const routed = route('Refactor the entire repository architecture in dependent stages.');
|
|
91
113
|
check(routed.status === 0, 'prompt hook creates workflow');
|
|
92
114
|
const initial = read(file);
|
|
@@ -149,6 +171,15 @@ try {
|
|
|
149
171
|
check(stop().status === 2, 'verification predating a new mutation is stale');
|
|
150
172
|
write(file, mutationState);
|
|
151
173
|
check(stop().status === 0 && read(file).state === 'satisfied', 'all covered stages and verifier resolve workflow to satisfied');
|
|
174
|
+
const satisfiedWorkflow = read(file);
|
|
175
|
+
const satisfiedNotification = route(taskNotification);
|
|
176
|
+
check(satisfiedNotification.status === 0 &&
|
|
177
|
+
read(file).workflowId === satisfiedWorkflow.workflowId &&
|
|
178
|
+
read(file).state === 'satisfied',
|
|
179
|
+
'satisfied workflow followed by host notification keeps the same satisfied contract');
|
|
180
|
+
check(!satisfiedNotification.stdout.includes('Run full test suite') &&
|
|
181
|
+
satisfiedNotification.stdout.includes('routing unchanged'),
|
|
182
|
+
'satisfied host notification text is not routed as new work');
|
|
152
183
|
check(route('Refactor the entire repository architecture in dependent stages.').status === 0 && read(file).workflowId !== initial.workflowId, 'next completed-task boundary creates a fresh workflow');
|
|
153
184
|
check(stop({ stop_hook_active: true }).status === 0 && read(file).state === 'blocked', 'Stop retry reports blocked instead of faking completion');
|
|
154
185
|
check(gate('Write', { file_path: path.join(linked, 'src.js') }).status === 2, 'blocked state still prohibits mutation');
|
|
@@ -196,9 +227,117 @@ try {
|
|
|
196
227
|
check(gate('Write', { file_path: path.join(repo, 'premature-primary.js') }, repo).status === 2, 'activated Tier 3 plan still blocks primary-tree mutation');
|
|
197
228
|
fs.unlinkSync(file);
|
|
198
229
|
check(route('Fix this checkout bug with a regression test').status === 0 && read(file).strategy === 'iterative-single', 'fresh bounded fix returns to Tier 2 iterative lifecycle');
|
|
230
|
+
let tier2 = read(file);
|
|
231
|
+
const tier2WorkflowId = tier2.workflowId;
|
|
232
|
+
write(file, { ...tier2, state: 'deferred' });
|
|
233
|
+
const deferredNotification = route(taskNotification);
|
|
234
|
+
check(deferredNotification.status === 0 &&
|
|
235
|
+
read(file).workflowId === tier2WorkflowId &&
|
|
236
|
+
read(file).state === 'deferred',
|
|
237
|
+
'deferred workflow followed by host notification keeps the same deferred contract');
|
|
238
|
+
check(!deferredNotification.stdout.includes('Run full test suite') &&
|
|
239
|
+
deferredNotification.stdout.includes('routing unchanged'),
|
|
240
|
+
'deferred host notification text is not routed as new work');
|
|
241
|
+
write(file, tier2);
|
|
242
|
+
|
|
199
243
|
const iterationLimit = read(file).workflowPlan.limits.maxIterations;
|
|
200
244
|
check(Number.isInteger(iterationLimit) && iterationLimit > 0, 'iterative route exposes a numeric iteration budget');
|
|
201
245
|
|
|
246
|
+
// #165: direct Edit/Write/apply_patch accounting follows workspace effects.
|
|
247
|
+
// Bookkeeping outside the repository must not consume Tier-2 iterations or
|
|
248
|
+
// advance workflow/Stop-gate mutation milestones.
|
|
249
|
+
const handoffFile = path.join(sessionDir, 'handoff-state.json');
|
|
250
|
+
const outsideTarget = path.join(temp, 'scratch-note.txt');
|
|
251
|
+
const beforeOutside = read(file);
|
|
252
|
+
const beforeOutsideIterations = beforeOutside.budget?.counters?.iterations || 0;
|
|
253
|
+
const beforeOutsideMutationAt = beforeOutside.lastMutationAt || 0;
|
|
254
|
+
const beforeOutsideEditAt = fs.existsSync(handoffFile) ? (read(handoffFile).lastEditAt || 0) : 0;
|
|
255
|
+
for (let i = 0; i < 3; i++) {
|
|
256
|
+
check(gate('Write', { file_path: outsideTarget }, repo).status === 0,
|
|
257
|
+
'out-of-workspace Write is admitted without Tier-2 mutation accounting');
|
|
258
|
+
fs.writeFileSync(outsideTarget, 'scratch ' + i + '\n');
|
|
259
|
+
const outsideResponse = i === 0
|
|
260
|
+
? { stderr: 'warning: successful bookkeeping write emitted diagnostic text' }
|
|
261
|
+
: {};
|
|
262
|
+
check(persistDirect('Write', { file_path: outsideTarget }, repo, outsideResponse).status === 0,
|
|
263
|
+
'out-of-workspace Write post-tool state persists without mutation milestone');
|
|
264
|
+
}
|
|
265
|
+
const afterOutside = read(file);
|
|
266
|
+
const afterOutsideHandoff = read(handoffFile);
|
|
267
|
+
check((afterOutside.budget?.counters?.iterations || 0) === beforeOutsideIterations &&
|
|
268
|
+
(afterOutside.lastMutationAt || 0) === beforeOutsideMutationAt &&
|
|
269
|
+
(afterOutsideHandoff.lastEditAt || 0) === beforeOutsideEditAt &&
|
|
270
|
+
afterOutsideHandoff.status !== 'failed',
|
|
271
|
+
'out-of-workspace Writes leave iterations/milestones unchanged and PostToolUse stderr is not a failure');
|
|
272
|
+
check(stop().status === 0, 'out-of-workspace-only bookkeeping does not force verification before Stop');
|
|
273
|
+
|
|
274
|
+
check(route('Fix this checkout bug with a regression test').status === 0 && read(file).strategy === 'iterative-single',
|
|
275
|
+
'real follow-up user prompt after notification/accounting checks routes normally');
|
|
276
|
+
let observedDirectIterations = read(file).budget?.counters?.iterations || 0;
|
|
277
|
+
const internalTarget = path.join(repo, 'direct-inside.txt');
|
|
278
|
+
const beforeInternalMutationAt = read(file).lastMutationAt || 0;
|
|
279
|
+
check(gate('Write', { file_path: internalTarget }, repo).status === 0, 'in-workspace Write is admitted');
|
|
280
|
+
fs.writeFileSync(internalTarget, 'inside\n');
|
|
281
|
+
check(persistDirect('Write', { file_path: internalTarget }, repo).status === 0, 'in-workspace Write post-tool state persists');
|
|
282
|
+
check(read(file).budget.counters.iterations === ++observedDirectIterations &&
|
|
283
|
+
read(file).lastMutationAt >= beforeInternalMutationAt &&
|
|
284
|
+
read(handoffFile).lastEditAt > beforeOutsideEditAt,
|
|
285
|
+
'in-workspace Write consumes exactly one iteration and advances mutation milestones');
|
|
286
|
+
|
|
287
|
+
const mixedPatch = [
|
|
288
|
+
'*** Begin Patch',
|
|
289
|
+
`*** Update File: ${outsideTarget}`,
|
|
290
|
+
'@@',
|
|
291
|
+
'-scratch',
|
|
292
|
+
'+scratch updated',
|
|
293
|
+
`*** Update File: ${path.join(repo, 'README.md')}`,
|
|
294
|
+
'@@',
|
|
295
|
+
'-fixture',
|
|
296
|
+
'+fixture updated',
|
|
297
|
+
'*** End Patch',
|
|
298
|
+
].join('\n');
|
|
299
|
+
check(gate('apply_patch', { patch: mixedPatch }, repo).status === 0, 'mixed apply_patch is admitted');
|
|
300
|
+
check(persistDirect('apply_patch', { patch: mixedPatch }, repo).status === 0, 'mixed apply_patch post-tool state persists');
|
|
301
|
+
check(read(file).budget.counters.iterations === ++observedDirectIterations,
|
|
302
|
+
'mixed apply_patch counts exactly once when any target is inside the workspace');
|
|
303
|
+
|
|
304
|
+
const pureOutsidePatch = [
|
|
305
|
+
'*** Begin Patch',
|
|
306
|
+
`*** Update File: ${outsideTarget}`,
|
|
307
|
+
'@@',
|
|
308
|
+
'-scratch updated',
|
|
309
|
+
'+scratch external only',
|
|
310
|
+
'*** End Patch',
|
|
311
|
+
].join('\n');
|
|
312
|
+
const beforePureOutside = read(file);
|
|
313
|
+
const beforePureOutsideEditAt = read(handoffFile).lastEditAt || 0;
|
|
314
|
+
check(gate('apply_patch', { patch: pureOutsidePatch }, repo).status === 0, 'pure external apply_patch is admitted');
|
|
315
|
+
check(persistDirect('apply_patch', { patch: pureOutsidePatch }, repo).status === 0, 'pure external apply_patch post-tool state persists');
|
|
316
|
+
check(read(file).budget.counters.iterations === observedDirectIterations &&
|
|
317
|
+
read(file).lastMutationAt === beforePureOutside.lastMutationAt &&
|
|
318
|
+
(read(handoffFile).lastEditAt || 0) === beforePureOutsideEditAt,
|
|
319
|
+
'pure external apply_patch consumes no iteration and advances no mutation milestone');
|
|
320
|
+
|
|
321
|
+
const linkedTarget = path.join(linked, 'direct-linked.txt');
|
|
322
|
+
check(gate('Write', { file_path: linkedTarget }, linked).status === 0,
|
|
323
|
+
'Write inside a linked worktree is treated as workspace mutation');
|
|
324
|
+
fs.writeFileSync(linkedTarget, 'linked\n');
|
|
325
|
+
check(persistDirect('Write', { file_path: linkedTarget }, linked).status === 0,
|
|
326
|
+
'linked-worktree Write post-tool state persists');
|
|
327
|
+
check(read(file).budget.counters.iterations === ++observedDirectIterations,
|
|
328
|
+
'linked-worktree Write consumes exactly one iteration');
|
|
329
|
+
|
|
330
|
+
// Keep the #165 regression fixture from consuming the budget used by the
|
|
331
|
+
// pre-existing lifecycle/exhaustion assertions below.
|
|
332
|
+
fs.unlinkSync(internalTarget);
|
|
333
|
+
fs.unlinkSync(linkedTarget);
|
|
334
|
+
fs.unlinkSync(file);
|
|
335
|
+
if (fs.existsSync(handoffFile)) fs.unlinkSync(handoffFile);
|
|
336
|
+
check(route('Fix this checkout bug with a regression test').status === 0 &&
|
|
337
|
+
read(file).strategy === 'iterative-single' &&
|
|
338
|
+
(read(file).budget?.counters?.iterations || 0) === 0,
|
|
339
|
+
'#165 regression fixture resets into a fresh Tier-2 lifecycle baseline');
|
|
340
|
+
|
|
202
341
|
// #155/#157 regression lock: read-only shell composition, including quoted
|
|
203
342
|
// and escaped separator characters inside arguments, must never look like an
|
|
204
343
|
// iterative mutation or advance the mutation clock.
|
|
@@ -16,14 +16,57 @@ helper.check(
|
|
|
16
16
|
|
|
17
17
|
helper.runHook('state-persist.js', {
|
|
18
18
|
session_id: helper.SESSION_ID,
|
|
19
|
+
hook_event_name: 'PostToolUse',
|
|
19
20
|
tool_name: 'Bash',
|
|
20
|
-
tool_response: {
|
|
21
|
+
tool_response: {
|
|
22
|
+
stdout: 'ok',
|
|
23
|
+
stderr: 'warning: successful command emitted diagnostic text',
|
|
24
|
+
},
|
|
21
25
|
});
|
|
22
26
|
const resolvedHandoff = helper.readState('handoff-state.json');
|
|
23
27
|
helper.check(
|
|
24
|
-
'2c.
|
|
28
|
+
'2c. Claude-shaped PostToolUse without exitCode clears failed state even with stderr diagnostics',
|
|
25
29
|
resolvedHandoff.status === 'idle' && !!resolvedHandoff.lastResolved,
|
|
26
30
|
`Got ${JSON.stringify(resolvedHandoff)}`
|
|
27
31
|
);
|
|
28
32
|
|
|
33
|
+
helper.runHook('state-persist.js', {
|
|
34
|
+
session_id: helper.SESSION_ID,
|
|
35
|
+
hook_event_name: 'PostToolUse',
|
|
36
|
+
tool_name: 'Bash',
|
|
37
|
+
tool_response: { stdout: '', stderr: 'numeric failure', exitCode: 2 },
|
|
38
|
+
});
|
|
39
|
+
const numericFailure = helper.readState('handoff-state.json');
|
|
40
|
+
helper.check(
|
|
41
|
+
'2c. Numeric non-zero exit still records failure on hosts that report exit codes',
|
|
42
|
+
numericFailure.status === 'failed' && numericFailure.exitCode === 2,
|
|
43
|
+
`Got ${JSON.stringify(numericFailure)}`
|
|
44
|
+
);
|
|
45
|
+
|
|
46
|
+
helper.runHook('state-persist.js', {
|
|
47
|
+
session_id: helper.SESSION_ID,
|
|
48
|
+
hook_event_name: 'PostToolUse',
|
|
49
|
+
tool_name: 'Bash',
|
|
50
|
+
tool_response: { stdout: 'ok', exitCode: 0 },
|
|
51
|
+
});
|
|
52
|
+
const numericResolved = helper.readState('handoff-state.json');
|
|
53
|
+
helper.check(
|
|
54
|
+
'2c. Numeric zero exit still clears failure state',
|
|
55
|
+
numericResolved.status === 'idle' && !!numericResolved.lastResolved,
|
|
56
|
+
`Got ${JSON.stringify(numericResolved)}`
|
|
57
|
+
);
|
|
58
|
+
|
|
59
|
+
helper.runHook('state-persist.js', {
|
|
60
|
+
session_id: helper.SESSION_ID,
|
|
61
|
+
hook_event_name: 'PostToolUseFailure',
|
|
62
|
+
tool_name: 'Bash',
|
|
63
|
+
error: 'host-declared failure without numeric exit code',
|
|
64
|
+
});
|
|
65
|
+
const eventFailure = helper.readState('handoff-state.json');
|
|
66
|
+
helper.check(
|
|
67
|
+
'2c. PostToolUseFailure remains authoritative without numeric exitCode',
|
|
68
|
+
eventFailure.status === 'failed',
|
|
69
|
+
`Got ${JSON.stringify(eventFailure)}`
|
|
70
|
+
);
|
|
71
|
+
|
|
29
72
|
helper.finish();
|
|
@@ -64,12 +64,4 @@ helper.check(
|
|
|
64
64
|
`exit ${negativeControls.status}\n${negativeControls.stdout}\n${negativeControls.stderr}`
|
|
65
65
|
);
|
|
66
66
|
|
|
67
|
-
const rollup = fs.readFileSync(path.join(helper.root, 'docs', 'issue-20-rollup.md'), 'utf8');
|
|
68
|
-
const deferredWork = ['baseline-debugging', 'baseline-performance', 'baseline-security-review', 'baseline-simple-bugfix', 'verify-before-done', 'verification-loop', 'zoom-out', 'security-review'];
|
|
69
|
-
helper.check(
|
|
70
|
-
'2t. issue rollup records unresolved acceptance work accurately',
|
|
71
|
-
/Issue #20 remains open/.test(rollup) && deferredWork.every(term => rollup.includes(term)),
|
|
72
|
-
'rollup must name the open issue and each deferred baseline/skill item'
|
|
73
|
-
);
|
|
74
|
-
|
|
75
67
|
helper.finish();
|
|
@@ -40,13 +40,29 @@ function verifyInstall(text) {
|
|
|
40
40
|
}
|
|
41
41
|
}
|
|
42
42
|
|
|
43
|
+
function isWorktreeRoot(dir) {
|
|
44
|
+
try {
|
|
45
|
+
return fs.existsSync(path.join(dir, '.git'));
|
|
46
|
+
} catch (_) {
|
|
47
|
+
return false;
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
|
|
43
51
|
function walk(directory) {
|
|
44
52
|
return fs.readdirSync(directory, { withFileTypes: true }).flatMap(entry => {
|
|
45
|
-
if (['node_modules', '.git'].includes(entry.name) || entry.isSymbolicLink()) return [];
|
|
53
|
+
if (['node_modules', '.git', '.worktrees'].includes(entry.name) || entry.isSymbolicLink()) return [];
|
|
46
54
|
const file = path.join(directory, entry.name);
|
|
47
55
|
const relative = path.relative(root, file).split(path.sep).join('/');
|
|
48
56
|
if (relative === 'benchmarks/results' || relative === 'CHANGELOG.md') return [];
|
|
49
|
-
|
|
57
|
+
if (relative === '.worktrees' || relative.startsWith('.worktrees/')) return [];
|
|
58
|
+
if (entry.isDirectory()) {
|
|
59
|
+
// Skip linked worktrees / nested checkouts: a `.git` file marks a
|
|
60
|
+
// linked worktree root, a `.git` dir marks a nested repo. Either way
|
|
61
|
+
// the content belongs to another checkout, not this one (#170).
|
|
62
|
+
if (isWorktreeRoot(file)) return [];
|
|
63
|
+
return walk(file);
|
|
64
|
+
}
|
|
65
|
+
return [file];
|
|
50
66
|
});
|
|
51
67
|
}
|
|
52
68
|
|
|
@@ -68,12 +84,28 @@ try {
|
|
|
68
84
|
}
|
|
69
85
|
assert.deepStrictEqual(invalidTargets('opencode-plugin/index.mjs .opencode/plugins/example.js'), []);
|
|
70
86
|
|
|
71
|
-
const
|
|
72
|
-
|
|
87
|
+
const scanViolations = (readText = file => fs.readFileSync(file, 'utf8')) => {
|
|
88
|
+
const violations = [];
|
|
89
|
+
for (const file of walk(root).filter(file => /\.(?:md|js|mjs|json|jsonc|yml|yaml|txt)$/.test(file))) {
|
|
90
|
+
const text = readText(file);
|
|
91
|
+
for (const target of invalidTargets(text)) violations.push(`${path.relative(root, file)}: ${target}`);
|
|
92
|
+
}
|
|
93
|
+
return violations;
|
|
94
|
+
};
|
|
95
|
+
assert.deepStrictEqual(scanViolations(), [], 'Unsupported OpenCode install destinations; only .js/.ts are auto-discovered');
|
|
96
|
+
|
|
97
|
+
const scannedFiles = walk(root);
|
|
98
|
+
assert.ok(scannedFiles.includes(readmeFile), 'Tracked OpenCode README must remain in the repository scan');
|
|
99
|
+
const plantedTarget = '.opencode/plugins/' + 'negative-control.mjs';
|
|
100
|
+
const planted = scanViolations(file => {
|
|
73
101
|
const text = fs.readFileSync(file, 'utf8');
|
|
74
|
-
|
|
75
|
-
}
|
|
76
|
-
assert.
|
|
102
|
+
return file === readmeFile ? text + '\n' + plantedTarget + '\n' : text;
|
|
103
|
+
});
|
|
104
|
+
assert.ok(
|
|
105
|
+
planted.some(item => item === `${path.relative(root, readmeFile)}: ${plantedTarget}`),
|
|
106
|
+
'Scanner negative control must fail when a tracked file contains an unsupported .mjs install target'
|
|
107
|
+
);
|
|
108
|
+
|
|
77
109
|
const readme = fs.readFileSync(readmeFile, 'utf8');
|
|
78
110
|
for (const destination of ['.opencode/plugins/harness-enforcement.js', '~/.config/opencode/plugins/harness-enforcement.js']) {
|
|
79
111
|
assert.ok(readme.includes(`cp opencode-plugin/index.mjs ${destination}`));
|
package/docs/architecture.md
CHANGED
|
@@ -73,6 +73,23 @@ For selected Fable topologies on Claude:
|
|
|
73
73
|
|
|
74
74
|
Other topologies use their own applicable mechanisms: loop budgets, regular verification stop-gates, action gates, and task-specific skills. Mechanism coverage is not assumed identical across hosts.
|
|
75
75
|
|
|
76
|
+
### Scoped runtime advisories and capabilities
|
|
77
|
+
|
|
78
|
+
Two kernel mechanisms adjust plans without changing the mandatory-workflow contract:
|
|
79
|
+
|
|
80
|
+
- **Ensemble policy (`harness-everything/scripts/ensemble-policy.js`):** an advisory plan annotation, not a separate topology. It is selected only when uncertainty is high (or stakes are high) **and** the task asks for comparable outputs or preserves explicit disagreement **and** the selected strategy is already `fable-*`. Creative generation, mechanical bulk work, an explicit user prohibition, unavailable subagents, or a blocked base plan force exclusion. When selected, the plan caps at 3 candidates with `preserve-disagreement` synthesis, an independent verifier, and `parallel-self-consistency` pattern invariants. Covered by deterministic mechanism tests; no live-host effectiveness claim follows from the annotation alone.
|
|
81
|
+
- **Single-use memory capability (`kernel-router.js#issueMemoryCapability`):** when the selected plan permits memory writes, the router issues one capability token bound to `sessionId`/`workflowId` with a stored SHA-256 hash (`memoryAuthorization`, coordinator-only writer role, `usedAt` tracking). Plans with `memory.write === 'none'` receive no capability. This binds a memory write to the workflow that authorized it; it is not a general credential and proves nothing about a live host honoring the binding.
|
|
82
|
+
|
|
83
|
+
### PreToolUse guard trio and denied-mutation probes
|
|
84
|
+
|
|
85
|
+
On the Claude hook path, three narrowly scoped `PreToolUse` guards sit alongside the workflow/action gates:
|
|
86
|
+
|
|
87
|
+
- `boundary-guard.js` (Grep/Glob/Read): hard-blocks whole-file reads over 512KB without `offset`/`limit` and searches inside known noise directories (`node_modules`, `.git`, build output, etc.). Fails open on parse/lookup errors.
|
|
88
|
+
- `depth-guard.js` (Write): blocks overwriting an existing file that was never Read earlier in the session transcript. New files are unaffected. Fails open on parse/lookup errors.
|
|
89
|
+
- `context-compact.js`: estimates working-tree context pressure from `git status`/`git diff --numstat` so later stages can compact or halt before lost-in-the-middle degradation.
|
|
90
|
+
|
|
91
|
+
Shell mutation accounting is per-tool-call: `workflow-isolation.js` fingerprints observed effects per tool-use identity, `state-persist.js` settles probes in a single ordered handler, and `workflow-mutation-denied.js` discards probes for denied shell executions so a denial never counts as an observed mutation. See [runtime transitions and limitations](workflow-runtime.md). These are mechanism-tested behaviors on the packaged hook surface, not proof that a live session loaded or fired them.
|
|
92
|
+
|
|
76
93
|
## Integration Touchpoints
|
|
77
94
|
|
|
78
95
|
Harness aligns to each host's actual lifecycle/tool APIs. Shared skill text defines a contract; only supported host mechanisms can mechanically block or observe particular transitions.
|
package/docs/audit.md
CHANGED
|
@@ -2,17 +2,58 @@
|
|
|
2
2
|
|
|
3
3
|
Maintainer-facing audit record. If you just want to know whether Harness works and how to verify it yourself, start at [VERIFICATION.md](../VERIFICATION.md) instead — this file records *how the scores below were obtained* and what changed between audit cycles.
|
|
4
4
|
|
|
5
|
-
> **Historical snapshot boundary:** the
|
|
5
|
+
> **Historical snapshot boundary:** the scorecards below are dated audits and are intentionally preserved as evidence of what was measured on each date. Platform capabilities have changed since the 2026-09-01 snapshot (including the OpenAI/Codex local plugin work and the OpenCode plugin implementation). Do not treat any dated platform row as current-state documentation; use [platform-capabilities.md](platform-capabilities.md) for the current matrix.
|
|
6
6
|
|
|
7
|
-
**Last self-audited: 2026-09-
|
|
7
|
+
**Last self-audited: 2026-09-18**, by running the deterministic test suite and gates on macOS (Node.js 24.14.1) — not by reading the code and assuming it works. The 2026-09-01 Windows 11 snapshot is preserved below. Live-host and behavioral evidence layers were not re-run in this cycle; retained artifacts are referenced, not re-measured.
|
|
8
8
|
|
|
9
|
-
## The 2026-
|
|
9
|
+
## The 2026-09-18 audit cycle
|
|
10
|
+
|
|
11
|
+
Measured on `docs/sync-skills-and-mechanisms` at `45f2123` (docs-only delta over v0.11.6): 26/26 on-disk canonical skills, 26/26 routing-eval directories (`evals/` also holds `evaluation-reports.md`), and the following gate results:
|
|
12
|
+
|
|
13
|
+
| Gate | Result on 2026-09-18 |
|
|
14
|
+
|---|---|
|
|
15
|
+
| `npm test` (self-regression, 43 passing checks) | Green except one pre-existing failure: `mechanism-30-opencode-plugin-loadability`, reproduced on the clean tree and caused by leftover `.worktrees/docs-skill-catalog` fixtures asserting `.mjs` destinations — not by product code |
|
|
16
|
+
| `npm run test:mutations` | Passed: 8 production mutations rejected by focused tests |
|
|
17
|
+
| `npm run test:consistency` (incl. doc-capability + repo-contract) | Passed |
|
|
18
|
+
| `npm run test:references` | Passed: 66 skill path references resolve across 26 skills |
|
|
19
|
+
| `npm run test:collision` | Passed: no description collisions at 0.75 threshold |
|
|
20
|
+
| `npm run test:routing:skills` | Passed: 34 positive cases across 26 direct skills; 4 nested skills classified |
|
|
21
|
+
| `npm run test:routing:invariants` | Passed: 275 checks, 0 failures (incl. workflow execution enforcement) |
|
|
22
|
+
| `node ci/disclosure-check.js` | Passed: 26 skills, 26 workflow docs |
|
|
23
|
+
| `npm run test:plugin:openai` / `test:plugin:submission` / `test:platform:compatibility` / `test:release` | All passed; release catalog green for v0.11.6 at 26/26 |
|
|
24
|
+
| `waza tokens check` | All 26 canonical skills within the 500-token limit; the single exceedance is the intentional `eval-framework/fixtures/over-budget-skill` negative control (855 tokens, exists to exercise the failure path) |
|
|
25
|
+
|
|
26
|
+
Since the 2026-09-01 snapshot, the product gained the mandatory-workflow contract (#142), runtime budget enforcement (#143), structured action gate (#144), memory governance (#145), self-evolve closed loop (#146), telemetry core (#147), contract-integrity phase 1 (#148), workflow controller recovery and isolation hardening (#150/#152/#154/#156/#158), observed-mutation accounting (#160), the Codex local plugin plus Claude certification boundary (#123), OpenCode `.js` loadability guard and partial live-host evidence (#128/#129), and the paired-benchmark contract (#130) — all covered by deterministic mechanism tests. The docs lag this cycle closed: stale `docs/workflows` rewrites, the README index gaps, and the previously undocumented ensemble/memory/guard/probe mechanisms are synced in the same change as this audit entry.
|
|
27
|
+
|
|
28
|
+
## Five Core Verification Criteria — 2026-09-18 re-measurement
|
|
29
|
+
|
|
30
|
+
| Criterion | Score | Verified basis |
|
|
31
|
+
|---|---|---|
|
|
32
|
+
| **Skill Description Completeness** | 10/10 | 26/26 skill directories carry complete frontmatter and the required trigger sections, verified by the consistency gate in this cycle. Internal hooks remain intentionally non-routable. |
|
|
33
|
+
| **Routing Accuracy** | 9/10 | 34/34 positive prompts across all 26 skills plus 275/275 invariant-routing checks in this cycle. Waza generalization (ambiguous/metaphorical prompts) and live model sessions remain unmeasured. |
|
|
34
|
+
| **Test Coverage of All Skills** | 9.5/10 | Self-regression now spans syntax, CLI, routing matrix, positive route coverage, invariant routing, references, behavioral-case structure, Fable contract, mutations, and 40+ mechanism suites. Waza full-matrix verify and live sessions remain separate on-demand evidence; `mechanism-30` is red only where the stale `.worktrees` leftover exists. |
|
|
35
|
+
| **Configuration Balance** | 9/10 | Hard `exit(2)` blocking hooks on Claude Code, mechanism-tested local Codex plugin package, and partial OpenCode live-host evidence (project `.js` scope, 1.18.31 macOS) — all centralized in `docs/platform-capabilities.md`. Live parity beyond the retained artifacts is still unclaimed. |
|
|
36
|
+
| **Workflow Conformance** | 9/10 | 26/26 skills have matching workflow documents (two stale ones rewritten in this cycle), the selected-workflow contract with escape/budget/stop-gate semantics is mechanism-tested, and disclosure passes. Runtime tool-call sequence conformance remains host-dependent and is not claimed. |
|
|
37
|
+
|
|
38
|
+
## Overall Scorecard — 2026-09-18
|
|
39
|
+
|
|
40
|
+
| Category | Score | Notes |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| **Architecture** | 9/10 | Per-session state isolation, worktree-gated Tier 3 mutation, per-tool mutation probes settled in one ordered handler, and single-use memory capabilities — all mechanism-tested in this cycle. |
|
|
43
|
+
| **Test Coverage** | 9.5/10 | 43 passing self-regression checks, 8/8 mutations rejected, 66/66 references resolve. Live model evaluation remains on-demand and is not represented by a local simulation. |
|
|
44
|
+
| **README Completeness** | 9.5/10 | Catalog counts, layer labels, route coverage, and the 26-skill tree match after this cycle's index fix (`contract-integrity`/`telemetry`/`eval-framework` rows added). |
|
|
45
|
+
| **Maintainability** | 9/10 | ~600 commits across 20+ PRs since 09-01 landed without breaking the consistency/release/disclosure gates; new non-skill directories are documented as non-skills rather than left ambiguous. |
|
|
46
|
+
| **Skills Design** | 9.5/10 | All 26 skills keep the Skill Contract format, `USE FOR`/`DO NOT USE FOR` sections, and the 500-token limit (verified by waza in this cycle); internal hooks remain intentionally non-routable. |
|
|
47
|
+
| **Agent Compatibility** | 9/10 | Claude hard-mechanism support measured, Codex local plugin mechanism-tested, OpenCode partial live evidence retained — see the current platform matrix, not this row, for status. |
|
|
48
|
+
| **Beginner Friendliness** | 7.5/10 | Unchanged since 09-01: Tier Routing / Rule of 3 / session-scoped state remain load-bearing concepts. The two rewritten workflow docs reduce stale-doc confusion but add no onboarding track. Not addressed in this audit round. |
|
|
49
|
+
|
|
50
|
+
## The 2026-07-23 mis-measurement incident (preserved)
|
|
10
51
|
|
|
11
52
|
Seven independent AI-model audits (Gemini 3.1 Pro, Gemini 3.5 Flash, GPT/Copilot on Mistral Medium/Small, and others) ran the same week and disagreed sharply on whether core hooks even functioned. Three of them tested `rule-of-3.js`, `boundary-guard.js`, and `state-persist.js` by piping JSON through `echo '...' | node script.js` in Windows Git Bash, which mangles stdin/TTY state and produced false "broken" verdicts. Re-running the identical payloads through Node's own `child_process` stdin API (the technique VERIFICATION.md's recipes now use) showed every one of those mechanisms working exactly as documented — the scores below reflect that corrected, verified state, not the average of the seven reports.
|
|
12
53
|
|
|
13
54
|
This incident is why VERIFICATION.md's recipes were rewritten to be Windows-safe (`node -e` stdin instead of a shell `echo` pipe), and why the §2 mechanism checks were automated as `ci/mechanism-test.js` (`npm run test:mechanism`) — copy-paste recipes that can be mis-executed are not a reliable verification substrate.
|
|
14
55
|
|
|
15
|
-
## Five Core Verification Criteria (per VERIFICATION.md §5a)
|
|
56
|
+
## Five Core Verification Criteria — 2026-09-01 snapshot (per VERIFICATION.md §5a)
|
|
16
57
|
|
|
17
58
|
| Criterion | Score | Verified basis |
|
|
18
59
|
|---|---|---|
|
|
@@ -22,7 +63,7 @@ This incident is why VERIFICATION.md's recipes were rewritten to be Windows-safe
|
|
|
22
63
|
| **Configuration Balance** | 8.5/10 | **Historical 2026-09-01 result:** hard `exit(2)` blocking hooks were measured on Claude Code and the other then-installed platform paths were evaluated as advisory. This row predates the later OpenAI/Codex local plugin and OpenCode plugin work; see `docs/platform-capabilities.md` for current status. |
|
|
23
64
|
| **Workflow Conformance** | 8.5/10 | Every skill in the snapshot had a matching workflow document, and `ci/skill-routing-check.js` executed every positive route case against the real router. Runtime tool-call sequence conformance remained host-dependent and was not claimed. |
|
|
24
65
|
|
|
25
|
-
## Overall Scorecard
|
|
66
|
+
## Overall Scorecard — 2026-09-01 snapshot
|
|
26
67
|
|
|
27
68
|
| Category | Score | Notes |
|
|
28
69
|
|---|---|---|
|
|
@@ -36,6 +77,12 @@ This incident is why VERIFICATION.md's recipes were rewritten to be Windows-safe
|
|
|
36
77
|
|
|
37
78
|
## Change log by audit cycle
|
|
38
79
|
|
|
80
|
+
### 2026-09-18
|
|
81
|
+
- Re-measured the deterministic suite and gates on macOS (Node.js 24.14.1): 43 passing self-regression checks, 8/8 mutations rejected, 66/66 references resolve, 34/34 positive routes, 275/275 invariant checks, disclosure 26/26, release catalog green for v0.11.6. Sole red is the pre-existing `mechanism-30` failure from leftover `.worktrees/docs-skill-catalog` fixtures, reproduced on the clean tree.
|
|
82
|
+
- Closed the docs lag accumulated since 09-01 in the same change: rewrote the stale `docs/workflows/harness-everything.md` and `install-cognitive-os.md`, added the missing README index rows (`contract-integrity`/`telemetry`/`eval-framework`), and documented the ensemble policy, single-use memory capability, guard trio, and mutation-probe lifecycle in `docs/architecture.md` / `docs/workflow-runtime.md`.
|
|
83
|
+
- Product changes since 09-01 now covered by mechanism tests: mandatory-workflow contract, budget enforcement, structured action gate, memory governance, self-evolve closed loop, telemetry core, contract-integrity phase 1, controller recovery/isolation hardening, observed-mutation accounting, Codex local plugin, OpenCode partial live evidence, paired-benchmark contract. Live-host and behavioral layers were not re-run; retained artifacts stand.
|
|
84
|
+
- Beginner Friendliness stays 7.5/10: still no onboarding track. Candidate for the next cycle.
|
|
85
|
+
|
|
39
86
|
### 2026-09-01
|
|
40
87
|
- Fable v3 integration exposed explicit Haiku, Sonnet, and Opus entrypoints, with deterministic model selection and visible inline fallback or blocked escalation in `fable-mode/model-matrix.json` and `fable-mode/scripts/model-selector.js`.
|
|
41
88
|
- The workflow used native host TODO tracking or Markdown checklists. The older shared TODO CLI/state-machine references below are retained only as historical audit evidence; no current runtime, test, or active workflow depends on them.
|
package/docs/release-evidence.md
CHANGED
|
@@ -9,6 +9,9 @@ The following evidence was collected on 2026-09-07 before the issue #20 integrat
|
|
|
9
9
|
| Version | Registry observation | GitHub release | Workflow run | Gate and publish result |
|
|
10
10
|
| --- | --- | --- | --- | --- |
|
|
11
11
|
| 0.3.6 | `npm view harness-everything version dist-tags --json`: version and latest are `0.3.6` | `v0.3.6`, published 2026-09-01T03:50:44Z | [33466676632](https://github.com/dyphn1/Harness-everything/actions/runs/33466676632), commit `5aa74119d08ff241a7d6fe8f163a6a0d40c3c932` | `gate=success`, `publish=success` |
|
|
12
|
+
| 0.11.6 | `npm view harness-everything version dist-tags --json` (collected 2026-09-18): version and latest are `0.11.6` | `v0.11.6`, published 2026-09-18T12:51:58Z | [35346673284](https://github.com/dyphn1/Harness-everything/actions/runs/35346673284), commit `23d87db8230dcb524651dc67e932f54aa6f5c737` | `gate=success` (12/12 jobs incl. skill-quality, runtime-floor, self-regression x3 OS, consistency x3 OS, installer-e2e x3 OS, release-semantics), `publish=success` (job `release`) |
|
|
13
|
+
|
|
14
|
+
Intermediate versions between 0.3.6 and 0.11.6 were published via the same tag-driven workflow but were not individually ledgered here; the rows above are the two deliberately collected observations. A registry/GitHub listing alone is still not a completed row — each field must be a command output or inspectable URL.
|
|
12
15
|
|
|
13
16
|
The source capture for this row is the local review artifact `.git/issue-review-20260907/release-evidence.md`. That review artifact is not expected to be present in every checkout; it is historical evidence, not a release artifact shipped in the npm package.
|
|
14
17
|
|
package/docs/workflow-runtime.md
CHANGED
|
@@ -63,6 +63,19 @@ have effects a hook cannot infer. OS/host sandboxing is needed to contain them.
|
|
|
63
63
|
Agent-writable state is also not tamper-proof. These mechanisms prevent tested
|
|
64
64
|
bypasses; they do not establish durable containment against a malicious agent.
|
|
65
65
|
|
|
66
|
+
## Mutation probes
|
|
67
|
+
|
|
68
|
+
Shell-effect accounting is keyed per tool call, not per command text. The gate
|
|
69
|
+
registers a pending probe per shell tool-use identity (`workflow-isolation.js`),
|
|
70
|
+
`state-persist.js` settles probes in a single ordered handler, and
|
|
71
|
+
`workflow-mutation-denied.js` discards the probe when that shell execution is
|
|
72
|
+
denied — a denial is never recorded as an observed mutation. Staging-insensitive
|
|
73
|
+
fingerprinting keeps the comparison on visible content, so metadata-only Git
|
|
74
|
+
operations do not register as workspace effects. These lifecycle semantics are
|
|
75
|
+
covered by deterministic workflow mechanism tests (`ci/mechanism-2ah-*`); live
|
|
76
|
+
host loading and behavioral improvement still require separately retained host
|
|
77
|
+
traces and paired evaluations.
|
|
78
|
+
|
|
66
79
|
## Escape, replan, and completion
|
|
67
80
|
|
|
68
81
|
Escape names one stage in the correlated run and records the uncovered scope,
|