@agentskit/harness 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,44 +1,38 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.3.0] - 2026-09-10
4
+
5
+ - Added portable issue/worktree claims and idempotent dispatch ledger.
6
+ - Added failure classification, bounded retry/backoff, and abortable watchdog.
7
+ - Added file-scoped preflight planning and shell-composition rejection.
8
+ - Added block manifests, status snapshots, retro learning promotion, model
9
+ policies, and provider-neutral Orca/tracking adapters.
10
+ - Added configurable machine pressure thresholds and adaptive workflow limits.
11
+
12
+ ## [Unreleased]
13
+
14
+ - Add bounded agent eval, safe context/read-only LLM cache, deterministic workflow fan-out/fan-in, and validated optimization observation contracts for token, memory, cache, and parallelism measurements.
15
+
3
16
  All notable changes to `@agentskit/harness` are documented here.
4
17
 
5
18
  The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the package follows Semantic Versioning.
6
19
 
7
- ## [0.1.0] - 2026-08-31
20
+ ## [0.2.0] - 2026-09-09
8
21
 
9
22
  ### Added
10
23
 
11
- - Replicated baseline collection with independent disposable samples, median
12
- duration recording, and atomic complete-manifest replacement.
13
- - Baseline samples use the same end-to-end measurement boundary as the harness
14
- check while retaining provider-only duration for diagnosis.
15
- - Incomplete baselines no longer qualify for duration or resource comparisons;
16
- escape reduction remains visible as a separate outcome metric.
17
- - Baseline reports expose artifact acceptance separately from protocol
18
- completion, preserving useful outcome data without weakening fail-closed
19
- comparison gates.
20
- - Benchmark execution supports collision-safe task-specific replacement
21
- samples, preserving blocked and timed-out evidence.
22
- - Benchmark comparisons expose artifact acceptance rates and protocol
23
- completion rates separately from performance comparability.
24
- - Protocol and artifact acceptance comparisons expose absolute deltas when
25
- relative improvement is undefined because the baseline was zero.
26
- - Benchmark tasks declare typed surface coverage; the AgentsKit OS corpus
27
- includes executable CLI and documentation tasks with real acceptance
28
- validation.
29
- - Fail-closed baseline sample policy and median baseline duration support for fair replicated comparisons.
30
- - Independent benchmark samples with median-duration comparisons and retry metrics that do not confuse replication with retries.
31
- - Configurable benchmark quality gates, confidence labels, and explicit retry metrics.
32
- - Portable `ak-verify` fixture gate for real coding-agent benchmark runs; provider delivery remains blocked until its result is complete and human approval is recorded.
33
- - Harness-equivalent benchmark preparation and collection for paired real-provider runs, with explicit human approval and honest non-comparability.
34
- - Source snapshots now ignore untracked directories while retaining hashes for untracked files.
35
- - Reproducible disposable-fixture runner and validator for the first real-provider AgentsKit OS baseline; improvement remains unavailable until equivalent harness runs exist.
36
- - Provenance-bearing benchmark tasks and a portable validator for AgentsKit OS coding benchmark reports; missing controlled observations remain explicitly non-comparable.
37
- - Playbook CI dogfooding for the package test suite, built CLI, and clean packed consumer.
38
- - Playbook CI now preserves the official harness run as an auditable artifact for human review.
39
- - Signed evidence flow now verifies a copied bundle from an isolated directory.
40
- - Benchmark reports now separate effective non-superseded delivery metrics from attempt history.
41
- - CI evidence now records automated preparation separately from human contract approval.
24
+ - Pilot-cohort validation that freezes a policy/baseline pair and rejects non-normal, partial, or silently substituted ten-issue cohorts.
25
+ - Bounded five-step improvement-cycle assessment with explicit adjustments, repeat decisions, and a deterministic quality matrix.
26
+
27
+ - Deterministic G2–G5 assessment helpers and CLI commands for independent preflight review, structured idempotent PR handoff, current integration evidence, safe production exposure, and acceptance.
28
+
29
+ - Current-source evidence now requires a committed Git `HEAD`; directories outside Git fail closed instead of receiving a synthetic revision.
30
+
31
+ - Discovery gate API and CLI that emit `ready` or an auditable human decision packet from structured ambiguities, approved assumptions and source/contract/context bindings.
32
+ - Deterministic WIP admission API and CLI that count blocked and awaiting-human deliveries, reserve resumed work, and reject duplicate ledger entries.
33
+ - Controlled runtime-selection API and CLI that reject incomparable Orca/Emdash samples and exclude failed hard gates.
34
+ - Configurable `runtime.kind` contract field plus a factory for bounded process or Docker-sandbox execution.
35
+ - Real Git snapshot coverage for committed, dirty, untracked, and task-state-excluded evidence.
42
36
  - Strict TypeScript modular core with generated declarations and source maps.
43
37
  - Contract-frozen lifecycle, structured evidence, stale detection, human approval, retry, and cleanup.
44
38
  - Explicit human cancellation and superseded retry history.
package/CONTRIBUTING.md CHANGED
@@ -1,25 +1,73 @@
1
1
  # Contributing to `@agentskit/harness`
2
2
 
3
- ## Before opening a change
3
+ The Harness is an enforcement engine, not a suggestion library. Changes must
4
+ preserve fail-closed behavior and produce criterion-level evidence.
4
5
 
5
- 1. State the behavior and failure mode the change addresses.
6
- 2. Keep the public API in `src/index.ts` intentionally small.
7
- 3. Add or update a typed test for every behavior change.
8
- 4. Update the README, changelog, and contract when the user-facing protocol changes.
6
+ ## Repository layout
9
7
 
10
- ## Local checks
8
+ - `src/`: capability-oriented TypeScript modules and the public `index.ts`.
9
+ - `src/adapters/`: optional provider integrations.
10
+ - `test/`: deterministic unit and contract tests.
11
+ - `scripts/`: real CLI and packaging checks.
12
+ - `docs/`: ADRs and protocol decisions.
13
+ - `.github/`: CI and release automation.
11
14
 
12
- ```bash
13
- pnpm --filter @agentskit/harness typecheck
14
- pnpm --filter @agentskit/harness test
15
- pnpm --filter @agentskit/harness build
15
+ See [docs/ORGANIZATION.md](./docs/ORGANIZATION.md) for the complete boundary
16
+ and naming rules.
17
+
18
+ ## Change workflow
19
+
20
+ 1. Start from an issue or explicit task contract.
21
+ 2. Identify the outcome and executable validation it requires.
22
+ 3. Keep the public API change in `src/index.ts` intentional.
23
+ 4. Add or update criterion-level tests for behavior changes.
24
+ 5. Update README, changelog, ADRs, or the verification contract when the
25
+ protocol changes.
26
+ 6. Run the complete local gate:
27
+
28
+ ```bash
29
+ pnpm typecheck
30
+ pnpm test
31
+ pnpm build
32
+ pnpm pack --pack-destination /tmp/agentskit-harness-pack
33
+ ak-verify run --config .codex/verification.json --json
34
+ ```
35
+
36
+ Do not report completion while a required gate is unavailable, blocked, stale,
37
+ or awaiting approval.
38
+
39
+ ## Angular Conventional Commits
40
+
41
+ Commit messages follow the Angular Conventional Commits format:
42
+
43
+ ```text
44
+ <type>(<scope>): <imperative description>
16
45
  ```
17
46
 
18
- Do not claim completion when a required check is unavailable. Preserve criterion-level evidence and report blocked work honestly.
47
+ Allowed types are `feat`, `fix`, `docs`, `refactor`, `test`, `build`, `ci`,
48
+ `perf`, `chore`, and `revert`. Use a breaking-change footer when needed:
49
+
50
+ ```text
51
+ feat(cli)!: reject incomplete evidence
52
+
53
+ BREAKING CHANGE: `run` now exits non-zero when evidence is missing.
54
+ ```
55
+
56
+ Keep commits focused. Never commit secrets, generated `dist/`, verification
57
+ state, or residue from another task.
19
58
 
20
59
  ## Pull requests
21
60
 
22
- Use a focused branch and commit. Describe the contract change, affected states, evidence produced, and exact commands executed. Breaking public API or CLI changes require a changelog entry and a maintainer review.
61
+ PRs must state the contract change, affected states, evidence produced, exact
62
+ commands executed, and any unresolved blockers. A maintainer review is required
63
+ for public API, CLI, state-machine, security, or release changes.
64
+
65
+ ## Releases
66
+
67
+ Version changes merge to `main` through a PR. The release workflow runs tests,
68
+ build, and pack, then publishes through npm Trusted Publishing (GitHub OIDC);
69
+ no `NPM_TOKEN` is used. Configure the npm trusted publisher for
70
+ `AgentsKit-io/harness` once before the first release.
23
71
 
24
72
  ## License
25
73
 
package/MANIFESTO.md ADDED
@@ -0,0 +1,23 @@
1
+ # Harness Manifesto
2
+
3
+ ## The boundary
4
+
5
+ The Playbook explains good practice. The Harness makes the SDLC observable and
6
+ enforceable. Advice may be ignored; a required gate may not.
7
+
8
+ ## The guarantees
9
+
10
+ 1. A task starts from an explicit, frozen contract.
11
+ 2. Ambiguities are surfaced with options and a recommendation; the Harness
12
+ never invents product decisions.
13
+ 3. Every required check produces structured evidence bound to source, config,
14
+ and contract hashes.
15
+ 4. A failed or missing gate blocks progression and remains auditable.
16
+ 5. Recovery resumes from the journal without silently replaying uncertain work.
17
+ 6. Integrations are replaceable adapters, not hidden kernel dependencies.
18
+ 7. Measurements report what was observed, including unavailable data.
19
+
20
+ ## Operating principle
21
+
22
+ Automate everything deterministic. Escalate only decisions that require human
23
+ judgment. Keep the smallest mechanism that proves the claim.
package/README.md CHANGED
@@ -1,3 +1,9 @@
1
+ ---
2
+ docbridge:
3
+ covers:
4
+ - package:@agentskit/harness
5
+ ---
6
+
1
7
  # @agentskit/harness
2
8
 
3
9
  Portable, evidence-backed development protocol for coding agents. The harness freezes a human-approved task contract, executes every configured check, binds evidence to the current source revision, detects stale results, and refuses completion without human approval.
@@ -24,79 +30,7 @@ ak-harness benchmark --manifest benchmarks/harness-phase-0.json --json
24
30
 
25
31
  `plan` rejects unresolved ambiguities and unauthorized dirty worktrees. After `start`, the contract is frozen. Any source, configuration, or contract change invalidates evidence and moves the run to `STALE`. A human can cancel an active run; retrying a blocked, stale, or cancelled run marks the previous run `SUPERSEDED`.
26
32
 
27
- Automated CI may prepare a run with `ak-harness plan prepared --by ci`. This records a CI preparation, never a human approval; the run remains unable to become `COMPLETE` until a human approves the verified result.
28
-
29
- ## External coding benchmark bridge
30
-
31
- The harness validates a provenance-bearing task manifest and the JSON report
32
- emitted by an external coding benchmark, including AgentsKit OS:
33
-
34
- ```bash
35
- AGENTSKIT_OS_ROOT=/path/to/agentskit-os \
36
- AGENTSKIT_OS_BENCHMARK_REPORT=/path/to/report.json \
37
- node scripts/verify-harness-agentskit-os-benchmark.mjs \
38
- --manifest benchmarks/agentskit-os-phase-28.json \
39
- --target "$AGENTSKIT_OS_ROOT" \
40
- --report "$AGENTSKIT_OS_BENCHMARK_REPORT"
41
- ```
42
-
43
- The bridge checks the pinned source revision, task definition, prompt digests,
44
- scope, and provider report shape. Provider status and heuristic scores remain
45
- observations: they do not grant human acceptance. With no controlled baseline,
46
- improvement is reported as unavailable rather than inferred.
47
-
48
- The phase-45 corpus adds explicit task surfaces and keeps the task definition
49
- bound to the real AgentsKit OS fixture. It covers logic, CLI, and documentation
50
- surfaces; the added CLI task executes compact, pretty, and invalid-input paths,
51
- while the documentation task checks the API contract and its reproducible test
52
- command:
53
-
54
- ```bash
55
- node scripts/verify-harness-agentskit-os-benchmark.mjs \
56
- --manifest benchmarks/agentskit-os-phase-45.json \
57
- --target /path/to/agentskit-os \
58
- --require-surface-coverage logic,cli,docs
59
- ```
60
-
61
- Corpus coverage is a readiness gate, not a performance claim. Live provider
62
- samples and human approval remain required before the expanded tasks become
63
- comparable benchmark evidence.
64
-
65
- Phase 29 adds a reproducible real-provider baseline runner for the AgentsKit OS
66
- seed corpus. It uses disposable fixture copies, the existing OS benchmark
67
- runner, and task-specific validation:
68
-
69
- ```bash
70
- node scripts/run-agentskit-os-baseline.mjs \
71
- --target /path/to/agentskit-os \
72
- --provider codex \
73
- --output benchmarks/agentskit-os-phase-29-baseline
74
- ```
75
-
76
- The observations are not an improvement claim until equivalent harness runs
77
- exist.
78
-
79
- To collect equivalent baseline samples, run the same corpus independently and
80
- record the aggregate only after every task has been sampled:
81
-
82
- ```bash
83
- node scripts/run-agentskit-os-baseline.mjs \
84
- --target /path/to/agentskit-os \
85
- --provider codex \
86
- --repeats 3 \
87
- --output benchmarks/agentskit-os-phase-36-baseline \
88
- --record-manifest benchmarks/agentskit-os-phase-28.json
89
- ```
90
-
91
- Each repeat uses a fresh disposable fixture. The report stores the end-to-end
92
- duration of fixture setup, provider execution, validation, and cleanup in
93
- `durationSamplesMs`; the provider's own duration remains nested in the raw
94
- report. The manifest is replaced atomically through
95
- the typed observation recorder. `--record-manifest` requires the complete task
96
- set; omit it to inspect an uncommitted collection. Replicas are not retries,
97
- and a failed sample keeps the aggregate failed. Escape reduction may remain
98
- visible as a separate outcome, but duration, attempt, and review improvements
99
- remain unavailable until the baseline proves complete delivery.
33
+ Current-source evidence requires a committed Git `HEAD`. A directory outside Git, or a repository without its first commit, is rejected as `GIT_REQUIRED`; it is not a supported pilot mode because it cannot prove revision currentness.
100
34
 
101
35
  ## Contract
102
36
 
@@ -151,6 +85,13 @@ the current state, so a post-approval edit cannot appear as `COMPLETE`.
151
85
  Concurrent event writers are serialized by an atomic per-run lock and fail
152
86
  closed if the log is busy.
153
87
 
88
+ Each harness event may also carry an optional `correlation` envelope. Its
89
+ `operationId` is the stable identity used when a lifecycle crosses into
90
+ AgentsKit, Chat, Doc Bridge, or Code Review; the optional `runId`, `sessionId`,
91
+ `turnId`, `actionId`, and `traceId` remain local identities. The envelope is
92
+ bounded metadata only and never contains prompts, arguments, results, or
93
+ secrets.
94
+
154
95
  Export a reconciled `COMPLETE` run for external review with an Ed25519 key:
155
96
 
156
97
  ```bash
@@ -162,9 +103,6 @@ The bundle includes the run projection, event log, and referenced check outputs,
162
103
  each with a SHA-256 digest. A trust store can mark keys `active` or `revoked` to
163
104
  support controlled key rotation. Treat exported outputs as potentially sensitive.
164
105
 
165
- The signed bundle is self-contained: it can be copied to an isolated directory
166
- and verified there with only the bundle and the trusted public key.
167
-
168
106
  Profiles are optional declarative overlays in `.codex/verification.json`. They
169
107
  inherit in order, override existing checks by ID, and are resolved before the
170
108
  contract is frozen:
@@ -181,6 +119,31 @@ contract is frozen:
181
119
  }
182
120
  ```
183
121
 
122
+ `runtime.kind` chooses the executor used by an integration: `process` is a bounded shell-free local child process; `docker` adds the Docker sandbox. The choice is frozen in the resolved contract and therefore changes its hash. Docker remains fail-closed when its daemon or image is unavailable.
123
+
124
+ `autonomy: "yolo"` removes the generic final review only after every applicable check passes, tracking is disabled, and the frozen contract has no ambiguity. It never auto-approves a material decision, external tracking, or a tool rule that requires approval.
125
+
126
+ Use named profiles to make the operational choice explicit:
127
+
128
+ ```json
129
+ {
130
+ "profile": "process",
131
+ "runtime": { "kind": "process" },
132
+ "profiles": {
133
+ "process": { "runtime": { "kind": "process" } },
134
+ "docker": { "runtime": { "kind": "docker" } }
135
+ }
136
+ }
137
+ ```
138
+
139
+ ```ts
140
+ const runtime = createConfiguredToolRuntime({
141
+ runtime: loaded.config.runtime,
142
+ process: { tools: processTools },
143
+ docker: { tools: dockerTools },
144
+ })
145
+ ```
146
+
184
147
  Doc Bridge and Playbook integrations can implement `ContextProvider` and
185
148
  register it through `CONTEXT_PROVIDER_SLOT`; the kernel records neither their
186
149
  credentials nor their transport and does not depend on either package. The
@@ -215,6 +178,148 @@ stable `contextHash`; resolution time is metadata and does not change the
215
178
  reproducibility hash. Context is resolved before planning and is frozen with
216
179
  the run, so later index changes cannot silently change its evidence.
217
180
 
181
+ ## Delivery gates
182
+
183
+ The optional delivery helpers evaluate gates without embedding a GitHub, Linear,
184
+ or deployment provider. Adapters perform external effects only after the
185
+ deterministic decision is recorded:
186
+
187
+ ```ts
188
+ const g2 = assessPreflight({
189
+ criteria,
190
+ implementerId: 'implementer',
191
+ reviewerId: 'independent-reviewer',
192
+ reviewKind: 'adversarial',
193
+ reviewApproved: true,
194
+ })
195
+ ```
196
+
197
+ G2 ignores later-gate pending criteria but blocks failed/pending G2 evidence,
198
+ self-review, and a third repair. `composePullRequest` creates a structured PR
199
+ body only from an approved, current G2 result; it reuses a confirmed matching
200
+ remote PR and preserves uncertain state. G3 binds CI to the candidate revision.
201
+ G4 requires an approved environment profile, identified artifact, isolation or
202
+ version-bound acceptance, technical evidence, and a 15-minute low-risk window.
203
+ G5 remains `awaiting-acceptance` until the applicable business/UX decision is
204
+ recorded. `assessWorktreeCleanup` permits cleanup only after remote branch SHA,
205
+ PR, and G3 all match.
206
+
207
+ ## Pilot cohort
208
+
209
+ `ak-harness pilot cohort.json` validates the frozen cohort before work starts.
210
+ It requires a policy hash, a baseline reference, exactly ten included issues,
211
+ and the `normal` classification for each. Excluded or aborted issues need a
212
+ reason and cannot be silently replaced in the same manifest.
213
+
214
+ ## Improvement cycle
215
+
216
+ The five pilot steps can be evaluated as a bounded cycle: adversarial review,
217
+ G2 preflight, baseline recording, pilot execution, and harness/no-harness
218
+ comparison. Each iteration must contain those steps in order. A failed,
219
+ blocked, or pending step requires a reason; repeating requires an explicit
220
+ adjustment. The assessment returns `complete`, `repeat` (with the next
221
+ iteration), or `blocked` when the adjustment is missing or the iteration
222
+ budget is exhausted, plus a criterion-level matrix:
223
+
224
+ ```bash
225
+ ak-harness cycle assess cycle.json --json
226
+ ```
227
+
228
+ This is a deterministic decision helper: Orca, Emdash, GitHub, Linear, and
229
+ other adapters remain responsible for executing external actions and supplying
230
+ their structured results.
231
+
232
+ ## Agent optimization and evaluation
233
+
234
+ The Harness also validates the optimization layer without owning a provider.
235
+ The memory boundary accepts only approved records with explicit scope, source
236
+ revision, and content hash; an adapter can back it with AgentsKit memory.
237
+ `runAgentEval` runs a bounded suite and returns criterion-level accuracy;
238
+ `createLlmCache` provides deterministic keys and hit/miss/invalidation evidence
239
+ for context and read-only calls; and `runWorkflow` executes independent nodes
240
+ in sorted, bounded fan-out/fan-in batches while serializing nodes that share a
241
+ `mutationKey`. `OptimizationObservation` carries
242
+ optional token, memory, cache, and parallelism measurements and refuses
243
+ incomparable provider/model/configuration bindings.
244
+
245
+ These are seams, not replacements for AgentsKit packages. An integration may
246
+ adapt `@agentskit/memory` and `@agentskit/eval` into them while keeping the
247
+ Harness provider-neutral. Missing measurements remain missing; they are never
248
+ reported as zero.
249
+
250
+ ## Discovery gate
251
+
252
+ Discovery is a small deterministic gate before implementation. An adapter or
253
+ agent supplies a structured list of ambiguities; the Harness does not decide
254
+ product questions. Material ambiguities produce one decision packet with
255
+ options and a recommendation. A non-material ambiguity can proceed only when
256
+ an approved policy assumption covers it, and the result records that policy in
257
+ the decision log.
258
+
259
+ ```bash
260
+ ak-harness discovery assess discovery.json --json
261
+ ```
262
+
263
+ The result is `ready` or `awaiting-decision`, and carries the source revision,
264
+ contract hash, context hash, decision log and digest. Recheck it before
265
+ implementation with `isDiscoveryCurrent`: any source, contract or context
266
+ change makes the earlier result stale. The input is intentionally portable so
267
+ Linear, Orca and Emdash adapters can produce it later without becoming kernel
268
+ dependencies.
269
+
270
+ ## WIP admission
271
+
272
+ Use the same portable approach to decide whether a new issue can start. The
273
+ default limit is three deliveries started and not terminal. `blocked`,
274
+ `awaiting-decision` and `awaiting-acceptance` still consume a delivery slot;
275
+ they release an executor but do not hide unfinished work. Resuming an existing
276
+ non-terminal issue keeps its reservation and takes priority over new work.
277
+
278
+ ```bash
279
+ ak-harness wip assess wip.json --json
280
+ ```
281
+
282
+ ## Runtime experiment
283
+
284
+ Compare Orca and Emdash only when both records carry the same source revision, contract, provider, model, and configuration hash. A failed hard gate is ineligible; the remaining candidates are ordered by human minutes, duration, cost, then `orca` only as the final tie-break.
285
+
286
+ ```sh
287
+ ak-harness experiment select experiment.json --json
288
+ ```
289
+
290
+ The Harness only assesses the supplied ledger. A future Linear adapter owns
291
+ reading and writing the tracker; it must persist the ledger/recovery identity
292
+ and use the existing event-log lock before acting.
293
+
294
+ ## Portable orchestration controls
295
+
296
+ The package includes the small controls needed by an external orchestrator without
297
+ embedding a tracker or provider:
298
+
299
+ ```ts
300
+ import { createDispatchLedger, createOrcaDispatchPlan, planFilePreflight, runWithRecovery } from '@agentskit/harness'
301
+
302
+ const ledger = createDispatchLedger('.codex/verification')
303
+ const claim = ledger.claim({ tracker: 'linear', repository: 'org/repo', issue: 'ENG-1', worktree: 'eng-1', branch: 'codex/eng-1', owner: 'agent' })
304
+ const dispatch = createOrcaDispatchPlan({ repository: 'org/repo', worktree: 'eng-1', branch: 'codex/eng-1', baseBranch: 'main', goalFile: 'GOAL.md' })
305
+ ledger.recordDispatch({ lease: claim.lease, idempotencyKey: dispatch.idempotencyKey, commandDigest: dispatch.commandDigest })
306
+ ```
307
+
308
+ Claims are keyed by tracker, repository, issue, worktree, and branch. They are
309
+ atomic, idempotent, and recoverable only by a human. The ledger never executes
310
+ the command; an Orca adapter may execute the returned argv after recording the
311
+ decision.
312
+
313
+ `planFilePreflight` skips documentation-only changes, selects colocated tests,
314
+ and `validateSafeCommand` rejects shell composition. `runWithRecovery` retries
315
+ only classified retryable failures with a bounded exponential delay and an
316
+ abortable watchdog. `parseRetro` produces proposed learnings; only a human can
317
+ promote them. `createStatusSnapshot` creates a digest-bound status projection.
318
+
319
+ Linear/GitHub and Orca integrations should implement the provider-neutral
320
+ tracking and dispatch adapters; no credentials or network clients belong in
321
+ the kernel.
322
+
218
323
  Agent sessions can record adapter identity, turns, and guarded tool actions
219
324
  during `IMPLEMENTING` without persisting prompt, argument, or result contents:
220
325
 
@@ -338,18 +443,11 @@ action without storing raw output.
338
443
 
339
444
  `benchmark` aggregates the local run history into a versioned JSON report. It
340
445
  includes check/outcome/evidence pass rates, retries, stale runs, human approvals,
341
- and average/median verification duration. It also reports `effective*` metrics
342
- over the latest non-superseded run in each retry lineage, so attempt history is
343
- not confused with the current delivery state. With `--manifest`, it also compares
446
+ and average/median verification duration. With `--manifest`, it also compares
344
447
  bound harness tasks with explicitly recorded baseline observations. A baseline
345
448
  must include evidence for every acceptance criterion; missing, duplicate, or
346
449
  unknown criterion evidence is rejected. Missing baselines and incomplete
347
- evidence remain non-comparable; the harness never invents a baseline. Baseline
348
- collectors also record `baselineArtifactAcceptanceRate`, the fraction of
349
- repeated samples whose real artifact validation passed. This is independent of
350
- protocol completion: a correct artifact with missing or incomplete protocol
351
- evidence is visible as an accepted artifact but remains non-comparable for
352
- duration and resource claims. These are
450
+ evidence remain non-comparable; the harness never invents a baseline. These are
353
451
  execution metrics, not a claim of productivity improvement; compare reports over
354
452
  a controlled task corpus to measure that outcome.
355
453
 
@@ -400,73 +498,32 @@ the report exposes their delta and direction. A positive improvement rate
400
498
  means the harness used less of that measured resource; these metrics do not
401
499
  establish causality or productivity improvement alone.
402
500
 
403
- When a structured check includes `artifactAcceptanceRate` directly or inside
404
- its benchmark `reports`, comparisons also expose the harness rate, contributing
405
- sample count, and acceptance improvement. Missing rates remain uncounted and
406
- never become a failed artifact implicitly.
407
-
408
- Baseline and harness comparisons also expose `protocolCompletionRate`, which
409
- measures whether the verification protocol completed independently of artifact
410
- acceptance. A protocol rate does not override the fail-closed comparability
411
- rules.
412
-
413
- When a baseline rate is zero, relative improvement remains `unavailable`; the
414
- report still exposes the absolute acceptance or protocol completion delta.
415
-
416
- For paired real-provider measurements, `scripts/run-agentskit-os-harness-benchmark.mjs`
417
- prepares one frozen contract per manifest task, binds each run to the suite and
418
- task ID, and leaves the lifecycle at the human approval gate. Use
419
- `--repeats 3` for three independent samples per task; each sample has its own
420
- state directory and must be approved separately. Run `--collect` after
421
- approvals to aggregate the isolated task state directories. Pending or blocked
422
- task runs remain non-comparable; only `COMPLETE` runs produce directional
423
- improvement values. Duration comparisons use the median of completed samples,
424
- while retry metrics count only superseded retry lineages, so experimental
425
- replication is not misreported as agent retry cost.
426
-
427
- When one sample is blocked or times out, prepare a distinct replacement without
428
- overwriting the original evidence:
501
+ ## Repository organization
429
502
 
430
- ```bash
431
- node scripts/run-agentskit-os-harness-benchmark.mjs \
432
- --execute --task-id feat-formatter --sample 4 \
433
- --phase-root .codex/verification/phase-39/harness
434
- ```
435
-
436
- `--sample` requires a positive sample number and refuses an existing config or
437
- state directory. The original blocked run remains part of the audit trail;
438
- replacement runs add evidence and do not retroactively turn the blocked sample
439
- into success.
440
-
441
- The AgentsKit OS benchmark runner can expose the built `ak-verify` CLI inside
442
- its disposable fixture. The fixture contract is prepared by CI, the provider
443
- inherits a fixture-local PATH entry, and human approval is never synthesized.
444
- This makes provider-side verification available without making the fixture a
445
- security boundary; the outer harness remains the authoritative evidence gate.
446
-
447
- Benchmark reports also expose a fail-closed `qualityGate`. It separates
448
- historical retries from effective delivery, reports duration regressions, and
449
- labels small samples as `insufficient-data` or `directional` instead of
450
- claiming enterprise-level improvement. Configure
451
- `minBaselineSamplesPerTask` alongside `minCompletedRunsPerTask`: a single
452
- baseline observation is not comparable to replicated harness runs. Baseline
453
- observations may store `durationSamplesMs`; the report compares medians.
503
+ The Playbook contains guidance; this repository contains the enforceable SDLC
504
+ engine. See [MANIFESTO.md](./MANIFESTO.md) for the boundary and
505
+ [docs/ORGANIZATION.md](./docs/ORGANIZATION.md) for the Angular Conventional
506
+ Commits and capability layout.
454
507
 
455
508
  ## Development
456
509
 
457
510
  ```bash
458
511
  pnpm install
459
- pnpm --filter @agentskit/harness typecheck
460
- pnpm --filter @agentskit/harness test
461
- pnpm --filter @agentskit/harness build
512
+ pnpm typecheck
513
+ pnpm test
514
+ pnpm build
515
+ pnpm pack --pack-destination /tmp/agentskit-harness-pack
462
516
  ```
463
517
 
464
518
  See [CONTRIBUTING.md](./CONTRIBUTING.md) for changes, tests, and release expectations. See [CHANGELOG.md](./CHANGELOG.md) for version history.
465
519
 
466
- The Playbook CI dogfoods the package with `harness:test`, `harness:cli`, and a
467
- clean packed-consumer probe before the broader repository checks run.
468
- It also uploads the resulting `.codex/verification/harness-phase-24` run
469
- directory so a human can inspect the exact evidence before approval.
520
+ ## Release
521
+
522
+ Releases are published by `.github/workflows/release-harness.yml` after a merge to
523
+ `main`. The workflow uses npm Trusted Publishing (GitHub OIDC) and does not read
524
+ or require an `NPM_TOKEN`. Configure the npm trusted publisher once for
525
+ `AgentsKit-io/harness`, workflow `release-harness.yml`, and package
526
+ `@agentskit/harness`; version changes remain the release trigger.
470
527
 
471
528
  ## License
472
529