@agentskit/harness 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -32
- package/CONTRIBUTING.md +60 -12
- package/MANIFESTO.md +23 -0
- package/README.md +199 -142
- package/dist/cli.js +796 -221
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +739 -121
- package/dist/index.js +1179 -303
- package/dist/index.js.map +1 -1
- package/docs/ADR-0025-portable-orchestration-controls.md +27 -0
- package/docs/ORGANIZATION.md +37 -0
- package/package.json +41 -34
- package/docs/ADR-0025-ci-dogfood.md +0 -22
- package/docs/ADR-0026-ci-evidence-artifact.md +0 -22
- package/docs/ADR-0027-portable-evidence.md +0 -19
- package/docs/ADR-0028-effective-metrics.md +0 -20
- package/docs/ADR-0029-honest-ci-preparation.md +0 -20
- package/docs/ADR-0030-agentskit-os-benchmark-bridge.md +0 -20
- package/docs/ADR-0031-real-provider-baseline.md +0 -18
- package/docs/ADR-0032-harness-equivalent-benchmark.md +0 -25
- package/docs/ADR-0033-portable-agent-gate.md +0 -25
- package/docs/ADR-0034-measurement-quality-gates.md +0 -25
- package/docs/ADR-0035-reproducible-benchmark-samples.md +0 -20
- package/docs/ADR-0036-comparable-baseline-samples.md +0 -20
- package/docs/ADR-0037-replicated-baseline-collection.md +0 -27
- package/docs/ADR-0038-end-to-end-benchmark-boundary.md +0 -28
- package/docs/ADR-0039-artifact-and-protocol-metrics.md +0 -39
- package/docs/ADR-0040-benchmark-corpus-surfaces.md +0 -32
package/CHANGELOG.md
CHANGED
|
@@ -1,44 +1,38 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.3.0] - 2026-09-10
|
|
4
|
+
|
|
5
|
+
- Added portable issue/worktree claims and idempotent dispatch ledger.
|
|
6
|
+
- Added failure classification, bounded retry/backoff, and abortable watchdog.
|
|
7
|
+
- Added file-scoped preflight planning and shell-composition rejection.
|
|
8
|
+
- Added block manifests, status snapshots, retro learning promotion, model
|
|
9
|
+
policies, and provider-neutral Orca/tracking adapters.
|
|
10
|
+
- Added configurable machine pressure thresholds and adaptive workflow limits.
|
|
11
|
+
|
|
12
|
+
## [Unreleased]
|
|
13
|
+
|
|
14
|
+
- Add bounded agent eval, safe context/read-only LLM cache, deterministic workflow fan-out/fan-in, and validated optimization observation contracts for token, memory, cache, and parallelism measurements.
|
|
15
|
+
|
|
3
16
|
All notable changes to `@agentskit/harness` are documented here.
|
|
4
17
|
|
|
5
18
|
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the package follows Semantic Versioning.
|
|
6
19
|
|
|
7
|
-
## [0.
|
|
20
|
+
## [0.2.0] - 2026-09-09
|
|
8
21
|
|
|
9
22
|
### Added
|
|
10
23
|
|
|
11
|
-
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
-
|
|
21
|
-
|
|
22
|
-
-
|
|
23
|
-
completion rates separately from performance comparability.
|
|
24
|
-
- Protocol and artifact acceptance comparisons expose absolute deltas when
|
|
25
|
-
relative improvement is undefined because the baseline was zero.
|
|
26
|
-
- Benchmark tasks declare typed surface coverage; the AgentsKit OS corpus
|
|
27
|
-
includes executable CLI and documentation tasks with real acceptance
|
|
28
|
-
validation.
|
|
29
|
-
- Fail-closed baseline sample policy and median baseline duration support for fair replicated comparisons.
|
|
30
|
-
- Independent benchmark samples with median-duration comparisons and retry metrics that do not confuse replication with retries.
|
|
31
|
-
- Configurable benchmark quality gates, confidence labels, and explicit retry metrics.
|
|
32
|
-
- Portable `ak-verify` fixture gate for real coding-agent benchmark runs; provider delivery remains blocked until its result is complete and human approval is recorded.
|
|
33
|
-
- Harness-equivalent benchmark preparation and collection for paired real-provider runs, with explicit human approval and honest non-comparability.
|
|
34
|
-
- Source snapshots now ignore untracked directories while retaining hashes for untracked files.
|
|
35
|
-
- Reproducible disposable-fixture runner and validator for the first real-provider AgentsKit OS baseline; improvement remains unavailable until equivalent harness runs exist.
|
|
36
|
-
- Provenance-bearing benchmark tasks and a portable validator for AgentsKit OS coding benchmark reports; missing controlled observations remain explicitly non-comparable.
|
|
37
|
-
- Playbook CI dogfooding for the package test suite, built CLI, and clean packed consumer.
|
|
38
|
-
- Playbook CI now preserves the official harness run as an auditable artifact for human review.
|
|
39
|
-
- Signed evidence flow now verifies a copied bundle from an isolated directory.
|
|
40
|
-
- Benchmark reports now separate effective non-superseded delivery metrics from attempt history.
|
|
41
|
-
- CI evidence now records automated preparation separately from human contract approval.
|
|
24
|
+
- Pilot-cohort validation that freezes a policy/baseline pair and rejects non-normal, partial, or silently substituted ten-issue cohorts.
|
|
25
|
+
- Bounded five-step improvement-cycle assessment with explicit adjustments, repeat decisions, and a deterministic quality matrix.
|
|
26
|
+
|
|
27
|
+
- Deterministic G2–G5 assessment helpers and CLI commands for independent preflight review, structured idempotent PR handoff, current integration evidence, safe production exposure, and acceptance.
|
|
28
|
+
|
|
29
|
+
- Current-source evidence now requires a committed Git `HEAD`; directories outside Git fail closed instead of receiving a synthetic revision.
|
|
30
|
+
|
|
31
|
+
- Discovery gate API and CLI that emit `ready` or an auditable human decision packet from structured ambiguities, approved assumptions and source/contract/context bindings.
|
|
32
|
+
- Deterministic WIP admission API and CLI that count blocked and awaiting-human deliveries, reserve resumed work, and reject duplicate ledger entries.
|
|
33
|
+
- Controlled runtime-selection API and CLI that reject incomparable Orca/Emdash samples and exclude failed hard gates.
|
|
34
|
+
- Configurable `runtime.kind` contract field plus a factory for bounded process or Docker-sandbox execution.
|
|
35
|
+
- Real Git snapshot coverage for committed, dirty, untracked, and task-state-excluded evidence.
|
|
42
36
|
- Strict TypeScript modular core with generated declarations and source maps.
|
|
43
37
|
- Contract-frozen lifecycle, structured evidence, stale detection, human approval, retry, and cleanup.
|
|
44
38
|
- Explicit human cancellation and superseded retry history.
|
package/CONTRIBUTING.md
CHANGED
|
@@ -1,25 +1,73 @@
|
|
|
1
1
|
# Contributing to `@agentskit/harness`
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
The Harness is an enforcement engine, not a suggestion library. Changes must
|
|
4
|
+
preserve fail-closed behavior and produce criterion-level evidence.
|
|
4
5
|
|
|
5
|
-
|
|
6
|
-
2. Keep the public API in `src/index.ts` intentionally small.
|
|
7
|
-
3. Add or update a typed test for every behavior change.
|
|
8
|
-
4. Update the README, changelog, and contract when the user-facing protocol changes.
|
|
6
|
+
## Repository layout
|
|
9
7
|
|
|
10
|
-
|
|
8
|
+
- `src/`: capability-oriented TypeScript modules and the public `index.ts`.
|
|
9
|
+
- `src/adapters/`: optional provider integrations.
|
|
10
|
+
- `test/`: deterministic unit and contract tests.
|
|
11
|
+
- `scripts/`: real CLI and packaging checks.
|
|
12
|
+
- `docs/`: ADRs and protocol decisions.
|
|
13
|
+
- `.github/`: CI and release automation.
|
|
11
14
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
15
|
+
See [docs/ORGANIZATION.md](./docs/ORGANIZATION.md) for the complete boundary
|
|
16
|
+
and naming rules.
|
|
17
|
+
|
|
18
|
+
## Change workflow
|
|
19
|
+
|
|
20
|
+
1. Start from an issue or explicit task contract.
|
|
21
|
+
2. Identify the outcome and executable validation it requires.
|
|
22
|
+
3. Keep the public API change in `src/index.ts` intentional.
|
|
23
|
+
4. Add or update criterion-level tests for behavior changes.
|
|
24
|
+
5. Update README, changelog, ADRs, or the verification contract when the
|
|
25
|
+
protocol changes.
|
|
26
|
+
6. Run the complete local gate:
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
pnpm typecheck
|
|
30
|
+
pnpm test
|
|
31
|
+
pnpm build
|
|
32
|
+
pnpm pack --pack-destination /tmp/agentskit-harness-pack
|
|
33
|
+
ak-verify run --config .codex/verification.json --json
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Do not report completion while a required gate is unavailable, blocked, stale,
|
|
37
|
+
or awaiting approval.
|
|
38
|
+
|
|
39
|
+
## Angular Conventional Commits
|
|
40
|
+
|
|
41
|
+
Commit messages follow the Angular Conventional Commits format:
|
|
42
|
+
|
|
43
|
+
```text
|
|
44
|
+
<type>(<scope>): <imperative description>
|
|
16
45
|
```
|
|
17
46
|
|
|
18
|
-
|
|
47
|
+
Allowed types are `feat`, `fix`, `docs`, `refactor`, `test`, `build`, `ci`,
|
|
48
|
+
`perf`, `chore`, and `revert`. Use a breaking-change footer when needed:
|
|
49
|
+
|
|
50
|
+
```text
|
|
51
|
+
feat(cli)!: reject incomplete evidence
|
|
52
|
+
|
|
53
|
+
BREAKING CHANGE: `run` now exits non-zero when evidence is missing.
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Keep commits focused. Never commit secrets, generated `dist/`, verification
|
|
57
|
+
state, or residue from another task.
|
|
19
58
|
|
|
20
59
|
## Pull requests
|
|
21
60
|
|
|
22
|
-
|
|
61
|
+
PRs must state the contract change, affected states, evidence produced, exact
|
|
62
|
+
commands executed, and any unresolved blockers. A maintainer review is required
|
|
63
|
+
for public API, CLI, state-machine, security, or release changes.
|
|
64
|
+
|
|
65
|
+
## Releases
|
|
66
|
+
|
|
67
|
+
Version changes merge to `main` through a PR. The release workflow runs tests,
|
|
68
|
+
build, and pack, then publishes through npm Trusted Publishing (GitHub OIDC);
|
|
69
|
+
no `NPM_TOKEN` is used. Configure the npm trusted publisher for
|
|
70
|
+
`AgentsKit-io/harness` once before the first release.
|
|
23
71
|
|
|
24
72
|
## License
|
|
25
73
|
|
package/MANIFESTO.md
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Harness Manifesto
|
|
2
|
+
|
|
3
|
+
## The boundary
|
|
4
|
+
|
|
5
|
+
The Playbook explains good practice. The Harness makes the SDLC observable and
|
|
6
|
+
enforceable. Advice may be ignored; a required gate may not.
|
|
7
|
+
|
|
8
|
+
## The guarantees
|
|
9
|
+
|
|
10
|
+
1. A task starts from an explicit, frozen contract.
|
|
11
|
+
2. Ambiguities are surfaced with options and a recommendation; the Harness
|
|
12
|
+
never invents product decisions.
|
|
13
|
+
3. Every required check produces structured evidence bound to source, config,
|
|
14
|
+
and contract hashes.
|
|
15
|
+
4. A failed or missing gate blocks progression and remains auditable.
|
|
16
|
+
5. Recovery resumes from the journal without silently replaying uncertain work.
|
|
17
|
+
6. Integrations are replaceable adapters, not hidden kernel dependencies.
|
|
18
|
+
7. Measurements report what was observed, including unavailable data.
|
|
19
|
+
|
|
20
|
+
## Operating principle
|
|
21
|
+
|
|
22
|
+
Automate everything deterministic. Escalate only decisions that require human
|
|
23
|
+
judgment. Keep the smallest mechanism that proves the claim.
|
package/README.md
CHANGED
|
@@ -1,3 +1,9 @@
|
|
|
1
|
+
---
|
|
2
|
+
docbridge:
|
|
3
|
+
covers:
|
|
4
|
+
- package:@agentskit/harness
|
|
5
|
+
---
|
|
6
|
+
|
|
1
7
|
# @agentskit/harness
|
|
2
8
|
|
|
3
9
|
Portable, evidence-backed development protocol for coding agents. The harness freezes a human-approved task contract, executes every configured check, binds evidence to the current source revision, detects stale results, and refuses completion without human approval.
|
|
@@ -24,79 +30,7 @@ ak-harness benchmark --manifest benchmarks/harness-phase-0.json --json
|
|
|
24
30
|
|
|
25
31
|
`plan` rejects unresolved ambiguities and unauthorized dirty worktrees. After `start`, the contract is frozen. Any source, configuration, or contract change invalidates evidence and moves the run to `STALE`. A human can cancel an active run; retrying a blocked, stale, or cancelled run marks the previous run `SUPERSEDED`.
|
|
26
32
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
## External coding benchmark bridge
|
|
30
|
-
|
|
31
|
-
The harness validates a provenance-bearing task manifest and the JSON report
|
|
32
|
-
emitted by an external coding benchmark, including AgentsKit OS:
|
|
33
|
-
|
|
34
|
-
```bash
|
|
35
|
-
AGENTSKIT_OS_ROOT=/path/to/agentskit-os \
|
|
36
|
-
AGENTSKIT_OS_BENCHMARK_REPORT=/path/to/report.json \
|
|
37
|
-
node scripts/verify-harness-agentskit-os-benchmark.mjs \
|
|
38
|
-
--manifest benchmarks/agentskit-os-phase-28.json \
|
|
39
|
-
--target "$AGENTSKIT_OS_ROOT" \
|
|
40
|
-
--report "$AGENTSKIT_OS_BENCHMARK_REPORT"
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
The bridge checks the pinned source revision, task definition, prompt digests,
|
|
44
|
-
scope, and provider report shape. Provider status and heuristic scores remain
|
|
45
|
-
observations: they do not grant human acceptance. With no controlled baseline,
|
|
46
|
-
improvement is reported as unavailable rather than inferred.
|
|
47
|
-
|
|
48
|
-
The phase-45 corpus adds explicit task surfaces and keeps the task definition
|
|
49
|
-
bound to the real AgentsKit OS fixture. It covers logic, CLI, and documentation
|
|
50
|
-
surfaces; the added CLI task executes compact, pretty, and invalid-input paths,
|
|
51
|
-
while the documentation task checks the API contract and its reproducible test
|
|
52
|
-
command:
|
|
53
|
-
|
|
54
|
-
```bash
|
|
55
|
-
node scripts/verify-harness-agentskit-os-benchmark.mjs \
|
|
56
|
-
--manifest benchmarks/agentskit-os-phase-45.json \
|
|
57
|
-
--target /path/to/agentskit-os \
|
|
58
|
-
--require-surface-coverage logic,cli,docs
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
Corpus coverage is a readiness gate, not a performance claim. Live provider
|
|
62
|
-
samples and human approval remain required before the expanded tasks become
|
|
63
|
-
comparable benchmark evidence.
|
|
64
|
-
|
|
65
|
-
Phase 29 adds a reproducible real-provider baseline runner for the AgentsKit OS
|
|
66
|
-
seed corpus. It uses disposable fixture copies, the existing OS benchmark
|
|
67
|
-
runner, and task-specific validation:
|
|
68
|
-
|
|
69
|
-
```bash
|
|
70
|
-
node scripts/run-agentskit-os-baseline.mjs \
|
|
71
|
-
--target /path/to/agentskit-os \
|
|
72
|
-
--provider codex \
|
|
73
|
-
--output benchmarks/agentskit-os-phase-29-baseline
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
The observations are not an improvement claim until equivalent harness runs
|
|
77
|
-
exist.
|
|
78
|
-
|
|
79
|
-
To collect equivalent baseline samples, run the same corpus independently and
|
|
80
|
-
record the aggregate only after every task has been sampled:
|
|
81
|
-
|
|
82
|
-
```bash
|
|
83
|
-
node scripts/run-agentskit-os-baseline.mjs \
|
|
84
|
-
--target /path/to/agentskit-os \
|
|
85
|
-
--provider codex \
|
|
86
|
-
--repeats 3 \
|
|
87
|
-
--output benchmarks/agentskit-os-phase-36-baseline \
|
|
88
|
-
--record-manifest benchmarks/agentskit-os-phase-28.json
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
Each repeat uses a fresh disposable fixture. The report stores the end-to-end
|
|
92
|
-
duration of fixture setup, provider execution, validation, and cleanup in
|
|
93
|
-
`durationSamplesMs`; the provider's own duration remains nested in the raw
|
|
94
|
-
report. The manifest is replaced atomically through
|
|
95
|
-
the typed observation recorder. `--record-manifest` requires the complete task
|
|
96
|
-
set; omit it to inspect an uncommitted collection. Replicas are not retries,
|
|
97
|
-
and a failed sample keeps the aggregate failed. Escape reduction may remain
|
|
98
|
-
visible as a separate outcome, but duration, attempt, and review improvements
|
|
99
|
-
remain unavailable until the baseline proves complete delivery.
|
|
33
|
+
Current-source evidence requires a committed Git `HEAD`. A directory outside Git, or a repository without its first commit, is rejected as `GIT_REQUIRED`; it is not a supported pilot mode because it cannot prove revision currentness.
|
|
100
34
|
|
|
101
35
|
## Contract
|
|
102
36
|
|
|
@@ -151,6 +85,13 @@ the current state, so a post-approval edit cannot appear as `COMPLETE`.
|
|
|
151
85
|
Concurrent event writers are serialized by an atomic per-run lock and fail
|
|
152
86
|
closed if the log is busy.
|
|
153
87
|
|
|
88
|
+
Each harness event may also carry an optional `correlation` envelope. Its
|
|
89
|
+
`operationId` is the stable identity used when a lifecycle crosses into
|
|
90
|
+
AgentsKit, Chat, Doc Bridge, or Code Review; the optional `runId`, `sessionId`,
|
|
91
|
+
`turnId`, `actionId`, and `traceId` remain local identities. The envelope is
|
|
92
|
+
bounded metadata only and never contains prompts, arguments, results, or
|
|
93
|
+
secrets.
|
|
94
|
+
|
|
154
95
|
Export a reconciled `COMPLETE` run for external review with an Ed25519 key:
|
|
155
96
|
|
|
156
97
|
```bash
|
|
@@ -162,9 +103,6 @@ The bundle includes the run projection, event log, and referenced check outputs,
|
|
|
162
103
|
each with a SHA-256 digest. A trust store can mark keys `active` or `revoked` to
|
|
163
104
|
support controlled key rotation. Treat exported outputs as potentially sensitive.
|
|
164
105
|
|
|
165
|
-
The signed bundle is self-contained: it can be copied to an isolated directory
|
|
166
|
-
and verified there with only the bundle and the trusted public key.
|
|
167
|
-
|
|
168
106
|
Profiles are optional declarative overlays in `.codex/verification.json`. They
|
|
169
107
|
inherit in order, override existing checks by ID, and are resolved before the
|
|
170
108
|
contract is frozen:
|
|
@@ -181,6 +119,31 @@ contract is frozen:
|
|
|
181
119
|
}
|
|
182
120
|
```
|
|
183
121
|
|
|
122
|
+
`runtime.kind` chooses the executor used by an integration: `process` is a bounded shell-free local child process; `docker` adds the Docker sandbox. The choice is frozen in the resolved contract and therefore changes its hash. Docker remains fail-closed when its daemon or image is unavailable.
|
|
123
|
+
|
|
124
|
+
`autonomy: "yolo"` removes the generic final review only after every applicable check passes, tracking is disabled, and the frozen contract has no ambiguity. It never auto-approves a material decision, external tracking, or a tool rule that requires approval.
|
|
125
|
+
|
|
126
|
+
Use named profiles to make the operational choice explicit:
|
|
127
|
+
|
|
128
|
+
```json
|
|
129
|
+
{
|
|
130
|
+
"profile": "process",
|
|
131
|
+
"runtime": { "kind": "process" },
|
|
132
|
+
"profiles": {
|
|
133
|
+
"process": { "runtime": { "kind": "process" } },
|
|
134
|
+
"docker": { "runtime": { "kind": "docker" } }
|
|
135
|
+
}
|
|
136
|
+
}
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
```ts
|
|
140
|
+
const runtime = createConfiguredToolRuntime({
|
|
141
|
+
runtime: loaded.config.runtime,
|
|
142
|
+
process: { tools: processTools },
|
|
143
|
+
docker: { tools: dockerTools },
|
|
144
|
+
})
|
|
145
|
+
```
|
|
146
|
+
|
|
184
147
|
Doc Bridge and Playbook integrations can implement `ContextProvider` and
|
|
185
148
|
register it through `CONTEXT_PROVIDER_SLOT`; the kernel records neither their
|
|
186
149
|
credentials nor their transport and does not depend on either package. The
|
|
@@ -215,6 +178,148 @@ stable `contextHash`; resolution time is metadata and does not change the
|
|
|
215
178
|
reproducibility hash. Context is resolved before planning and is frozen with
|
|
216
179
|
the run, so later index changes cannot silently change its evidence.
|
|
217
180
|
|
|
181
|
+
## Delivery gates
|
|
182
|
+
|
|
183
|
+
The optional delivery helpers evaluate gates without embedding a GitHub, Linear,
|
|
184
|
+
or deployment provider. Adapters perform external effects only after the
|
|
185
|
+
deterministic decision is recorded:
|
|
186
|
+
|
|
187
|
+
```ts
|
|
188
|
+
const g2 = assessPreflight({
|
|
189
|
+
criteria,
|
|
190
|
+
implementerId: 'implementer',
|
|
191
|
+
reviewerId: 'independent-reviewer',
|
|
192
|
+
reviewKind: 'adversarial',
|
|
193
|
+
reviewApproved: true,
|
|
194
|
+
})
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
G2 ignores later-gate pending criteria but blocks failed/pending G2 evidence,
|
|
198
|
+
self-review, and a third repair. `composePullRequest` creates a structured PR
|
|
199
|
+
body only from an approved, current G2 result; it reuses a confirmed matching
|
|
200
|
+
remote PR and preserves uncertain state. G3 binds CI to the candidate revision.
|
|
201
|
+
G4 requires an approved environment profile, identified artifact, isolation or
|
|
202
|
+
version-bound acceptance, technical evidence, and a 15-minute low-risk window.
|
|
203
|
+
G5 remains `awaiting-acceptance` until the applicable business/UX decision is
|
|
204
|
+
recorded. `assessWorktreeCleanup` permits cleanup only after remote branch SHA,
|
|
205
|
+
PR, and G3 all match.
|
|
206
|
+
|
|
207
|
+
## Pilot cohort
|
|
208
|
+
|
|
209
|
+
`ak-harness pilot cohort.json` validates the frozen cohort before work starts.
|
|
210
|
+
It requires a policy hash, a baseline reference, exactly ten included issues,
|
|
211
|
+
and the `normal` classification for each. Excluded or aborted issues need a
|
|
212
|
+
reason and cannot be silently replaced in the same manifest.
|
|
213
|
+
|
|
214
|
+
## Improvement cycle
|
|
215
|
+
|
|
216
|
+
The five pilot steps can be evaluated as a bounded cycle: adversarial review,
|
|
217
|
+
G2 preflight, baseline recording, pilot execution, and harness/no-harness
|
|
218
|
+
comparison. Each iteration must contain those steps in order. A failed,
|
|
219
|
+
blocked, or pending step requires a reason; repeating requires an explicit
|
|
220
|
+
adjustment. The assessment returns `complete`, `repeat` (with the next
|
|
221
|
+
iteration), or `blocked` when the adjustment is missing or the iteration
|
|
222
|
+
budget is exhausted, plus a criterion-level matrix:
|
|
223
|
+
|
|
224
|
+
```bash
|
|
225
|
+
ak-harness cycle assess cycle.json --json
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
This is a deterministic decision helper: Orca, Emdash, GitHub, Linear, and
|
|
229
|
+
other adapters remain responsible for executing external actions and supplying
|
|
230
|
+
their structured results.
|
|
231
|
+
|
|
232
|
+
## Agent optimization and evaluation
|
|
233
|
+
|
|
234
|
+
The Harness also validates the optimization layer without owning a provider.
|
|
235
|
+
The memory boundary accepts only approved records with explicit scope, source
|
|
236
|
+
revision, and content hash; an adapter can back it with AgentsKit memory.
|
|
237
|
+
`runAgentEval` runs a bounded suite and returns criterion-level accuracy;
|
|
238
|
+
`createLlmCache` provides deterministic keys and hit/miss/invalidation evidence
|
|
239
|
+
for context and read-only calls; and `runWorkflow` executes independent nodes
|
|
240
|
+
in sorted, bounded fan-out/fan-in batches while serializing nodes that share a
|
|
241
|
+
`mutationKey`. `OptimizationObservation` carries
|
|
242
|
+
optional token, memory, cache, and parallelism measurements and refuses
|
|
243
|
+
incomparable provider/model/configuration bindings.
|
|
244
|
+
|
|
245
|
+
These are seams, not replacements for AgentsKit packages. An integration may
|
|
246
|
+
adapt `@agentskit/memory` and `@agentskit/eval` into them while keeping the
|
|
247
|
+
Harness provider-neutral. Missing measurements remain missing; they are never
|
|
248
|
+
reported as zero.
|
|
249
|
+
|
|
250
|
+
## Discovery gate
|
|
251
|
+
|
|
252
|
+
Discovery is a small deterministic gate before implementation. An adapter or
|
|
253
|
+
agent supplies a structured list of ambiguities; the Harness does not decide
|
|
254
|
+
product questions. Material ambiguities produce one decision packet with
|
|
255
|
+
options and a recommendation. A non-material ambiguity can proceed only when
|
|
256
|
+
an approved policy assumption covers it, and the result records that policy in
|
|
257
|
+
the decision log.
|
|
258
|
+
|
|
259
|
+
```bash
|
|
260
|
+
ak-harness discovery assess discovery.json --json
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
The result is `ready` or `awaiting-decision`, and carries the source revision,
|
|
264
|
+
contract hash, context hash, decision log and digest. Recheck it before
|
|
265
|
+
implementation with `isDiscoveryCurrent`: any source, contract or context
|
|
266
|
+
change makes the earlier result stale. The input is intentionally portable so
|
|
267
|
+
Linear, Orca and Emdash adapters can produce it later without becoming kernel
|
|
268
|
+
dependencies.
|
|
269
|
+
|
|
270
|
+
## WIP admission
|
|
271
|
+
|
|
272
|
+
Use the same portable approach to decide whether a new issue can start. The
|
|
273
|
+
default limit is three deliveries started and not terminal. `blocked`,
|
|
274
|
+
`awaiting-decision` and `awaiting-acceptance` still consume a delivery slot;
|
|
275
|
+
they release an executor but do not hide unfinished work. Resuming an existing
|
|
276
|
+
non-terminal issue keeps its reservation and takes priority over new work.
|
|
277
|
+
|
|
278
|
+
```bash
|
|
279
|
+
ak-harness wip assess wip.json --json
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
## Runtime experiment
|
|
283
|
+
|
|
284
|
+
Compare Orca and Emdash only when both records carry the same source revision, contract, provider, model, and configuration hash. A failed hard gate is ineligible; the remaining candidates are ordered by human minutes, duration, cost, then `orca` only as the final tie-break.
|
|
285
|
+
|
|
286
|
+
```sh
|
|
287
|
+
ak-harness experiment select experiment.json --json
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
The Harness only assesses the supplied ledger. A future Linear adapter owns
|
|
291
|
+
reading and writing the tracker; it must persist the ledger/recovery identity
|
|
292
|
+
and use the existing event-log lock before acting.
|
|
293
|
+
|
|
294
|
+
## Portable orchestration controls
|
|
295
|
+
|
|
296
|
+
The package includes the small controls needed by an external orchestrator without
|
|
297
|
+
embedding a tracker or provider:
|
|
298
|
+
|
|
299
|
+
```ts
|
|
300
|
+
import { createDispatchLedger, createOrcaDispatchPlan, planFilePreflight, runWithRecovery } from '@agentskit/harness'
|
|
301
|
+
|
|
302
|
+
const ledger = createDispatchLedger('.codex/verification')
|
|
303
|
+
const claim = ledger.claim({ tracker: 'linear', repository: 'org/repo', issue: 'ENG-1', worktree: 'eng-1', branch: 'codex/eng-1', owner: 'agent' })
|
|
304
|
+
const dispatch = createOrcaDispatchPlan({ repository: 'org/repo', worktree: 'eng-1', branch: 'codex/eng-1', baseBranch: 'main', goalFile: 'GOAL.md' })
|
|
305
|
+
ledger.recordDispatch({ lease: claim.lease, idempotencyKey: dispatch.idempotencyKey, commandDigest: dispatch.commandDigest })
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
Claims are keyed by tracker, repository, issue, worktree, and branch. They are
|
|
309
|
+
atomic, idempotent, and recoverable only by a human. The ledger never executes
|
|
310
|
+
the command; an Orca adapter may execute the returned argv after recording the
|
|
311
|
+
decision.
|
|
312
|
+
|
|
313
|
+
`planFilePreflight` skips documentation-only changes, selects colocated tests,
|
|
314
|
+
and `validateSafeCommand` rejects shell composition. `runWithRecovery` retries
|
|
315
|
+
only classified retryable failures with a bounded exponential delay and an
|
|
316
|
+
abortable watchdog. `parseRetro` produces proposed learnings; only a human can
|
|
317
|
+
promote them. `createStatusSnapshot` creates a digest-bound status projection.
|
|
318
|
+
|
|
319
|
+
Linear/GitHub and Orca integrations should implement the provider-neutral
|
|
320
|
+
tracking and dispatch adapters; no credentials or network clients belong in
|
|
321
|
+
the kernel.
|
|
322
|
+
|
|
218
323
|
Agent sessions can record adapter identity, turns, and guarded tool actions
|
|
219
324
|
during `IMPLEMENTING` without persisting prompt, argument, or result contents:
|
|
220
325
|
|
|
@@ -338,18 +443,11 @@ action without storing raw output.
|
|
|
338
443
|
|
|
339
444
|
`benchmark` aggregates the local run history into a versioned JSON report. It
|
|
340
445
|
includes check/outcome/evidence pass rates, retries, stale runs, human approvals,
|
|
341
|
-
and average/median verification duration.
|
|
342
|
-
over the latest non-superseded run in each retry lineage, so attempt history is
|
|
343
|
-
not confused with the current delivery state. With `--manifest`, it also compares
|
|
446
|
+
and average/median verification duration. With `--manifest`, it also compares
|
|
344
447
|
bound harness tasks with explicitly recorded baseline observations. A baseline
|
|
345
448
|
must include evidence for every acceptance criterion; missing, duplicate, or
|
|
346
449
|
unknown criterion evidence is rejected. Missing baselines and incomplete
|
|
347
|
-
evidence remain non-comparable; the harness never invents a baseline.
|
|
348
|
-
collectors also record `baselineArtifactAcceptanceRate`, the fraction of
|
|
349
|
-
repeated samples whose real artifact validation passed. This is independent of
|
|
350
|
-
protocol completion: a correct artifact with missing or incomplete protocol
|
|
351
|
-
evidence is visible as an accepted artifact but remains non-comparable for
|
|
352
|
-
duration and resource claims. These are
|
|
450
|
+
evidence remain non-comparable; the harness never invents a baseline. These are
|
|
353
451
|
execution metrics, not a claim of productivity improvement; compare reports over
|
|
354
452
|
a controlled task corpus to measure that outcome.
|
|
355
453
|
|
|
@@ -400,73 +498,32 @@ the report exposes their delta and direction. A positive improvement rate
|
|
|
400
498
|
means the harness used less of that measured resource; these metrics do not
|
|
401
499
|
establish causality or productivity improvement alone.
|
|
402
500
|
|
|
403
|
-
|
|
404
|
-
its benchmark `reports`, comparisons also expose the harness rate, contributing
|
|
405
|
-
sample count, and acceptance improvement. Missing rates remain uncounted and
|
|
406
|
-
never become a failed artifact implicitly.
|
|
407
|
-
|
|
408
|
-
Baseline and harness comparisons also expose `protocolCompletionRate`, which
|
|
409
|
-
measures whether the verification protocol completed independently of artifact
|
|
410
|
-
acceptance. A protocol rate does not override the fail-closed comparability
|
|
411
|
-
rules.
|
|
412
|
-
|
|
413
|
-
When a baseline rate is zero, relative improvement remains `unavailable`; the
|
|
414
|
-
report still exposes the absolute acceptance or protocol completion delta.
|
|
415
|
-
|
|
416
|
-
For paired real-provider measurements, `scripts/run-agentskit-os-harness-benchmark.mjs`
|
|
417
|
-
prepares one frozen contract per manifest task, binds each run to the suite and
|
|
418
|
-
task ID, and leaves the lifecycle at the human approval gate. Use
|
|
419
|
-
`--repeats 3` for three independent samples per task; each sample has its own
|
|
420
|
-
state directory and must be approved separately. Run `--collect` after
|
|
421
|
-
approvals to aggregate the isolated task state directories. Pending or blocked
|
|
422
|
-
task runs remain non-comparable; only `COMPLETE` runs produce directional
|
|
423
|
-
improvement values. Duration comparisons use the median of completed samples,
|
|
424
|
-
while retry metrics count only superseded retry lineages, so experimental
|
|
425
|
-
replication is not misreported as agent retry cost.
|
|
426
|
-
|
|
427
|
-
When one sample is blocked or times out, prepare a distinct replacement without
|
|
428
|
-
overwriting the original evidence:
|
|
501
|
+
## Repository organization
|
|
429
502
|
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
```
|
|
435
|
-
|
|
436
|
-
`--sample` requires a positive sample number and refuses an existing config or
|
|
437
|
-
state directory. The original blocked run remains part of the audit trail;
|
|
438
|
-
replacement runs add evidence and do not retroactively turn the blocked sample
|
|
439
|
-
into success.
|
|
440
|
-
|
|
441
|
-
The AgentsKit OS benchmark runner can expose the built `ak-verify` CLI inside
|
|
442
|
-
its disposable fixture. The fixture contract is prepared by CI, the provider
|
|
443
|
-
inherits a fixture-local PATH entry, and human approval is never synthesized.
|
|
444
|
-
This makes provider-side verification available without making the fixture a
|
|
445
|
-
security boundary; the outer harness remains the authoritative evidence gate.
|
|
446
|
-
|
|
447
|
-
Benchmark reports also expose a fail-closed `qualityGate`. It separates
|
|
448
|
-
historical retries from effective delivery, reports duration regressions, and
|
|
449
|
-
labels small samples as `insufficient-data` or `directional` instead of
|
|
450
|
-
claiming enterprise-level improvement. Configure
|
|
451
|
-
`minBaselineSamplesPerTask` alongside `minCompletedRunsPerTask`: a single
|
|
452
|
-
baseline observation is not comparable to replicated harness runs. Baseline
|
|
453
|
-
observations may store `durationSamplesMs`; the report compares medians.
|
|
503
|
+
The Playbook contains guidance; this repository contains the enforceable SDLC
|
|
504
|
+
engine. See [MANIFESTO.md](./MANIFESTO.md) for the boundary and
|
|
505
|
+
[docs/ORGANIZATION.md](./docs/ORGANIZATION.md) for the Angular Conventional
|
|
506
|
+
Commits and capability layout.
|
|
454
507
|
|
|
455
508
|
## Development
|
|
456
509
|
|
|
457
510
|
```bash
|
|
458
511
|
pnpm install
|
|
459
|
-
pnpm
|
|
460
|
-
pnpm
|
|
461
|
-
pnpm
|
|
512
|
+
pnpm typecheck
|
|
513
|
+
pnpm test
|
|
514
|
+
pnpm build
|
|
515
|
+
pnpm pack --pack-destination /tmp/agentskit-harness-pack
|
|
462
516
|
```
|
|
463
517
|
|
|
464
518
|
See [CONTRIBUTING.md](./CONTRIBUTING.md) for changes, tests, and release expectations. See [CHANGELOG.md](./CHANGELOG.md) for version history.
|
|
465
519
|
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
520
|
+
## Release
|
|
521
|
+
|
|
522
|
+
Releases are published by `.github/workflows/release-harness.yml` after a merge to
|
|
523
|
+
`main`. The workflow uses npm Trusted Publishing (GitHub OIDC) and does not read
|
|
524
|
+
or require an `NPM_TOKEN`. Configure the npm trusted publisher once for
|
|
525
|
+
`AgentsKit-io/harness`, workflow `release-harness.yml`, and package
|
|
526
|
+
`@agentskit/harness`; version changes remain the release trigger.
|
|
470
527
|
|
|
471
528
|
## License
|
|
472
529
|
|