@agentskit/harness 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +77 -0
- package/CODE_OF_CONDUCT.md +5 -0
- package/CONTRIBUTING.md +26 -0
- package/LICENSE +21 -0
- package/README.md +473 -0
- package/SECURITY.md +11 -0
- package/dist/cli.js +1308 -0
- package/dist/cli.js.map +1 -0
- package/dist/index.d.ts +968 -0
- package/dist/index.js +1828 -0
- package/dist/index.js.map +1 -0
- package/docs/ADR-0001-extensible-kernel.md +41 -0
- package/docs/ADR-0002-profiles-and-context.md +22 -0
- package/docs/ADR-0003-doc-bridge-context-binding.md +36 -0
- package/docs/ADR-0004-run-metrics.md +29 -0
- package/docs/ADR-0005-benchmark-manifest.md +27 -0
- package/docs/ADR-0006-agent-session-protocol.md +36 -0
- package/docs/ADR-0007-policy-gate.md +33 -0
- package/docs/ADR-0008-runtime-executor.md +33 -0
- package/docs/ADR-0009-process-runtime-boundary.md +34 -0
- package/docs/ADR-0010-docker-sandbox-runtime.md +32 -0
- package/docs/ADR-0011-runtime-attestation.md +30 -0
- package/docs/ADR-0012-controlled-baseline-observations.md +27 -0
- package/docs/ADR-0013-honest-benchmark-comparability.md +27 -0
- package/docs/ADR-0014-criterion-level-benchmark-evidence.md +24 -0
- package/docs/ADR-0015-directional-benchmark-outcomes.md +23 -0
- package/docs/ADR-0016-baseline-evidence-digests.md +21 -0
- package/docs/ADR-0017-event-log-integrity.md +25 -0
- package/docs/ADR-0018-verification-projection-attestation.md +23 -0
- package/docs/ADR-0019-human-decision-attestation.md +27 -0
- package/docs/ADR-0020-terminal-reconciliation.md +26 -0
- package/docs/ADR-0021-event-lock-recovery.md +25 -0
- package/docs/ADR-0022-signed-evidence-bundle.md +27 -0
- package/docs/ADR-0023-safe-action-recovery.md +30 -0
- package/docs/ADR-0024-controlled-completion-metrics.md +27 -0
- package/docs/ADR-0025-ci-dogfood.md +22 -0
- package/docs/ADR-0026-ci-evidence-artifact.md +22 -0
- package/docs/ADR-0027-portable-evidence.md +19 -0
- package/docs/ADR-0028-effective-metrics.md +20 -0
- package/docs/ADR-0029-honest-ci-preparation.md +20 -0
- package/docs/ADR-0030-agentskit-os-benchmark-bridge.md +20 -0
- package/docs/ADR-0031-real-provider-baseline.md +18 -0
- package/docs/ADR-0032-harness-equivalent-benchmark.md +25 -0
- package/docs/ADR-0033-portable-agent-gate.md +25 -0
- package/docs/ADR-0034-measurement-quality-gates.md +25 -0
- package/docs/ADR-0035-reproducible-benchmark-samples.md +20 -0
- package/docs/ADR-0036-comparable-baseline-samples.md +20 -0
- package/docs/ADR-0037-replicated-baseline-collection.md +27 -0
- package/docs/ADR-0038-end-to-end-benchmark-boundary.md +28 -0
- package/docs/ADR-0039-artifact-and-protocol-metrics.md +39 -0
- package/docs/ADR-0040-benchmark-corpus-surfaces.md +32 -0
- package/package.json +68 -0
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# ADR-0001: Extensible kernel with append-only run events
|
|
2
|
+
|
|
3
|
+
- Status: accepted
|
|
4
|
+
- Date: 2026-08-29
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
The harness must work with different coding agents, repositories, browsers,
|
|
9
|
+
databases, documentation systems, and tracking providers without moving its
|
|
10
|
+
completion guarantees into optional integrations. A mutable summary alone is
|
|
11
|
+
also insufficient for diagnosing retries and stale evidence.
|
|
12
|
+
|
|
13
|
+
## Decision
|
|
14
|
+
|
|
15
|
+
Keep the contract, lifecycle transitions, evidence validation, source binding,
|
|
16
|
+
staleness rules, approval boundaries, and security checks in the kernel. Expose
|
|
17
|
+
an in-process, typed plugin registry for optional capabilities. Plugins declare
|
|
18
|
+
their API version and dependencies, contribute through named typed slots, and
|
|
19
|
+
are cleaned up deterministically.
|
|
20
|
+
|
|
21
|
+
Persist lifecycle facts as append-only NDJSON under each run. `run.json` remains
|
|
22
|
+
the compatible projection used by the current CLI; event records carry the run
|
|
23
|
+
ID, source revision, and contract hash so projections and diagnostics can be
|
|
24
|
+
checked against the same execution.
|
|
25
|
+
|
|
26
|
+
## Constraints
|
|
27
|
+
|
|
28
|
+
- No remote plugin installation, hot reload, marketplace, or distributed event
|
|
29
|
+
store is part of this decision.
|
|
30
|
+
- A plugin cannot approve a run or bypass kernel evidence validation.
|
|
31
|
+
- Event records are appended by the harness and rejected when malformed or out
|
|
32
|
+
of order.
|
|
33
|
+
- Doc Bridge, Playbook profiles, browser adapters, and SCM integrations are
|
|
34
|
+
consumers of this seam, not kernel dependencies.
|
|
35
|
+
|
|
36
|
+
## Consequences
|
|
37
|
+
|
|
38
|
+
The public API can grow by adding typed slots and event payloads while the
|
|
39
|
+
existing CLI and JSON projection remain stable. The current event log records
|
|
40
|
+
run creation and state transitions; check-level event types and remote stores
|
|
41
|
+
can be added when a measured diagnostic need justifies them.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# ADR-0002: Profiles and optional context providers
|
|
2
|
+
|
|
3
|
+
- Status: accepted
|
|
4
|
+
- Date: 2026-08-29
|
|
5
|
+
|
|
6
|
+
## Decision
|
|
7
|
+
|
|
8
|
+
Verification contracts may define named profiles. A selected profile resolves
|
|
9
|
+
its parent chain deterministically, then overlays budgets, cleanup roots, and
|
|
10
|
+
existing checks by ID before validation and contract hashing. Missing profiles,
|
|
11
|
+
cycles, and unknown check IDs fail closed.
|
|
12
|
+
|
|
13
|
+
Context retrieval is an optional typed plugin slot. A provider returns a query,
|
|
14
|
+
references, content provenance, and a snapshot hash. Doc Bridge and Playbook
|
|
15
|
+
adapters can use the seam without becoming kernel dependencies.
|
|
16
|
+
|
|
17
|
+
## Boundaries
|
|
18
|
+
|
|
19
|
+
Profiles cannot bypass required checks or human approval. Providers cannot
|
|
20
|
+
approve runs, alter evidence validation, or silently replace the frozen
|
|
21
|
+
contract. Remote provider discovery, credential management, and context cache
|
|
22
|
+
policy remain integration concerns.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# ADR-0003: Bind optional Doc Bridge context to a run
|
|
2
|
+
|
|
3
|
+
- Status: accepted
|
|
4
|
+
- Date: 2026-08-29
|
|
5
|
+
|
|
6
|
+
## Decision
|
|
7
|
+
|
|
8
|
+
The harness provides a dependency-free `createDocBridgeContextProvider` adapter
|
|
9
|
+
that reads the local `.doc-bridge/index.json` contract. It returns at most
|
|
10
|
+
eight deterministic references, carries the index `contentHash` as source
|
|
11
|
+
provenance, and computes a stable snapshot hash.
|
|
12
|
+
|
|
13
|
+
`planRun` accepts resolved context snapshots and freezes them into `run.json`
|
|
14
|
+
with a `contextHash`. The lifecycle log records one `context.attached` event per
|
|
15
|
+
snapshot. Resolution timestamps remain audit metadata and are excluded from the
|
|
16
|
+
reproducibility hash.
|
|
17
|
+
|
|
18
|
+
Shell-based agents use the same wire boundary with `context resolve` and
|
|
19
|
+
`plan --context-file`. A context file contains one snapshot or an array of
|
|
20
|
+
snapshots, allowing another provider to participate without a dynamic runtime
|
|
21
|
+
loader.
|
|
22
|
+
|
|
23
|
+
The file loader verifies the snapshot hash before planning. This makes the
|
|
24
|
+
portable handoff tamper-evident while leaving provider-specific source
|
|
25
|
+
provenance and authorization at the provider boundary.
|
|
26
|
+
|
|
27
|
+
The kernel applies the same verification to snapshots supplied directly to
|
|
28
|
+
`planRun`, so typed callers cannot bypass the integrity boundary.
|
|
29
|
+
|
|
30
|
+
## Boundaries
|
|
31
|
+
|
|
32
|
+
The adapter does not install or import Doc Bridge, access remote services, or
|
|
33
|
+
refresh context after planning. A caller owns query selection and must resolve
|
|
34
|
+
context before the contract is verified. Index schema validation and richer
|
|
35
|
+
semantic search remain responsibilities of Doc Bridge; malformed or missing
|
|
36
|
+
indexes fail the adapter closed.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# ADR-0004: Derive benchmark metrics from immutable run projections
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The harness needs to measure improvement across repeated development runs without
|
|
10
|
+
introducing a database or a remote telemetry service. Each run already stores its
|
|
11
|
+
source, contract, checks, evidence, lifecycle state, approval, retry lineage,
|
|
12
|
+
and duration.
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
Expose `benchmarkRuns(stateDir)` and the `ak-harness benchmark` command. They
|
|
17
|
+
read the historical `run.json` projections, sort by run ID, and emit a versioned
|
|
18
|
+
JSON report with per-run facts and aggregate rates for checks, outcomes, evidence,
|
|
19
|
+
approvals, retries, stale runs, and duration.
|
|
20
|
+
|
|
21
|
+
## Consequences
|
|
22
|
+
|
|
23
|
+
- Metrics are local, reproducible, portable, and usable without a service.
|
|
24
|
+
- Historical reports can be committed or exported by CI when the project wants a
|
|
25
|
+
trend record.
|
|
26
|
+
- The report measures harness execution and review friction; it does not claim
|
|
27
|
+
developer productivity or causality without a controlled task benchmark.
|
|
28
|
+
- A future remote metrics store can consume this versioned report without changing
|
|
29
|
+
the verification kernel.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# ADR-0005: Use explicit benchmark manifests and task bindings
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
Run IDs identify executions, not the task being measured. Comparing harness
|
|
10
|
+
performance requires a stable task corpus and an honest baseline. A missing
|
|
11
|
+
baseline must not be silently inferred from harness history.
|
|
12
|
+
|
|
13
|
+
## Decision
|
|
14
|
+
|
|
15
|
+
Add a versioned benchmark manifest containing task IDs, titles, acceptance
|
|
16
|
+
criteria, and optional baseline observations. Verification contracts may bind a
|
|
17
|
+
run to a suite and task ID. The benchmark report compares only bound harness runs
|
|
18
|
+
with explicit, non-`not-run` baseline observations and reports the rest as
|
|
19
|
+
non-comparable.
|
|
20
|
+
|
|
21
|
+
## Consequences
|
|
22
|
+
|
|
23
|
+
- Multiple runs can be grouped as attempts of one task.
|
|
24
|
+
- Baseline provenance is visible and reviewable.
|
|
25
|
+
- The first corpus can be committed without fabricating baseline data.
|
|
26
|
+
- Controlled task execution and baseline collection remain a later phase; this
|
|
27
|
+
schema provides the stable seam now.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# ADR-0006: Record agent sessions through a protocol-first seam
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The harness must make agent work observable without coupling the kernel to a
|
|
10
|
+
specific model, runtime, tool host, or provider. It also must not turn prompts,
|
|
11
|
+
tool arguments, results, or chain-of-thought into persisted application data.
|
|
12
|
+
|
|
13
|
+
## Decision
|
|
14
|
+
|
|
15
|
+
Expose a small `AgentAdapter` descriptor and `createSessionRecorder` API. The
|
|
16
|
+
recorder accepts hashes and metadata, writes correlated session, turn, and tool
|
|
17
|
+
events to the existing append-only run log, and rejects invalid local ordering.
|
|
18
|
+
It can start only from `IMPLEMENTING`; with `resume: true`, it reconstructs
|
|
19
|
+
unfinished actions from the event log and emits a `session.resumed` event. A
|
|
20
|
+
persisted `tool.execution.started` event marks an unfinished action as
|
|
21
|
+
ambiguous; resumption requires an explicit human `retry` or `abandon` decision
|
|
22
|
+
through `recoverTool` before the runtime can be entered again. It does not
|
|
23
|
+
execute tools or make policy decisions.
|
|
24
|
+
|
|
25
|
+
## Consequences
|
|
26
|
+
|
|
27
|
+
- Any coding agent can implement the descriptor without importing an agent SDK.
|
|
28
|
+
- Session evidence is reproducible and correlated by run, session, turn, and
|
|
29
|
+
action IDs.
|
|
30
|
+
- Raw content stays outside the harness event log.
|
|
31
|
+
- An interrupted process can resume pending approvals and actions without
|
|
32
|
+
replaying completed tools or bypassing the policy gate.
|
|
33
|
+
- An interrupted action that may have reached an external runtime is fail-closed
|
|
34
|
+
until a human explicitly chooses retry or abandonment.
|
|
35
|
+
- A later policy gate or runtime adapter can consume this protocol without
|
|
36
|
+
changing the verification kernel.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# ADR-0007: Gate tool actions with ordered deny-by-default policy
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The session protocol observes tool calls, but observation alone cannot prevent
|
|
10
|
+
an unsafe or out-of-scope action from reaching a runtime. The kernel needs a
|
|
11
|
+
portable control point that works before provider-specific tool execution.
|
|
12
|
+
|
|
13
|
+
## Decision
|
|
14
|
+
|
|
15
|
+
Require every `SessionRecorder` to receive a typed `PolicyGate`. The built-in
|
|
16
|
+
`createPolicyGate` evaluates ordered exact-tool rules, uses the first match,
|
|
17
|
+
and blocks by default when no rule allows the tool. Rules may allow, block, or
|
|
18
|
+
require human approval. Each decision is recorded; approval-required attempts
|
|
19
|
+
cannot enter the pending execution lifecycle until a human approves them.
|
|
20
|
+
|
|
21
|
+
The gate receives hashes and identifiers only. It does not execute tools,
|
|
22
|
+
inspect raw arguments, or approve the overall task.
|
|
23
|
+
|
|
24
|
+
## Consequences
|
|
25
|
+
|
|
26
|
+
- A tool cannot be requested through the public session API without a policy
|
|
27
|
+
decision.
|
|
28
|
+
- Sensitive tools can pause for an explicit human decision before execution.
|
|
29
|
+
- Policy behavior is deterministic, inspectable, and provider-independent.
|
|
30
|
+
- Rejected or unresolved approvals remain fail-closed and generate an audit
|
|
31
|
+
trail without storing raw prompts, arguments, or results.
|
|
32
|
+
- Existing integrations must provide an explicit allowlist or their tools are
|
|
33
|
+
blocked by default.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# ADR-0008: Execute allowed actions through a bounded runtime seam
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The policy gate can decide whether a tool action is allowed, but the session
|
|
10
|
+
protocol still needs a portable way to execute that action and close it with
|
|
11
|
+
reproducible evidence. Tool providers differ widely, so the kernel must not
|
|
12
|
+
depend on a shell, MCP host, model SDK, or container runtime.
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
Require every `SessionRecorder` to receive a `ToolRuntime`. The built-in
|
|
17
|
+
runtime dispatches to explicitly registered handlers, propagates an
|
|
18
|
+
`AbortSignal`, applies a timeout, hashes successful results, and returns
|
|
19
|
+
structured failures for missing tools, exceptions, and timeouts. The recorder
|
|
20
|
+
invokes it only after a policy allow decision and converts the result into the
|
|
21
|
+
existing `tool.completed` or `tool.failed` event.
|
|
22
|
+
|
|
23
|
+
The built-in runtime is in-process. It is an execution boundary, not a hard
|
|
24
|
+
security sandbox; isolated process or container runtimes remain provider
|
|
25
|
+
adapters for a later phase.
|
|
26
|
+
|
|
27
|
+
## Consequences
|
|
28
|
+
|
|
29
|
+
- Runtime handlers are portable and independently replaceable.
|
|
30
|
+
- Blocked actions cannot reach a handler through the public session API.
|
|
31
|
+
- Raw arguments and results remain in memory only; the event log stores hashes.
|
|
32
|
+
- Timeouts bound the recorder's wait, but a handler that ignores abort cannot be
|
|
33
|
+
force-killed by an in-process runtime.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# ADR-0009: Add a bounded shell-free process runtime
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
An in-process handler is useful for adapters, but it cannot be killed if it
|
|
10
|
+
ignores abort and is not an isolation boundary for untrusted code. The harness
|
|
11
|
+
needs a portable process seam for commands that require stronger lifecycle
|
|
12
|
+
control without pretending that every host has a container runtime.
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
Expose `createProcessToolRuntime`. Each tool has a fixed executable, argument
|
|
17
|
+
list, optional working directory, and explicit environment. The runtime uses
|
|
18
|
+
`spawn` with `shell: false`, sends one JSON request over stdin, kills timed-out
|
|
19
|
+
or oversized children, and returns only hashed stdout or structured failure
|
|
20
|
+
metadata.
|
|
21
|
+
|
|
22
|
+
Recovery closes the failed action and requires a fresh policy-approved action;
|
|
23
|
+
the harness does not automatically replay potentially side-effecting tools.
|
|
24
|
+
|
|
25
|
+
This is a bounded child-process boundary, not a complete security sandbox.
|
|
26
|
+
Container, OS policy, filesystem, network, and credential isolation remain
|
|
27
|
+
provider-specific responsibilities.
|
|
28
|
+
|
|
29
|
+
## Consequences
|
|
30
|
+
|
|
31
|
+
- Shell interpolation is removed from the portable process path.
|
|
32
|
+
- Timeouts and output limits prevent common runaway-process failures.
|
|
33
|
+
- Explicit environments reduce accidental secret inheritance.
|
|
34
|
+
- Hard isolation and resource quotas require a later sandbox provider.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# ADR-0010: Optional Docker sandbox runtime
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The process runtime bounds execution but does not isolate an agent tool from
|
|
10
|
+
the host. Coding tasks sometimes need a stronger boundary while the harness
|
|
11
|
+
must remain portable and must not require Docker for its core API.
|
|
12
|
+
|
|
13
|
+
## Decision
|
|
14
|
+
|
|
15
|
+
Expose `createDockerToolRuntime` as an optional provider built on the existing
|
|
16
|
+
process runtime. Each tool declares a fixed image and argv; the provider never
|
|
17
|
+
constructs or invokes a shell. By default it uses a cached image only, disables
|
|
18
|
+
container networking, makes the root filesystem read-only, drops Linux
|
|
19
|
+
capabilities, enables `no-new-privileges`, runs as an unprivileged UID, and
|
|
20
|
+
applies PIDs, memory, CPU, timeout, and output limits. Host mounts are absent
|
|
21
|
+
unless explicitly declared and default to read-only.
|
|
22
|
+
|
|
23
|
+
## Consequences
|
|
24
|
+
|
|
25
|
+
- Consumers without Docker can keep using the portable process or in-process
|
|
26
|
+
providers; Docker failures are structured runtime failures.
|
|
27
|
+
- A Docker image is a trust boundary input and should be pinned by consumers
|
|
28
|
+
when reproducibility matters. `pull: 'never'` prevents an implicit network
|
|
29
|
+
fetch by default.
|
|
30
|
+
- The provider is not a VM or a guarantee against a compromised Docker daemon.
|
|
31
|
+
Stronger VM, rootless, Windows, and remote sandbox providers can implement
|
|
32
|
+
the same `ToolRuntime` seam later without changing the kernel.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# ADR-0011: Runtime attestation in tool evidence
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
A sandbox result hash proves what the tool returned, but not which image or
|
|
10
|
+
limits produced it. Enterprise review needs to correlate terminal tool events
|
|
11
|
+
with the exact runtime profile used for that action.
|
|
12
|
+
|
|
13
|
+
## Decision
|
|
14
|
+
|
|
15
|
+
Runtime providers may attach typed runtime evidence to completed or failed tool
|
|
16
|
+
results. The Docker provider resolves the local image ID before execution,
|
|
17
|
+
hashes the effective profile plus that ID, and carries the attestation into the
|
|
18
|
+
terminal event. Missing images or an unavailable daemon fail before execution.
|
|
19
|
+
|
|
20
|
+
The evidence is descriptive and bound to the harness event's run, source
|
|
21
|
+
revision, and configuration hash; it is not a claim that the Docker daemon or
|
|
22
|
+
host kernel is trustworthy.
|
|
23
|
+
|
|
24
|
+
## Consequences
|
|
25
|
+
|
|
26
|
+
- Reviewers can identify the provider, image digest, security profile, and
|
|
27
|
+
resource limits for each action without storing raw command output.
|
|
28
|
+
- Custom runtimes remain compatible because runtime evidence is optional.
|
|
29
|
+
- Future VM or remote providers can publish their own typed evidence without
|
|
30
|
+
changing the session protocol.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# ADR-0012: Record controlled baseline observations through the public interface
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The harness already aggregates run history, but a benchmark comparison is only
|
|
6
|
+
meaningful when the baseline is real, attributable, and tied to the same task
|
|
7
|
+
identity. Manual JSON editing is easy to get wrong and makes the measurement
|
|
8
|
+
process hard to reproduce.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Expose `recordBenchmarkObservation` and `ak-harness benchmark baseline` as the
|
|
13
|
+
single supported write path for one baseline observation per task. The API
|
|
14
|
+
validates the manifest and observation, rejects unknown tasks and duplicates,
|
|
15
|
+
and writes through a temporary file followed by an atomic rename. It also
|
|
16
|
+
checks that the source manifest did not change during the operation.
|
|
17
|
+
|
|
18
|
+
`not-run` remains an explicit non-comparable status. The benchmark report must
|
|
19
|
+
not infer a baseline from harness history or report improvement without both a
|
|
20
|
+
valid baseline and a bound harness run.
|
|
21
|
+
|
|
22
|
+
## Consequences
|
|
23
|
+
|
|
24
|
+
- Baselines have a reproducible CLI/API entry point and visible provenance.
|
|
25
|
+
- Concurrent or stale writes fail instead of silently overwriting newer data.
|
|
26
|
+
- A controlled task run is still required; this API records evidence but does
|
|
27
|
+
not manufacture experimental results.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# ADR-0013: Require completed evidence for benchmark comparability
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
An explicit baseline is necessary but not sufficient for a meaningful
|
|
6
|
+
comparison. A blocked or still-running harness attempt must not be presented as
|
|
7
|
+
an achieved result, and aggregate pass rates alone do not explain why a task is
|
|
8
|
+
or is not comparable.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
`benchmarkRuns` reports a task as comparable only when it has a baseline whose
|
|
13
|
+
status is not `not-run` and the latest bound harness run is `COMPLETE`. Every
|
|
14
|
+
other case has a typed reason: missing baseline, not-run baseline, harness not
|
|
15
|
+
run, or harness not complete.
|
|
16
|
+
|
|
17
|
+
The comparison also exposes the latest harness check, outcome, evidence,
|
|
18
|
+
duration, attempt, and human-review metrics. Human-review duration is derived
|
|
19
|
+
only when approval is recorded. The report remains descriptive; it does not
|
|
20
|
+
claim causality or productivity improvement.
|
|
21
|
+
|
|
22
|
+
## Consequences
|
|
23
|
+
|
|
24
|
+
- A blocked or partial attempt cannot inflate the comparison dataset.
|
|
25
|
+
- Reviewers can distinguish missing evidence from failed execution.
|
|
26
|
+
- Future statistical analysis has stable inputs without changing the run
|
|
27
|
+
protocol.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# ADR-0014: Criterion-level benchmark evidence
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
Scalar baseline numbers do not prove that a benchmark task exercised its
|
|
6
|
+
acceptance criteria. Treating those numbers as comparable can make an
|
|
7
|
+
incomplete delivery look like an improvement.
|
|
8
|
+
|
|
9
|
+
## Decision
|
|
10
|
+
|
|
11
|
+
Baseline observations may include an `evidence` array. Each entry names one
|
|
12
|
+
acceptance criterion exactly, records its status, and points to a source. The
|
|
13
|
+
manifest validator rejects unknown or duplicate criteria and invalid evidence
|
|
14
|
+
metadata. A comparison is `comparable` only when every criterion is covered,
|
|
15
|
+
the baseline is not `not-run`, and the latest bound harness run is `COMPLETE`.
|
|
16
|
+
The report exposes `baselineEvidenceCoverageRate` and returns
|
|
17
|
+
`baseline-evidence-missing` when coverage is incomplete.
|
|
18
|
+
|
|
19
|
+
## Consequences
|
|
20
|
+
|
|
21
|
+
Baseline collection has a small additional input requirement, but benchmark
|
|
22
|
+
reports can distinguish missing proof from a real completed comparison. The
|
|
23
|
+
evidence source remains a reference; this phase does not claim causality or
|
|
24
|
+
replace criterion-level harness evidence for the current run.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# ADR-0015: Directional benchmark outcomes
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The harness already reports raw deltas, but a negative duration delta is easy
|
|
6
|
+
to misread and non-comparable tasks can be mistaken for missing data. Operators
|
|
7
|
+
need a small, explicit interpretation layer for the measured resources.
|
|
8
|
+
|
|
9
|
+
## Decision
|
|
10
|
+
|
|
11
|
+
For comparable tasks, report a rate for duration, attempts, and human review:
|
|
12
|
+
`(baseline - harness) / baseline`. Positive means the harness used less of the
|
|
13
|
+
resource and is labelled `improved`; negative is `regressed`; zero is
|
|
14
|
+
`unchanged`. A missing value, zero baseline, or non-comparable task is labelled
|
|
15
|
+
`unavailable`. These are per-resource signals, not a composite productivity
|
|
16
|
+
score or a causal claim.
|
|
17
|
+
|
|
18
|
+
## Consequences
|
|
19
|
+
|
|
20
|
+
Benchmark consumers can read the direction without reimplementing arithmetic,
|
|
21
|
+
while trade-offs remain visible instead of being hidden in one score. A real
|
|
22
|
+
baseline with complete criterion evidence is still required before any outcome
|
|
23
|
+
is reported as comparable.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# ADR-0016: Baseline evidence digests
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
Criterion coverage and directional metrics can still be detached from the
|
|
6
|
+
exact JSON input used during baseline collection. Without a digest, a later
|
|
7
|
+
review cannot distinguish the recorded evidence file from a changed copy.
|
|
8
|
+
|
|
9
|
+
## Decision
|
|
10
|
+
|
|
11
|
+
When `ak-harness benchmark baseline` reads `--evidence-file`, it computes and
|
|
12
|
+
stores the file's lowercase SHA-256 digest in `evidenceDigest`. Manifest
|
|
13
|
+
validation accepts only a 64-character lowercase SHA-256 value. The digest is
|
|
14
|
+
of the evidence file content, not of the external artifact named by each
|
|
15
|
+
criterion source.
|
|
16
|
+
|
|
17
|
+
## Consequences
|
|
18
|
+
|
|
19
|
+
The baseline record is tamper-evident with respect to its input file and can be
|
|
20
|
+
replayed from the same bytes. External systems remain an explicit integration
|
|
21
|
+
boundary; their content verification is not claimed by the local harness.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# ADR-0017: Hash chained lifecycle event logs
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The lifecycle log already preserves order and binds events to a run revision
|
|
6
|
+
and contract hash, but a later edit to an existing record would not be
|
|
7
|
+
detected. Enterprise review needs a deterministic integrity signal without
|
|
8
|
+
persisting prompts, tool arguments, or adding a remote service.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
New events include `previousHash` and `eventHash`. The hash covers the exact
|
|
13
|
+
serialized event body, including the preceding hash, so changing or reordering
|
|
14
|
+
records makes the chain invalid. `FileEventStore.read` validates the chain and
|
|
15
|
+
`FileEventStore.verify` exposes a typed `verified` or `legacy` result. Existing
|
|
16
|
+
logs without integrity fields remain readable and are never reported as
|
|
17
|
+
verified; appending to them preserves their legacy status.
|
|
18
|
+
|
|
19
|
+
## Consequences
|
|
20
|
+
|
|
21
|
+
The log is tamper-evident for new records and can be checked locally or through
|
|
22
|
+
`ak-harness events verify`. This is not a signature or external notarization:
|
|
23
|
+
an attacker able to rewrite both the log and its consumer can recompute the
|
|
24
|
+
chain. A future signed export can add that trust boundary without changing the
|
|
25
|
+
event lifecycle API.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# ADR-0018: Attest verification projections before approval
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
`run.json` is a convenient projection of checks and outcomes, but it is not
|
|
6
|
+
itself an immutable store. Approval must not trust a projection that was
|
|
7
|
+
changed after verification while the source and contract remain unchanged.
|
|
8
|
+
|
|
9
|
+
## Decision
|
|
10
|
+
|
|
11
|
+
After every verification pass, the harness hashes the checks, outcomes, and
|
|
12
|
+
metrics projection, persists that `verificationDigest` in `run.json`, and
|
|
13
|
+
appends a `verification.completed` event carrying the same digest. Human
|
|
14
|
+
approval recomputes the digest, verifies the audit log, and requires the latest
|
|
15
|
+
completion event to match it.
|
|
16
|
+
|
|
17
|
+
## Consequences
|
|
18
|
+
|
|
19
|
+
Manual edits to the verification projection are rejected before approval while
|
|
20
|
+
the existing source and contract freshness checks remain in force. The digest
|
|
21
|
+
protects the local projection through the audit log; it is not a digital
|
|
22
|
+
signature or external notarization. Dogfood uses the public API, and prior
|
|
23
|
+
Docker sandbox evidence remains covered by the earlier runtime phase.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# ADR-0019: Record human decisions as attested lifecycle events
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The verification result is now attested before approval, but the approval and
|
|
6
|
+
tracking authorization themselves only existed in the mutable `run.json`
|
|
7
|
+
projection. An audit consumer could see that a run reached `COMPLETE` without
|
|
8
|
+
an immutable record of which human decision accepted the exact evidence.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Record `approval.recorded` and `authorization.recorded` events in the existing
|
|
13
|
+
hash-chained lifecycle log. Each event contains the decision, resulting state,
|
|
14
|
+
verification digest, source revision, contract hash, and human actor; tracking
|
|
15
|
+
authorization also contains its declared target. Rejections are recorded too,
|
|
16
|
+
so every human terminal decision is auditable.
|
|
17
|
+
|
|
18
|
+
The existing freshness and verification-attestation checks run before either
|
|
19
|
+
decision is accepted. The `run.json` fields remain a convenient projection,
|
|
20
|
+
but the event is the audit record.
|
|
21
|
+
|
|
22
|
+
## Consequences
|
|
23
|
+
|
|
24
|
+
Consumers can independently prove which verified result was approved and which
|
|
25
|
+
external tracking target was authorized. Legacy logs remain readable and are
|
|
26
|
+
reported as legacy until they contain the new hashed event protocol. This is an
|
|
27
|
+
audit attestation, not a digital signature or external notarization.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# ADR-0020: Reconcile terminal projections before reporting completion
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The event log and human decision attestations protect the audit trail, but
|
|
6
|
+
`run.json` is still a mutable projection. Reading it directly after approval
|
|
7
|
+
could report `COMPLETE` even if its digest, decision projection, or decision
|
|
8
|
+
event had been removed or changed.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Expose `reconcileRun` in the public API and `ak-harness audit` in the CLI. The
|
|
13
|
+
reconciliation verifies the event hash chain, event-to-run binding, current
|
|
14
|
+
verification digest, and the required approval/authorization event and
|
|
15
|
+
projection for terminal states. `ak-harness status` uses this same gate.
|
|
16
|
+
|
|
17
|
+
Event appends use an atomic per-run lock and fail closed when another writer is
|
|
18
|
+
active, preventing concurrent lifecycle operations from duplicating a sequence
|
|
19
|
+
number or corrupting the append-only log.
|
|
20
|
+
|
|
21
|
+
## Consequences
|
|
22
|
+
|
|
23
|
+
Post-approval projection tampering and removal of a decision event fail closed
|
|
24
|
+
with a harness error. Historical non-terminal projections remain inspectable;
|
|
25
|
+
terminal runs from before the attestation protocol must be reverified rather
|
|
26
|
+
than being treated as enterprise-grade evidence.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# ADR-0021: Explicit recovery for orphaned event-log locks
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The per-run event-log lock prevents concurrent append and read operations from
|
|
6
|
+
duplicating sequence numbers or observing a partial write. A process can still
|
|
7
|
+
terminate after acquiring the lock, leaving a file that must not be removed
|
|
8
|
+
automatically because another writer may still own it.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Write the owning process ID and creation timestamp into every lock. Expose lock
|
|
13
|
+
inspection and an explicit `human`-authorized recovery operation. Recovery is
|
|
14
|
+
allowed only when the lock is older than the requested threshold and its owner
|
|
15
|
+
process is no longer alive. Malformed, young, or live locks remain fail-closed.
|
|
16
|
+
|
|
17
|
+
The CLI exposes this maintenance operation as `events unlock`; it never runs as
|
|
18
|
+
part of normal status, verification, or append flows.
|
|
19
|
+
|
|
20
|
+
## Consequences
|
|
21
|
+
|
|
22
|
+
An interrupted local run can be recovered without manually deleting an audit
|
|
23
|
+
artifact, while active or ambiguous locks cannot be silently bypassed. A lock
|
|
24
|
+
owner PID is local-process evidence; distributed writers still require a shared
|
|
25
|
+
coordination primitive outside this file store.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# ADR-0022: Signed portable evidence bundles
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
Hash chaining protects the local event log from accidental or casual edits,
|
|
6
|
+
but a reviewer or CI system needs a portable artifact whose contents and
|
|
7
|
+
provenance can be checked outside the originating workspace.
|
|
8
|
+
|
|
9
|
+
## Decision
|
|
10
|
+
|
|
11
|
+
`exportEvidenceBundle` packages a reconciled `COMPLETE` run, its projection,
|
|
12
|
+
event log, and referenced check outputs. Each file is hashed with SHA-256 and
|
|
13
|
+
the canonical bundle payload is signed with an Ed25519 private key. The key ID
|
|
14
|
+
and signature travel with the bundle; `verifyEvidenceBundle` independently
|
|
15
|
+
validates every file hash, the payload hash, and the signature. Callers can
|
|
16
|
+
additionally provide an explicit trust store to require an active key with the
|
|
17
|
+
expected identity instead of trusting the embedded public key.
|
|
18
|
+
|
|
19
|
+
The bundle contains check outputs as captured, so operators must treat it as a
|
|
20
|
+
potentially sensitive artifact and apply their existing retention controls.
|
|
21
|
+
|
|
22
|
+
## Consequences
|
|
23
|
+
|
|
24
|
+
Evidence can cross machine and CI boundaries without adding a remote service.
|
|
25
|
+
Key custody remains an operator responsibility; this is a signed artifact, not
|
|
26
|
+
a replacement for an enterprise KMS, certificate authority, or notarization
|
|
27
|
+
service. Trust-store entries support explicit rotation and revocation.
|