@agentskit/harness 0.1.0 → 0.4.0

package/docs/ADR-0027-portable-evidence.md

@@ -1,19 +0,0 @@
-# ADR-0027: Verify signed evidence outside the source workspace
-
-## Context
-
-An evidence artifact is only useful to CI reviewers and release systems if it
-does not depend on the original checkout. Existing bundle tests verified
-signatures and trust, but did not prove path independence.
-
-## Decision
-
-The CLI flow copies the signed bundle and trust store into a separate temporary
-directory and verifies them there. The bundle remains self-contained and the
-existing trusted-key model is unchanged.
-
-## Consequences
-
-The portability guarantee is exercised without a new transport or dependency.
-The probe remains local and deterministic; CI or a release system can upload
-the same bundle as an artifact when its completed run is available.

package/docs/ADR-0028-effective-metrics.md

@@ -1,20 +0,0 @@
-# ADR-0028: Separate effective delivery metrics from retry history
-
-## Context
-
-Aggregate pass rates include blocked or stale attempts that were later
-superseded. Those facts are useful for measuring friction, but they can make a
-successful current delivery look like a failed one.
-
-## Decision
-
-Keep historical attempt metrics unchanged and add `effective*` summary metrics
-computed from runs that are not superseded by another run in the same local
-history. Report effective run count, completion count/rate, and check,
-outcome, and evidence rates.
-
-## Consequences
-
-Reviewers can distinguish process friction from the current delivery state
-without losing retry history. The lineage rule is local and deterministic; a
-future remote aggregation system must preserve `supersedes` relationships.

package/docs/ADR-0029-honest-ci-preparation.md

@@ -1,20 +0,0 @@
-# ADR-0029: Separate CI preparation from human approval
-
-## Context
-
-The CI dogfood flow needs to create and verify a run, but CI cannot truthfully
-act as the human who approved the task contract. Recording `--by human` in CI
-would make the audit trail misleading.
-
-## Decision
-
-`plan prepared --by ci` creates a planned run with a `contractPreparation`
-record. It may proceed through verification and produce evidence, but only the
-existing human approval path can transition the run to `COMPLETE`. Human
-planning continues to use `plan approved --by human`.
-
-## Consequences
-
-CI evidence is honest and reviewable without weakening the completion gate.
-The run schema remains readable for older v1 records through the optional
-legacy `contractApproval` field.

package/docs/ADR-0030-agentskit-os-benchmark-bridge.md

@@ -1,20 +0,0 @@
-# ADR-0030: Provenance-bearing external coding benchmark bridge
-
-## Decision
-
-Keep the harness benchmark schema portable and add optional task provenance:
-the source repository, immutable revision, prompt path and digest, and read /
-write scope. Validate the existing AgentsKit OS coding benchmark report through
-a small dependency-free adapter.
-
-The adapter validates report integrity only. Provider status, completeness
-heuristics, and `successPassed` are observations; they do not grant human
-acceptance or make a benchmark comparable without a controlled baseline,
-criterion evidence, and a `COMPLETE` harness run.
-
-## Consequences
-
-The same manifest can be checked from another agent or repository, while the
-AgentsKit OS benchmark remains responsible for running providers. Phase 28
-ships the seed corpus and bridge. Real baseline collection across repeated,
-reviewed tasks is the next measurement step; no improvement claim is made here.

package/docs/ADR-0031-real-provider-baseline.md

@@ -1,18 +0,0 @@
-# ADR-0031: Real-provider baseline before efficacy claims
-
-## Decision
-
-Collect the baseline by running the existing AgentsKit OS coding benchmark on
-disposable, git-initialized copies of its fixture workspace. Record one report
-per manifest task with provider output, duration, validation command, criterion
-evidence, and the pinned source revision.
-
-The baseline runner is separate from verification. Verification validates
-recorded observations; it does not invoke a paid or nondeterministic provider on
-every harness run.
-
-## Measurement rule
-
-Baseline data is not comparable by itself. A directional improvement requires
-the same task, scope, provider conditions, acceptance evidence, and a completed
-harness run. Missing harness-equivalent runs remain `comparableTaskCount: 0`.

package/docs/ADR-0032-harness-equivalent-benchmark.md

@@ -1,25 +0,0 @@
-# ADR-0032: Harness-equivalent benchmark runs
-
-## Status
-
-Accepted for the benchmark pilot.
-
-## Decision
-
-Run the frozen benchmark corpus once per task with a task-specific harness
-contract and `benchmark: { suiteId, taskId, mode: "harness" }` binding. Keep each
-task's lifecycle state isolated so a blocked provider result cannot be mistaken
-for another task's approval. Collection delegates directional calculations to
-the existing `benchmarkRuns` implementation.
-
-The runner prepares and executes plans but never records human approval. A task
-is comparable only after the real checks pass and a human moves it to
-`COMPLETE`; pending and blocked runs remain visible and non-comparable.
-
-## Consequences
-
-- Baseline and harness runs share task identity and provenance.
-- Human approval remains an auditable gate rather than an automation shortcut.
-- The first collection may report no improvement when the provider is blocked;
-  that is a valid result, not missing data to be filled in.
-- Temporary untracked directories no longer crash source freshness checks.

package/docs/ADR-0033-portable-agent-gate.md

@@ -1,25 +0,0 @@
-# ADR-0033: Portable agent-side verification gate
-
-## Status
-
-Accepted for the benchmark pilot.
-
-## Decision
-
-Disposable coding-agent fixtures expose the built `ak-verify` command through a
-fixture-local wrapper and PATH entry. The fixture contract is prepared with
-`plan prepared --by ci` before the provider starts. The provider can therefore
-run the same real verification command from inside its working directory.
-
-The wrapper is a convenience for provider-side feedback, not the final trust
-boundary. The outer harness still validates the provider report and acceptance
-evidence, and only a human can move an outer run to `COMPLETE`.
-
-## Consequences
-
-- Missing `ak-verify` is observable as an environment/setup failure instead of
-  an unexplained provider partial result.
-- No human decision is fabricated inside the benchmark fixture.
-- The wrapper executes the pinned local harness build; it does not install
-  dependencies or grant network access.
-- Provider status and artifact validation remain separate signals.

package/docs/ADR-0034-measurement-quality-gates.md

@@ -1,25 +0,0 @@
-# ADR-0034: Make benchmark improvement claims fail closed
-
-## Context
-
-A benchmark can complete every acceptance criterion while still costing more
-time or retries. Aggregate pass rates alone hide that trade-off, and a small
-sample cannot support a reliable general claim.
-
-## Decision
-
-Benchmark manifests may declare a policy with `minComparableTasks`,
-`maxDurationRegressionRate`, `minCompletedRunsPerTask`, and
-`requireZeroEscapedIncomplete`.
-
-Reports expose `retryCount`, `completedRuns`, per-task confidence, and a
-`qualityGate`. The gate is `insufficient-data` when the task sample is too
-small, `failed` when policy detects a regression, and `passed` only when all
-configured quality conditions hold. Historical retries remain visible but do
-not make a superseded run the effective result.
-
-## Consequences
-
-The harness shows quality/cost trade-offs instead of claiming that every
-improvement is positive. Directional results support iteration; reliable
-samples are required for release claims.

package/docs/ADR-0035-reproducible-benchmark-samples.md

@@ -1,20 +0,0 @@
-# ADR-0035: Use independent samples for benchmark confidence
-
-## Context
-
-A single provider run is too noisy to support a reliable performance claim.
-Counting repeated measurements as retries also exaggerates delivery cost.
-
-## Decision
-
-The benchmark runner accepts `--repeats N` and creates an isolated contract and
-state directory for each sample. Reports count `COMPLETE` samples toward
-confidence, compare duration using their median, and count retries only from
-superseded run lineages.
-
-## Consequences
-
-Three or more completed samples can reach reliable confidence when the manifest
-policy requires it. Human approval remains required for every sample. A median
-reduces the influence of one slow provider run but does not prove causality or
-productivity improvement.

package/docs/ADR-0036-comparable-baseline-samples.md

@@ -1,20 +0,0 @@
-# ADR-0036: Require comparable baseline samples
-
-## Context
-
-Comparing three harness samples with one baseline observation makes the
-performance result look more certain than it is. Provider latency can vary
-between runs, so the baseline needs the same replication discipline.
-
-## Decision
-
-Benchmark policy includes `minBaselineSamplesPerTask`. Baseline observations
-can carry `durationSamplesMs`; reports use their median and keep a legacy
-single `durationMs` as one sample. A task is non-comparable until both the
-baseline and harness meet their configured sample minimums.
-
-## Consequences
-
-The quality gate becomes `insufficient-data` until a fair baseline exists.
-This delays performance claims but prevents a replicated harness result from
-being presented as a reliable improvement against a one-off baseline.

package/docs/ADR-0037-replicated-baseline-collection.md

@@ -1,27 +0,0 @@
-# ADR-0037: Replicated baseline collection
-
-## Decision
-
-The AgentsKit OS baseline runner accepts `--repeats N` and executes every task
-in an independent disposable fixture. It records the measured durations as
-`durationSamplesMs`, keeps the task failed when any sample fails, and can
-replace the complete baseline observation set with `--record-manifest`.
-
-The replacement is staged through the existing typed
-`recordBenchmarkObservation` API and committed with one atomic rename. A
-partial task selection cannot update a complete manifest. The benchmark report
-then compares the median baseline duration with the median harness duration.
-
-## Rationale
-
-A single baseline run is not a fair comparator for three harness replicas.
-Independent baseline samples reduce sensitivity to provider and machine
-variance without misclassifying replicas as retries. Atomic recording prevents
-an interrupted collection from leaving a partially refreshed manifest.
-
-## Limits
-
-This controls sample count and provenance, not all sources of variance. The
-same provider, task corpus, pinned source revision, and comparable runtime
-should be used. Three samples are the current minimum policy, not a universal
-statistical guarantee; larger studies belong in a later benchmark program.

package/docs/ADR-0038-end-to-end-benchmark-boundary.md

@@ -1,28 +0,0 @@
-# ADR-0038: Use an end-to-end benchmark boundary
-
-## Decision
-
-Baseline samples measure elapsed time from disposable fixture setup through
-provider execution, artifact validation, and the end of the sample. Harness
-comparisons continue to use the elapsed duration of the configured real check.
-The provider-reported duration is retained as diagnostic data, not as the
-comparable duration.
-
-Baseline duration is comparable only when the baseline status is `passed` and
-every criterion evidence entry is `passed`. Escape reduction is reported
-independently, so an incomplete baseline cannot create a false duration or
-resource improvement while still exposing the observed delivery gap.
-
-## Rationale
-
-Comparing provider-only baseline time with the harness check time mixes two
-different scopes and can manufacture a regression. A common boundary measures
-the delivery workflow the human actually waits for. It also exposes whether a
-slow result comes from the provider or from validation overhead.
-
-## Limits
-
-The boundary does not remove provider, machine, network, or human-approval
-variance. Independent samples and median aggregation remain required; deeper
-latency decomposition is a later optimization if the comparable end-to-end
-measurement still regresses.

package/docs/ADR-0039-artifact-and-protocol-metrics.md

@@ -1,39 +0,0 @@
-# ADR-0039: Separate artifact acceptance from protocol completion
-
-## Decision
-
-Repeated external benchmark samples record `artifactAcceptanceRate`: the
-fraction whose task-specific artifact validation passed. The observation
-`status` remains protocol-aware and only qualifies for comparable duration or
-resource metrics when the baseline delivery and criterion evidence are
-complete.
-
-Harness runs may expose the same metric through structured check evidence. The
-benchmark report aggregates the available run-level rates and reports their
-sample count; missing evidence is not converted to an acceptance failure.
-
-Baseline observations also record `protocolCompletionRate`: the fraction of
-samples whose provider completed the verification protocol. Harness reports
-expose the corresponding per-task rate and sample count. This makes protocol
-reliability measurable without treating an incomplete baseline as comparable.
-
-Relative improvement remains unavailable for a zero baseline. The comparison
-also records an absolute delta and derives its direction from that delta, so
-zero-to-positive outcomes remain visible without invalid division.
-
-## Rationale
-
-An agent can produce a correct artifact while failing to provide the required
-verification protocol, for example because the verifier is unavailable. That
-is valuable evidence about task outcome and protocol reliability, but treating
-it as a complete baseline would make timing and resource comparisons unsafe.
-Keeping the dimensions separate makes the failure visible without converting
-partial evidence into a completion claim.
-
-## Limits
-
-Artifact acceptance is only as strong as the real task validator supplied by
-the benchmark. It does not prove human approval, protocol compliance, or
-enterprise readiness. A future benchmark schema may add criterion-level sample
-rates when the corpus needs per-criterion variance rather than one aggregate
-rate.

package/docs/ADR-0040-benchmark-corpus-surfaces.md

@@ -1,32 +0,0 @@
-# ADR-0040: benchmark corpus surface coverage
-
-## Status
-
-Accepted — 2026-08-31
-
-## Context
-
-The AgentsKit OS bridge initially measured three real coding tasks, all backed
-by a small logic fixture. That was enough to prove the protocol binding, but it
-could not show that the harness preserved different delivery surfaces.
-
-## Decision
-
-Benchmark tasks may declare a typed `surfaces` list using the same surface names
-as the verification contract. The phase-45 corpus remains anchored to the
-AgentsKit OS benchmark definition and adds executable CLI and documentation
-tasks. The runner validates the CLI through real child-process invocations and
-validates the documentation contract against the resulting README.
-
-The surface list is descriptive metadata. It does not make a task comparable,
-complete, or human-approved; those claims still require the normal lifecycle,
-criterion evidence, and current-source verification.
-
-## Consequences
-
-- Corpus validators can require coverage of named surfaces without guessing
-  from file paths.
-- The bridge now exercises logic, CLI, and documentation delivery shapes.
-- Endpoint, database, MCP, and UI tasks remain future corpus additions and must
-  bring their real runtime checks when introduced.
-- No benchmark performance improvement is inferred from corpus expansion alone.