@agentskit/harness 0.1.0 → 0.4.0

package/README.md

@@ -1,6 +1,12 @@
+---
+docbridge:
+  covers:
+    - package:@agentskit/harness
+---
+
 # @agentskit/harness
 
-Portable, evidence-backed development protocol for coding agents. The harness freezes a human-approved task contract, executes every configured check, binds evidence to the current source revision, detects stale results, and refuses completion without human approval.
+Portable, evidence-backed development protocol for coding agents. The harness freezes a task contract, executes every configured check, binds evidence to the current source revision, detects stale results, and applies the configured controlled or YOLO approval policy.
 
 ## Install
 
@@ -8,6 +14,10 @@ Portable, evidence-backed development protocol for coding agents. The harness fr
 pnpm add -D @agentskit/harness
 ```
 
+New consumers can run [`examples/minimum-profile.mjs`](examples/minimum-profile.mjs)
+after `pnpm build`; the walkthrough is in [`docs/GETTING-STARTED.md`](docs/GETTING-STARTED.md).
+Common gate and runtime failures are documented in [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md).
+
 The package requires Node.js 22 or newer and exposes both `ak-harness` and the common-protocol alias `ak-verify`.
 
 ## Workflow
@@ -24,79 +34,7 @@ ak-harness benchmark --manifest benchmarks/harness-phase-0.json --json
 
 `plan` rejects unresolved ambiguities and unauthorized dirty worktrees. After `start`, the contract is frozen. Any source, configuration, or contract change invalidates evidence and moves the run to `STALE`. A human can cancel an active run; retrying a blocked, stale, or cancelled run marks the previous run `SUPERSEDED`.
 
-Automated CI may prepare a run with `ak-harness plan prepared --by ci`. This records a CI preparation, never a human approval; the run remains unable to become `COMPLETE` until a human approves the verified result.
-
-## External coding benchmark bridge
-
-The harness validates a provenance-bearing task manifest and the JSON report
-emitted by an external coding benchmark, including AgentsKit OS:
-
-```bash
-AGENTSKIT_OS_ROOT=/path/to/agentskit-os \
-AGENTSKIT_OS_BENCHMARK_REPORT=/path/to/report.json \
-node scripts/verify-harness-agentskit-os-benchmark.mjs \
-  --manifest benchmarks/agentskit-os-phase-28.json \
-  --target "$AGENTSKIT_OS_ROOT" \
-  --report "$AGENTSKIT_OS_BENCHMARK_REPORT"
-```
-
-The bridge checks the pinned source revision, task definition, prompt digests,
-scope, and provider report shape. Provider status and heuristic scores remain
-observations: they do not grant human acceptance. With no controlled baseline,
-improvement is reported as unavailable rather than inferred.
-
-The phase-45 corpus adds explicit task surfaces and keeps the task definition
-bound to the real AgentsKit OS fixture. It covers logic, CLI, and documentation
-surfaces; the added CLI task executes compact, pretty, and invalid-input paths,
-while the documentation task checks the API contract and its reproducible test
-command:
-
-```bash
-node scripts/verify-harness-agentskit-os-benchmark.mjs \
-  --manifest benchmarks/agentskit-os-phase-45.json \
-  --target /path/to/agentskit-os \
-  --require-surface-coverage logic,cli,docs
-```
-
-Corpus coverage is a readiness gate, not a performance claim. Live provider
-samples and human approval remain required before the expanded tasks become
-comparable benchmark evidence.
-
-Phase 29 adds a reproducible real-provider baseline runner for the AgentsKit OS
-seed corpus. It uses disposable fixture copies, the existing OS benchmark
-runner, and task-specific validation:
-
-```bash
-node scripts/run-agentskit-os-baseline.mjs \
-  --target /path/to/agentskit-os \
-  --provider codex \
-  --output benchmarks/agentskit-os-phase-29-baseline
-```
-
-The observations are not an improvement claim until equivalent harness runs
-exist.
-
-To collect equivalent baseline samples, run the same corpus independently and
-record the aggregate only after every task has been sampled:
-
-```bash
-node scripts/run-agentskit-os-baseline.mjs \
-  --target /path/to/agentskit-os \
-  --provider codex \
-  --repeats 3 \
-  --output benchmarks/agentskit-os-phase-36-baseline \
-  --record-manifest benchmarks/agentskit-os-phase-28.json
-```
-
-Each repeat uses a fresh disposable fixture. The report stores the end-to-end
-duration of fixture setup, provider execution, validation, and cleanup in
-`durationSamplesMs`; the provider's own duration remains nested in the raw
-report. The manifest is replaced atomically through
-the typed observation recorder. `--record-manifest` requires the complete task
-set; omit it to inspect an uncommitted collection. Replicas are not retries,
-and a failed sample keeps the aggregate failed. Escape reduction may remain
-visible as a separate outcome, but duration, attempt, and review improvements
-remain unavailable until the baseline proves complete delivery.
+Current-source evidence requires a committed Git `HEAD`. A directory outside Git, or a repository without its first commit, is rejected as `GIT_REQUIRED`; it is not a supported pilot mode because it cannot prove revision currentness.
 
 ## Contract
 
@@ -110,7 +48,7 @@ Endpoint, database, CLI, MCP, and UI checks must declare `execution: "real"`. UI
 
 ## API
 
-The public TypeScript API is exported from `src/index.ts` and includes configuration loading, lifecycle operations, state transitions, evidence verification, approvals, cancellation, retries, and task-owned cleanup. Internal modules are not part of the supported API.
+The public TypeScript API is exported from `src/index.ts` and includes configuration loading, lifecycle operations, state transitions, evidence verification, approvals, cancellation, retries, task-owned cleanup, versioned capability manifests, event-envelope validation, deterministic phase execution, and stable error classification. Internal modules are not part of the supported API. The checked-in [capability manifest](./capabilities/public-surface.json) is generated from this entry point; run `pnpm test:capabilities` to detect drift.
 
 ## Extensibility
 
@@ -151,6 +89,36 @@ the current state, so a post-approval edit cannot appear as `COMPLETE`.
 Concurrent event writers are serialized by an atomic per-run lock and fail
 closed if the log is busy.
 
+Structured plans, findings, decisions, repairs, blockers, approvals, and phase
+results can be persisted as provenance-bound `ArtifactEnvelope` records. Each
+artifact has a version, run/issue/source/contract/config/context hashes, a
+content digest, and both JSON and Markdown representations. `FileArtifactStore`
+is idempotent: retrying the same write does not duplicate the event-log record.
+Use `resumeStateFromArtifacts` to rebuild completed phase outputs after an
+interruption, and inspect records with `ak-harness artifacts inspect <path>` or
+`ak-harness artifacts list [run-id]`.
+
+The legacy event-log record remains schema version 1 for compatibility. New
+provider-neutral integrations can exchange the schema-versioned v2
+`HarnessEventEnvelope`, which requires event identity, correlation, source
+revision, idempotency, and provenance metadata. `classifyHarnessError` maps
+stable Harness error codes to `retry`, `block`, or `escalate` dispositions.
+
+Replaceable integrations use the shared `AdapterMetadata` contract: every
+adapter declares an assurance level (`unverified`, `contract-tested`, or
+`runtime-attested`) and measured/unknown telemetry. Coding agents return
+structured output, diff, usage, timeout/cancellation status, and failure
+classification; Doc Bridge reports relevance and context cost; Orca exposes
+lease/lock/worktree/SHA projections; and tracking adapters deduplicate effects
+by idempotency key (with a dry-run mode).
+
+Each harness event may also carry an optional `correlation` envelope. Its
+`operationId` is the stable identity used when a lifecycle crosses into
+AgentsKit, Chat, Doc Bridge, or Code Review; the optional `runId`, `sessionId`,
+`turnId`, `actionId`, and `traceId` remain local identities. The envelope is
+bounded metadata only and never contains prompts, arguments, results, or
+secrets.
+
 Export a reconciled `COMPLETE` run for external review with an Ed25519 key:
 
 ```bash
@@ -162,9 +130,6 @@ The bundle includes the run projection, event log, and referenced check outputs,
 each with a SHA-256 digest. A trust store can mark keys `active` or `revoked` to
 support controlled key rotation. Treat exported outputs as potentially sensitive.
 
-The signed bundle is self-contained: it can be copied to an isolated directory
-and verified there with only the bundle and the trusted public key.
-
 Profiles are optional declarative overlays in `.codex/verification.json`. They
 inherit in order, override existing checks by ID, and are resolved before the
 contract is frozen:
@@ -181,6 +146,65 @@ contract is frozen:
 }
 ```
 
+`runtime.kind` chooses the executor used by an integration: `process` is a bounded shell-free local child process; `docker` adds the Docker sandbox. The choice is frozen in the resolved contract and therefore changes its hash. Docker remains fail-closed when its daemon or image is unavailable.
+
+`autonomy: "yolo"` removes the generic final review only after every applicable check passes, tracking is disabled, and the frozen contract has no ambiguity. It never auto-approves a material decision, external tracking, or a tool rule that requires approval.
+
+The phase executor applies the same rule to a declarative SDLC profile. A profile
+declares dependencies, inputs/outputs, gates, bounded retries, budgets, and an
+effect class (`read`, `write`, or `external`). `safe`, `yolo`, and `dry-run`
+profiles share the engine; only the effect policy changes:
+
+```ts
+const profile = createPhaseProfile({
+  id: 'feature', mode: 'yolo',
+  phases: [
+    { id: 'discover', outputs: ['plan'], effect: 'read' },
+    { id: 'implement', inputs: ['plan'], dependsOn: ['discover'], effect: 'write' },
+  ],
+})
+const result = await executePhaseProfile(profile, {
+  preflight: grillMeAndPreflight,
+  handlers: { discover, implement },
+})
+```
+
+Preflight runs for all mutating phases before any effect. Material ambiguities
+are returned as one structured decision packet; dry-run previews mutating phases
+without invoking their handlers. `planPhaseProfile` exposes the deterministic
+route without executing it.
+
+`runAdversarialReview` executes independent review lenses with bounded
+concurrency/retries and blocks empty or non-reproducible verdicts. Delivery
+helpers hash-bind the approved PR body/metadata and only emit a QA transition
+after feature validation and G5 acceptance; failed QA returns to verification.
+
+`createQualityMatrix` aggregates phase evidence, outcomes, duration, token/cache,
+machine, and concurrency signals into bounded 0–100 dimensions with baseline
+deltas. Missing measurements remain `unknown`; `evaluateWatchdog` emits typed
+budget/resource/contention blockers instead of treating absent data as success.
+
+Use named profiles to make the operational choice explicit:
+
+```json
+{
+  "profile": "process",
+  "runtime": { "kind": "process" },
+  "profiles": {
+    "process": { "runtime": { "kind": "process" } },
+    "docker": { "runtime": { "kind": "docker" } }
+  }
+}
+```
+
+```ts
+const runtime = createConfiguredToolRuntime({
+  runtime: loaded.config.runtime,
+  process: { tools: processTools },
+  docker: { tools: dockerTools },
+})
+```
+
 Doc Bridge and Playbook integrations can implement `ContextProvider` and
 register it through `CONTEXT_PROVIDER_SLOT`; the kernel records neither their
 credentials nor their transport and does not depend on either package. The
@@ -215,6 +239,159 @@ stable `contextHash`; resolution time is metadata and does not change the
 reproducibility hash. Context is resolved before planning and is frozen with
 the run, so later index changes cannot silently change its evidence.
 
+## Delivery gates
+
+The optional delivery helpers evaluate gates without embedding a GitHub, Linear,
+or deployment provider. Adapters perform external effects only after the
+deterministic decision is recorded:
+
+```ts
+const g2 = assessPreflight({
+  criteria,
+  implementerId: 'implementer',
+  reviewerId: 'independent-reviewer',
+  reviewKind: 'adversarial',
+  reviewApproved: true,
+})
+```
+
+G2 ignores later-gate pending criteria but blocks failed/pending G2 evidence,
+self-review, and a third repair. `composePullRequest` creates a structured PR
+body only from an approved, current G2 result; it reuses a confirmed matching
+remote PR and preserves uncertain state. G3 binds CI to the candidate revision.
+G4 requires an approved environment profile, identified artifact, isolation or
+version-bound acceptance, technical evidence, and a 15-minute low-risk window.
+G5 remains `awaiting-acceptance` until the applicable business/UX decision is
+recorded. `assessWorktreeCleanup` permits cleanup only after remote branch SHA,
+PR, and G3 all match.
+
+## Pilot cohort
+
+`ak-harness pilot cohort.json` validates the frozen cohort before work starts.
+It requires a policy hash, a baseline reference, exactly ten included issues,
+and the `normal` classification for each. Excluded or aborted issues need a
+reason and cannot be silently replaced in the same manifest.
+
+## Improvement cycle
+
+The five pilot steps can be evaluated as a bounded cycle: adversarial review,
+G2 preflight, baseline recording, pilot execution, and harness/no-harness
+comparison. Each iteration must contain those steps in order. A failed,
+blocked, or pending step requires a reason; repeating requires an explicit
+adjustment. The assessment returns `complete`, `repeat` (with the next
+iteration), or `blocked` when the adjustment is missing or the iteration
+budget is exhausted, plus a criterion-level matrix:
+
+```bash
+ak-harness cycle assess cycle.json --json
+```
+
+This is a deterministic decision helper: Orca, Emdash, GitHub, Linear, and
+other adapters remain responsible for executing external actions and supplying
+their structured results.
+
+## Agent optimization and evaluation
+
+The Harness also validates the optimization layer without owning a provider.
+The memory boundary accepts only approved records with explicit scope, source
+revision, and content hash; an adapter can back it with AgentsKit memory.
+`runAgentEval` runs a bounded suite and returns criterion-level accuracy;
+`createLlmCache` provides deterministic keys and hit/miss/invalidation evidence
+for context and read-only calls; and `runWorkflow` executes independent nodes
+in sorted, bounded fan-out/fan-in batches while serializing nodes that share a
+`mutationKey`. `OptimizationObservation` carries
+optional token, memory, cache, and parallelism measurements and refuses
+incomparable provider/model/configuration bindings.
+
+`evals/manifest.json` is the versioned evaluation battery. `validateEvalManifest`
+requires contract, deterministic, integration, quality, regression, and
+resource layers plus coverage for every supported component. `runEvalBattery`
+repeats each case and reports min/median/max scores; unknown, stale, critical,
+subjective, or unapproved regression results block the gate.
+
+`compatibility/manifest.json` pins the AgentsKit ecosystem revisions and the
+upstream test/eval commands. `assessCompatibility` accepts only complete,
+evidence-bound real-adapter observations and blocks unknown or failed upstream
+results; migration and rollback procedures are kept beside the manifest.
+
+These are seams, not replacements for AgentsKit packages. An integration may
+adapt `@agentskit/memory` and `@agentskit/eval` into them while keeping the
+Harness provider-neutral. Missing measurements remain missing; they are never
+reported as zero.
+
+## Discovery gate
+
+Discovery is a small deterministic gate before implementation. An adapter or
+agent supplies a structured list of ambiguities; the Harness does not decide
+product questions. Material ambiguities produce one decision packet with
+options and a recommendation. A non-material ambiguity can proceed only when
+an approved policy assumption covers it, and the result records that policy in
+the decision log.
+
+```bash
+ak-harness discovery assess discovery.json --json
+```
+
+The result is `ready` or `awaiting-decision`, and carries the source revision,
+contract hash, context hash, decision log and digest. Recheck it before
+implementation with `isDiscoveryCurrent`: any source, contract or context
+change makes the earlier result stale. The input is intentionally portable so
+Linear, Orca and Emdash adapters can produce it later without becoming kernel
+dependencies.
+
+## WIP admission
+
+Use the same portable approach to decide whether a new issue can start. The
+default limit is three deliveries started and not terminal. `blocked`,
+`awaiting-decision` and `awaiting-acceptance` still consume a delivery slot;
+they release an executor but do not hide unfinished work. Resuming an existing
+non-terminal issue keeps its reservation and takes priority over new work.
+
+```bash
+ak-harness wip assess wip.json --json
+```
+
+## Runtime experiment
+
+Compare Orca and Emdash only when both records carry the same source revision, contract, provider, model, and configuration hash. A failed hard gate is ineligible; the remaining candidates are ordered by human minutes, duration, cost, then `orca` only as the final tie-break.
+
+```sh
+ak-harness experiment select experiment.json --json
+```
+
+The Harness only assesses the supplied ledger. A future Linear adapter owns
+reading and writing the tracker; it must persist the ledger/recovery identity
+and use the existing event-log lock before acting.
+
+## Portable orchestration controls
+
+The package includes the small controls needed by an external orchestrator without
+embedding a tracker or provider:
+
+```ts
+import { createDispatchLedger, createOrcaDispatchPlan, planFilePreflight, runWithRecovery } from '@agentskit/harness'
+
+const ledger = createDispatchLedger('.codex/verification')
+const claim = ledger.claim({ tracker: 'linear', repository: 'org/repo', issue: 'ENG-1', worktree: 'eng-1', branch: 'codex/eng-1', owner: 'agent' })
+const dispatch = createOrcaDispatchPlan({ repository: 'org/repo', worktree: 'eng-1', branch: 'codex/eng-1', baseBranch: 'main', goalFile: 'GOAL.md' })
+ledger.recordDispatch({ lease: claim.lease, idempotencyKey: dispatch.idempotencyKey, commandDigest: dispatch.commandDigest })
+```
+
+Claims are keyed by tracker, repository, issue, worktree, and branch. They are
+atomic, idempotent, and recoverable only by a human. The ledger never executes
+the command; an Orca adapter may execute the returned argv after recording the
+decision.
+
+`planFilePreflight` skips documentation-only changes, selects colocated tests,
+and `validateSafeCommand` rejects shell composition. `runWithRecovery` retries
+only classified retryable failures with a bounded exponential delay and an
+abortable watchdog. `parseRetro` produces proposed learnings; only a human can
+promote them. `createStatusSnapshot` creates a digest-bound status projection.
+
+Linear/GitHub and Orca integrations should implement the provider-neutral
+tracking and dispatch adapters; no credentials or network clients belong in
+the kernel.
+
 Agent sessions can record adapter identity, turns, and guarded tool actions
 during `IMPLEMENTING` without persisting prompt, argument, or result contents:
 
@@ -338,18 +515,11 @@ action without storing raw output.
 
 `benchmark` aggregates the local run history into a versioned JSON report. It
 includes check/outcome/evidence pass rates, retries, stale runs, human approvals,
-and average/median verification duration. It also reports `effective*` metrics
-over the latest non-superseded run in each retry lineage, so attempt history is
-not confused with the current delivery state. With `--manifest`, it also compares
+and average/median verification duration. With `--manifest`, it also compares
 bound harness tasks with explicitly recorded baseline observations. A baseline
 must include evidence for every acceptance criterion; missing, duplicate, or
 unknown criterion evidence is rejected. Missing baselines and incomplete
-evidence remain non-comparable; the harness never invents a baseline. Baseline
-collectors also record `baselineArtifactAcceptanceRate`, the fraction of
-repeated samples whose real artifact validation passed. This is independent of
-protocol completion: a correct artifact with missing or incomplete protocol
-evidence is visible as an accepted artifact but remains non-comparable for
-duration and resource claims. These are
+evidence remain non-comparable; the harness never invents a baseline. These are
 execution metrics, not a claim of productivity improvement; compare reports over
 a controlled task corpus to measure that outcome.
 
@@ -400,73 +570,35 @@ the report exposes their delta and direction. A positive improvement rate
 means the harness used less of that measured resource; these metrics do not
 establish causality or productivity improvement alone.
 
-When a structured check includes `artifactAcceptanceRate` directly or inside
-its benchmark `reports`, comparisons also expose the harness rate, contributing
-sample count, and acceptance improvement. Missing rates remain uncounted and
-never become a failed artifact implicitly.
-
-Baseline and harness comparisons also expose `protocolCompletionRate`, which
-measures whether the verification protocol completed independently of artifact
-acceptance. A protocol rate does not override the fail-closed comparability
-rules.
-
-When a baseline rate is zero, relative improvement remains `unavailable`; the
-report still exposes the absolute acceptance or protocol completion delta.
-
-For paired real-provider measurements, `scripts/run-agentskit-os-harness-benchmark.mjs`
-prepares one frozen contract per manifest task, binds each run to the suite and
-task ID, and leaves the lifecycle at the human approval gate. Use
-`--repeats 3` for three independent samples per task; each sample has its own
-state directory and must be approved separately. Run `--collect` after
-approvals to aggregate the isolated task state directories. Pending or blocked
-task runs remain non-comparable; only `COMPLETE` runs produce directional
-improvement values. Duration comparisons use the median of completed samples,
-while retry metrics count only superseded retry lineages, so experimental
-replication is not misreported as agent retry cost.
-
-When one sample is blocked or times out, prepare a distinct replacement without
-overwriting the original evidence:
+## Repository organization
 
-```bash
-node scripts/run-agentskit-os-harness-benchmark.mjs \
-  --execute --task-id feat-formatter --sample 4 \
-  --phase-root .codex/verification/phase-39/harness
-```
-
-`--sample` requires a positive sample number and refuses an existing config or
-state directory. The original blocked run remains part of the audit trail;
-replacement runs add evidence and do not retroactively turn the blocked sample
-into success.
-
-The AgentsKit OS benchmark runner can expose the built `ak-verify` CLI inside
-its disposable fixture. The fixture contract is prepared by CI, the provider
-inherits a fixture-local PATH entry, and human approval is never synthesized.
-This makes provider-side verification available without making the fixture a
-security boundary; the outer harness remains the authoritative evidence gate.
-
-Benchmark reports also expose a fail-closed `qualityGate`. It separates
-historical retries from effective delivery, reports duration regressions, and
-labels small samples as `insufficient-data` or `directional` instead of
-claiming enterprise-level improvement. Configure
-`minBaselineSamplesPerTask` alongside `minCompletedRunsPerTask`: a single
-baseline observation is not comparable to replicated harness runs. Baseline
-observations may store `durationSamplesMs`; the report compares medians.
+The Playbook contains guidance; this repository contains the enforceable SDLC
+engine. See [MANIFESTO.md](./MANIFESTO.md) for the boundary and
+[docs/ORGANIZATION.md](./docs/ORGANIZATION.md) for the Angular Conventional
+Commits and capability layout.
 
 ## Development
 
 ```bash
 pnpm install
-pnpm --filter @agentskit/harness typecheck
-pnpm --filter @agentskit/harness test
-pnpm --filter @agentskit/harness build
+pnpm typecheck
+pnpm test
+pnpm build
+pnpm pack --pack-destination /tmp/agentskit-harness-pack
 ```
 
 See [CONTRIBUTING.md](./CONTRIBUTING.md) for changes, tests, and release expectations. See [CHANGELOG.md](./CHANGELOG.md) for version history.
 
-The Playbook CI dogfoods the package with `harness:test`, `harness:cli`, and a
-clean packed-consumer probe before the broader repository checks run.
-It also uploads the resulting `.codex/verification/harness-phase-24` run
-directory so a human can inspect the exact evidence before approval.
+## Release
+
+Releases are published by `.github/workflows/release-harness.yml` after a merge to
+`main`. The workflow uses npm Trusted Publishing (GitHub OIDC) and does not read
+or require an `NPM_TOKEN`. Configure the npm trusted publisher once for
+`AgentsKit-io/harness`, workflow `release-harness.yml`, and package
+`@agentskit/harness`; version changes remain the release trigger.
+The 0.4.0 candidate checklist and explicit blockers live in
+[`release/manifest.json`](release/manifest.json) and
+[`release/notes.md`](release/notes.md).
 
 ## License