@agentskit/harness 0.1.0 → 0.4.0

package/CHANGELOG.md

@@ -1,44 +1,45 @@
 # Changelog
 
+## [0.4.0] - 2026-09-10
+
+- Added phase quality matrices, watchdog classification, and resource telemetry.
+- Added versioned eval and ecosystem compatibility manifests with fail-closed
+  evidence handling.
+- Added runnable consumer onboarding, adapter examples, and troubleshooting.
+
+## [0.3.0] - 2026-09-10
+
+- Added portable issue/worktree claims and idempotent dispatch ledger.
+- Added failure classification, bounded retry/backoff, and abortable watchdog.
+- Added file-scoped preflight planning and shell-composition rejection.
+- Added block manifests, status snapshots, retro learning promotion, model
+  policies, and provider-neutral Orca/tracking adapters.
+- Added configurable machine pressure thresholds and adaptive workflow limits.
+
+## [Unreleased]
+
+- Add bounded agent eval, safe context/read-only LLM cache, deterministic workflow fan-out/fan-in, and validated optimization observation contracts for token, memory, cache, and parallelism measurements.
+
 All notable changes to `@agentskit/harness` are documented here.
 
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the package follows Semantic Versioning.
 
-## [0.1.0] - 2026-08-31
+## [0.2.0] - 2026-09-09
 
 ### Added
 
-- Replicated baseline collection with independent disposable samples, median
-  duration recording, and atomic complete-manifest replacement.
-- Baseline samples use the same end-to-end measurement boundary as the harness
-  check while retaining provider-only duration for diagnosis.
-- Incomplete baselines no longer qualify for duration or resource comparisons;
-  escape reduction remains visible as a separate outcome metric.
-- Baseline reports expose artifact acceptance separately from protocol
-  completion, preserving useful outcome data without weakening fail-closed
-  comparison gates.
-- Benchmark execution supports collision-safe task-specific replacement
-  samples, preserving blocked and timed-out evidence.
-- Benchmark comparisons expose artifact acceptance rates and protocol
-  completion rates separately from performance comparability.
-- Protocol and artifact acceptance comparisons expose absolute deltas when
-  relative improvement is undefined because the baseline was zero.
-- Benchmark tasks declare typed surface coverage; the AgentsKit OS corpus
-  includes executable CLI and documentation tasks with real acceptance
-  validation.
-- Fail-closed baseline sample policy and median baseline duration support for fair replicated comparisons.
-- Independent benchmark samples with median-duration comparisons and retry metrics that do not confuse replication with retries.
-- Configurable benchmark quality gates, confidence labels, and explicit retry metrics.
-- Portable `ak-verify` fixture gate for real coding-agent benchmark runs; provider delivery remains blocked until its result is complete and human approval is recorded.
-- Harness-equivalent benchmark preparation and collection for paired real-provider runs, with explicit human approval and honest non-comparability.
-- Source snapshots now ignore untracked directories while retaining hashes for untracked files.
-- Reproducible disposable-fixture runner and validator for the first real-provider AgentsKit OS baseline; improvement remains unavailable until equivalent harness runs exist.
-- Provenance-bearing benchmark tasks and a portable validator for AgentsKit OS coding benchmark reports; missing controlled observations remain explicitly non-comparable.
-- Playbook CI dogfooding for the package test suite, built CLI, and clean packed consumer.
-- Playbook CI now preserves the official harness run as an auditable artifact for human review.
-- Signed evidence flow now verifies a copied bundle from an isolated directory.
-- Benchmark reports now separate effective non-superseded delivery metrics from attempt history.
-- CI evidence now records automated preparation separately from human contract approval.
+- Pilot-cohort validation that freezes a policy/baseline pair and rejects non-normal, partial, or silently substituted ten-issue cohorts.
+- Bounded five-step improvement-cycle assessment with explicit adjustments, repeat decisions, and a deterministic quality matrix.
+
+- Deterministic G2–G5 assessment helpers and CLI commands for independent preflight review, structured idempotent PR handoff, current integration evidence, safe production exposure, and acceptance.
+
+- Current-source evidence now requires a committed Git `HEAD`; directories outside Git fail closed instead of receiving a synthetic revision.
+
+- Discovery gate API and CLI that emit `ready` or an auditable human decision packet from structured ambiguities, approved assumptions and source/contract/context bindings.
+- Deterministic WIP admission API and CLI that count blocked and awaiting-human deliveries, reserve resumed work, and reject duplicate ledger entries.
+- Controlled runtime-selection API and CLI that reject incomparable Orca/Emdash samples and exclude failed hard gates.
+- Configurable `runtime.kind` contract field plus a factory for bounded process or Docker-sandbox execution.
+- Real Git snapshot coverage for committed, dirty, untracked, and task-state-excluded evidence.
 - Strict TypeScript modular core with generated declarations and source maps.
 - Contract-frozen lifecycle, structured evidence, stale detection, human approval, retry, and cleanup.
 - Explicit human cancellation and superseded retry history.

package/CONTRIBUTING.md

@@ -1,25 +1,73 @@
 # Contributing to `@agentskit/harness`
 
-## Before opening a change
+The Harness is an enforcement engine, not a suggestion library. Changes must
+preserve fail-closed behavior and produce criterion-level evidence.
 
-1. State the behavior and failure mode the change addresses.
-2. Keep the public API in `src/index.ts` intentionally small.
-3. Add or update a typed test for every behavior change.
-4. Update the README, changelog, and contract when the user-facing protocol changes.
+## Repository layout
 
-## Local checks
+- `src/`: capability-oriented TypeScript modules and the public `index.ts`.
+- `src/adapters/`: optional provider integrations.
+- `test/`: deterministic unit and contract tests.
+- `scripts/`: real CLI and packaging checks.
+- `docs/`: ADRs and protocol decisions.
+- `.github/`: CI and release automation.
 
-```bash
-pnpm --filter @agentskit/harness typecheck
-pnpm --filter @agentskit/harness test
-pnpm --filter @agentskit/harness build
+See [docs/ORGANIZATION.md](./docs/ORGANIZATION.md) for the complete boundary
+and naming rules.
+
+## Change workflow
+
+1. Start from an issue or explicit task contract.
+2. Identify the outcome and executable validation it requires.
+3. Keep the public API change in `src/index.ts` intentional.
+4. Add or update criterion-level tests for behavior changes.
+5. Update README, changelog, ADRs, or the verification contract when the
+   protocol changes.
+6. Run the complete local gate:
+
+   ```bash
+   pnpm typecheck
+   pnpm test
+   pnpm build
+   pnpm pack --pack-destination /tmp/agentskit-harness-pack
+   ak-verify run --config .codex/verification.json --json
+   ```
+
+Do not report completion while a required gate is unavailable, blocked, stale,
+or awaiting approval.
+
+## Angular Conventional Commits
+
+Commit messages follow the Angular Conventional Commits format:
+
+```text
+<type>(<scope>): <imperative description>
 ```
 
-Do not claim completion when a required check is unavailable. Preserve criterion-level evidence and report blocked work honestly.
+Allowed types are `feat`, `fix`, `docs`, `refactor`, `test`, `build`, `ci`,
+`perf`, `chore`, and `revert`. Use a breaking-change footer when needed:
+
+```text
+feat(cli)!: reject incomplete evidence
+
+BREAKING CHANGE: `run` now exits non-zero when evidence is missing.
+```
+
+Keep commits focused. Never commit secrets, generated `dist/`, verification
+state, or residue from another task.
 
 ## Pull requests
 
-Use a focused branch and commit. Describe the contract change, affected states, evidence produced, and exact commands executed. Breaking public API or CLI changes require a changelog entry and a maintainer review.
+PRs must state the contract change, affected states, evidence produced, exact
+commands executed, and any unresolved blockers. A maintainer review is required
+for public API, CLI, state-machine, security, or release changes.
+
+## Releases
+
+Version changes merge to `main` through a PR. The release workflow runs tests,
+build, and pack, then publishes through npm Trusted Publishing (GitHub OIDC);
+no `NPM_TOKEN` is used. Configure the npm trusted publisher for
+`AgentsKit-io/harness` once before the first release.
 
 ## License
 

package/MANIFESTO.md

@@ -0,0 +1,23 @@
+# Harness Manifesto
+
+## The boundary
+
+The Playbook explains good practice. The Harness makes the SDLC observable and
+enforceable. Advice may be ignored; a required gate may not.
+
+## The guarantees
+
+1. A task starts from an explicit, frozen contract.
+2. Ambiguities are surfaced with options and a recommendation; the Harness
+   never invents product decisions.
+3. Every required check produces structured evidence bound to source, config,
+   and contract hashes.
+4. A failed or missing gate blocks progression and remains auditable.
+5. Recovery resumes from the journal without silently replaying uncertain work.
+6. Integrations are replaceable adapters, not hidden kernel dependencies.
+7. Measurements report what was observed, including unavailable data.
+
+## Operating principle
+
+Automate everything deterministic. Escalate only decisions that require human
+judgment. Keep the smallest mechanism that proves the claim.