@agentskit/harness 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +77 -2
- package/capabilities/public-surface.json +668 -0
- package/compatibility/manifest.json +17 -0
- package/compatibility/migration.md +10 -0
- package/compatibility/report.json +23 -0
- package/compatibility/report.md +22 -0
- package/compatibility/rollback.md +8 -0
- package/dist/cli.js +185 -41
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +633 -35
- package/dist/index.js +1344 -300
- package/dist/index.js.map +1 -1
- package/docs/ADR-0026-kernel-adapters-boundary.md +82 -0
- package/docs/GETTING-STARTED.md +18 -0
- package/docs/MODULE-BOUNDARIES.md +143 -0
- package/docs/ORGANIZATION.md +13 -4
- package/docs/TROUBLESHOOTING.md +24 -0
- package/examples/minimum-profile.mjs +27 -0
- package/package.json +13 -2
- package/release/manifest.json +14 -0
- package/release/notes.md +10 -0
- package/release/qualification.json +14 -0
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# ADR-0026: Kernel and adapter boundary for 0.4.0
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted — approved by the owner on 2026-09-10.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The Harness now contains deterministic SDLC controls, local execution
|
|
10
|
+
plumbing, and three provider-facing adapters. The 0.4.0 goal is a modular
|
|
11
|
+
engine that can plug in orchestrators, documentation, trackers, memory,
|
|
12
|
+
models, MCP, and event bridges without making the kernel provider-specific.
|
|
13
|
+
H-040 requires a baseline before changing behavior and must not infer
|
|
14
|
+
architecture decisions that belong to the owner.
|
|
15
|
+
|
|
16
|
+
## Decision
|
|
17
|
+
|
|
18
|
+
Adopt the following boundary for review:
|
|
19
|
+
|
|
20
|
+
1. **Kernel:** deterministic contracts and decisions: state machine, discovery,
|
|
21
|
+
WIP admission, delivery gates, eval scoring, cache/memory record contracts,
|
|
22
|
+
workflow scheduling, policy, preflight, coordination, resilience, status,
|
|
23
|
+
learning, model bindings, and metric projections. Kernel modules may use
|
|
24
|
+
Node standard library and other kernel contracts only.
|
|
25
|
+
2. **Execution support:** local filesystem, Git/source snapshots, process or
|
|
26
|
+
Docker execution, event logs, evidence, run persistence, reconciliation,
|
|
27
|
+
and the CLI. It may depend on the kernel but does not define provider
|
|
28
|
+
semantics.
|
|
29
|
+
3. **Adapters/plugins:** Doc Bridge, Orca, Linear/GitHub (or another tracker),
|
|
30
|
+
LLM providers, memory stores, MCP, and event bridges. They implement
|
|
31
|
+
generic contracts and own credentials/network side effects.
|
|
32
|
+
4. **Composition:** `src/index.ts` and `src/cli.ts` compose the surfaces. A
|
|
33
|
+
consumer must not import private module paths as a supported API.
|
|
34
|
+
5. **Fail-closed rule:** provider failure, missing evidence, stale source, or
|
|
35
|
+
ambiguous product decisions cannot be converted into a passing kernel
|
|
36
|
+
decision by an adapter.
|
|
37
|
+
|
|
38
|
+
## Dependency constraints
|
|
39
|
+
|
|
40
|
+
```text
|
|
41
|
+
consumer -> composition -> execution support -> kernel
|
|
42
|
+
consumer -> composition -> adapters -> kernel contracts
|
|
43
|
+
kernel -> kernel / Node stdlib only
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Type-only references between `context`, `plugins`, `events`, `runtime`, and
|
|
47
|
+
`types` are allowed. `bundle` may call run reconciliation because signed
|
|
48
|
+
evidence must be exported only from a reconciled terminal run. These are the
|
|
49
|
+
only baseline exceptions; a new runtime import direction requires an ADR.
|
|
50
|
+
|
|
51
|
+
## Human decisions required
|
|
52
|
+
|
|
53
|
+
The following decisions are intentionally not inferred by this ADR:
|
|
54
|
+
|
|
55
|
+
- approve the module classifications in `docs/MODULE-BOUNDARIES.md`;
|
|
56
|
+
- approve `codex / gpt-5.6-luna / high` as the pilot's fixed provider/model
|
|
57
|
+
binding, or provide a replacement;
|
|
58
|
+
- approve the proposed eval thresholds and three repetitions, or provide
|
|
59
|
+
different values;
|
|
60
|
+
- provide or authorize a controlled no-Harness cohort and its task IDs;
|
|
61
|
+
- decide whether future MCP/event-bridge adapters belong in this package or a
|
|
62
|
+
separate integration package once their contracts are specified.
|
|
63
|
+
|
|
64
|
+
The architecture decision is accepted. H-040 still requires empirical baseline
|
|
65
|
+
collection and verification evidence before it can be reported complete.
|
|
66
|
+
|
|
67
|
+
## Consequences
|
|
68
|
+
|
|
69
|
+
- Swapping Orca, a tracker, Doc Bridge, a model, or a memory backend does not
|
|
70
|
+
require changing kernel decisions.
|
|
71
|
+
- External side effects stay auditable and testable through adapter contracts.
|
|
72
|
+
- The kernel remains usable in a local process or Docker execution mode.
|
|
73
|
+
- Provider-specific integration tests and evals are required before an adapter
|
|
74
|
+
can be described as production-ready.
|
|
75
|
+
|
|
76
|
+
## Alternatives considered
|
|
77
|
+
|
|
78
|
+
- **Provider logic in the kernel:** rejected; it couples decisions to vendors
|
|
79
|
+
and makes deterministic comparison impossible.
|
|
80
|
+
- **A new abstraction layer for every capability:** rejected for now; one-file
|
|
81
|
+
capabilities stay in the existing capability-first layout until a second
|
|
82
|
+
implementation or boundary test requires a directory.
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Getting started
|
|
2
|
+
|
|
3
|
+
Install the package, then run the included consumer example:
|
|
4
|
+
|
|
5
|
+
```bash
|
|
6
|
+
pnpm add -D @agentskit/harness
|
|
7
|
+
pnpm build
|
|
8
|
+
node examples/minimum-profile.mjs
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
The example uses a fake profile in YOLO mode, a coding-agent adapter, a local
|
|
12
|
+
Doc Bridge index, adversarial code review, and a dry-run tracking adapter. It
|
|
13
|
+
prints a structured result and never performs a network mutation.
|
|
14
|
+
|
|
15
|
+
For a real project, keep phase decisions in the kernel, select a named profile,
|
|
16
|
+
and provide integrations through adapters. Start with `mode: "dry-run"`, inspect
|
|
17
|
+
the evidence, then move to `safe` or `yolo` only after the preflight contract is
|
|
18
|
+
current.
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
# Module and boundary inventory (H-040)
|
|
2
|
+
|
|
3
|
+
This document is the reviewed starting point for the 0.4.0 work. It describes
|
|
4
|
+
the repository at revision `f4b2b092af97d528bd5fe5955ed38f101521b98d` (the
|
|
5
|
+
`main` baseline captured on 2026-09-10) and is intentionally descriptive: it
|
|
6
|
+
does not claim that the proposed 0.4.0 boundary has already been accepted.
|
|
7
|
+
|
|
8
|
+
## Classification
|
|
9
|
+
|
|
10
|
+
| Class | Meaning | Dependency rule |
|
|
11
|
+
| --- | --- | --- |
|
|
12
|
+
| Kernel | Deterministic contracts, state transitions, policy and metric projections | May use Node standard library and other kernel modules; must not import adapters, provider SDKs, CLI composition, or credentials. |
|
|
13
|
+
| Execution support | Local process, filesystem, event, evidence, and run lifecycle plumbing | May depend on the kernel; must expose provenance and fail closed at trust boundaries. |
|
|
14
|
+
| Adapter | Provider-specific or external-system integration | May depend on kernel contracts; must not be imported by kernel modules. |
|
|
15
|
+
| Composition | Package and CLI entry points | May compose kernel, execution support, and adapters; consumers use `src/index.ts`. |
|
|
16
|
+
|
|
17
|
+
## Complete source inventory
|
|
18
|
+
|
|
19
|
+
Every current TypeScript module is listed below. Relative imports are the
|
|
20
|
+
observed imports at the baseline revision; `stdlib` means Node's built-in
|
|
21
|
+
modules, not a provider dependency.
|
|
22
|
+
|
|
23
|
+
| Module | Class | Responsibility | Relative dependencies | External boundary |
|
|
24
|
+
| --- | --- | --- | --- | --- |
|
|
25
|
+
| `src/kernel/constants.ts` | Kernel | State, decision, and surface constants | `types` (type-only) | None |
|
|
26
|
+
| `src/kernel/errors.ts` | Kernel | Typed fail-closed errors | None | None |
|
|
27
|
+
| `src/kernel/hash.ts` | Kernel | SHA-256 and JSON digests | stdlib | None |
|
|
28
|
+
| `src/kernel/types.ts` | Kernel | Shared contract, run, evidence, and metric types | `context` (type-only) | None |
|
|
29
|
+
| `src/kernel/state-machine.ts` | Kernel | Legal lifecycle transitions and human decisions | `constants`, `types` | None |
|
|
30
|
+
| `src/kernel/discovery.ts` | Kernel | Discovery freshness, ambiguity, and decision packets | `errors`, `hash` | None |
|
|
31
|
+
| `src/kernel/wip.ts` | Kernel | WIP admission and capacity decisions | `errors` | None |
|
|
32
|
+
| `src/kernel/experiment.ts` | Kernel | Comparable runtime/provider selection | `errors` | None |
|
|
33
|
+
| `src/delivery/index.ts` | Delivery | G2–G5 delivery gates and deterministic PR projection | `errors`, `hash` | None |
|
|
34
|
+
| `src/delivery/review.ts` | Delivery | Bounded parallel adversarial review lenses and evidence verdicts | `errors`, `hash`, `workflow`, `delivery/index` (type-only) | Reviewer callback supplied by caller |
|
|
35
|
+
| `src/kernel/cycle.ts` | Kernel | Bounded improvement-cycle assessment | stdlib, `errors` | None |
|
|
36
|
+
| `src/kernel/eval.ts` | Kernel | Versioned eval manifest validation, deterministic battery runner, min/median/max aggregation, and fail-closed assessment | `errors`, `hash` | Grader callback supplied by caller |
|
|
37
|
+
| `src/kernel/compatibility.ts` | Kernel | Pinned AgentsKit component manifest and evidence-bound compatibility assessment | `errors`, `hash` | Upstream commands and reports supplied by caller |
|
|
38
|
+
| `src/kernel/cache.ts` | Kernel | Safe cache keys and in-memory LLM cache contract | `hash`, `errors` | Cache backend supplied by caller |
|
|
39
|
+
| `src/kernel/optimization.ts` | Kernel | Token, memory, cache, parallelism comparisons | `hash`, `errors` | None |
|
|
40
|
+
| `src/kernel/memory.ts` | Kernel | Memory record validation and memory adapter contract | `errors` | KV store supplied by caller |
|
|
41
|
+
| `src/kernel/policy.ts` | Kernel | Tool/action policy gate | `errors` | None |
|
|
42
|
+
| `src/kernel/preflight.ts` | Kernel | File-scoped checks and safe-command validation | stdlib, `errors` | Shell command is data, never executed here |
|
|
43
|
+
| `src/kernel/block.ts` | Kernel | Portable execution block manifest and dependency admission | `errors`, `hash` | None |
|
|
44
|
+
| `src/kernel/learning.ts` | Kernel | Retrospective parsing and human learning promotion | stdlib, `errors` | None |
|
|
45
|
+
| `src/kernel/status.ts` | Kernel | Deterministic status snapshot and digest | `errors`, `hash`, `block`, `types` (type-only) | None |
|
|
46
|
+
| `src/kernel/model-policy.ts` | Kernel | Role-to-model binding and validation | `errors`, `hash` | Provider is data, not an SDK |
|
|
47
|
+
| `src/execution/machine.ts` | Execution support | Machine sampling and adaptive concurrency | stdlib, `types`, `errors` | Host CPU/memory metrics |
|
|
48
|
+
| `src/execution/coordination.ts` | Execution support | Atomic issue/worktree claims and dispatch ledger | stdlib, `errors`, `hash` | Local state directory only |
|
|
49
|
+
| `src/kernel/resilience.ts` | Kernel | Failure classification and bounded retry/recovery policy | `errors` | Operation callback supplied by caller |
|
|
50
|
+
| `src/kernel/workflow.ts` | Kernel | Bounded workflow scheduling | `errors` | Node callbacks supplied by caller |
|
|
51
|
+
| `src/kernel/phase-executor.ts` | Kernel | Declarative phase routing, preflight, effect policy, and bounded decisions | `errors`, `workflow` | Phase handlers, gates, and Grill-me callback supplied by caller |
|
|
52
|
+
| `src/kernel/artifacts.ts` | Kernel support | Versioned provenance-bound artifacts, Markdown rendering, and idempotent phase resume projection | stdlib, `errors`, `hash`, `events`, `phase-executor` (type-only) | Local state directory only |
|
|
53
|
+
| `src/kernel/adapter-contract.ts` | Kernel | Shared assurance levels and bounded telemetry contract | `errors` | Provider measurements supplied by adapters |
|
|
54
|
+
| `src/kernel/quality.ts` | Kernel | Phase telemetry validation, 0–100 quality matrix, baseline deltas, and watchdog blockers | `errors`, `hash` | Metrics supplied by phases/adapters |
|
|
55
|
+
| `src/kernel/pilot.ts` | Kernel | Cohort freeze and pilot assessment | `errors`, `hash` | None |
|
|
56
|
+
| `src/kernel/plugins.ts` | Kernel | Generic slots, dependency checks, and lifecycle listeners | `errors`, `events` (type-only) | Plugin implementation supplied by caller |
|
|
57
|
+
| `src/context/index.ts` | Context | Context snapshot contract, hashing, and provider slot | stdlib, `plugins`, `hash`, `errors` | Provider implementation supplied by caller |
|
|
58
|
+
| `src/execution/metrics.ts` | Execution support | Benchmark manifest validation, run projection, and comparison | stdlib, `errors`, `files`, `types` | Reads local state only |
|
|
59
|
+
| `src/execution/config.ts` | Execution support | Contract loading, profile resolution, and config hashing | stdlib, `constants`, `profiles`, `errors`, `hash`, `files`, `types` | Local `.codex/verification.json` |
|
|
60
|
+
| `src/profiles/index.ts` | Profiles | Profile defaults and overrides | `errors` | None |
|
|
61
|
+
| `src/execution/files.ts` | Execution support | Run/config JSON and task-artifact filesystem helpers | stdlib, `errors`, `types` | Local filesystem |
|
|
62
|
+
| `src/execution/source.ts` | Execution support | Git/source snapshot and dirty-tree detection | stdlib, `hash`, `errors`, `types` | Git CLI |
|
|
63
|
+
| `src/execution/runs.ts` | Execution support | Run persistence and lifecycle event creation | `hash`, `files`, `events`, `context`, `types` | Local state directory |
|
|
64
|
+
| `src/execution/verification.ts` | Execution support | Plan/start/verify/reconcile/approval orchestration | stdlib, `runs`, `config`, `errors`, `evidence`, `state-machine`, `source`, `files`, `context`, `hash`, `types`, `events`, `machine` | Configured check commands and local processes |
|
|
65
|
+
| `src/kernel/events.ts` | Kernel support | Event contracts, append-only log, hash chain, and lock recovery | stdlib, `errors`, `hash`, `context` (type-only), `types` (type-only), `runtime` (type-only) | Local filesystem |
|
|
66
|
+
| `src/execution/evidence.ts` | Execution support | Structured evidence parsing and artifact validation | stdlib, `hash`, `files`, `types` | Check output and local artifacts |
|
|
67
|
+
| `src/execution/bundle.ts` | Execution support | Signed evidence bundle export and verification | stdlib, `errors`, `files`, `hash`, `verification`, `events`, `config`, `types` | Local keys and files |
|
|
68
|
+
| `src/execution/agent.ts` | Execution support | Agent session recorder, resume, and tool lifecycle | stdlib, `events`, `errors`, `policy`, `runtime`, `types` | Agent/runtime callbacks |
|
|
69
|
+
| `src/execution/runtime.ts` | Execution support | Process, Docker, and generic tool runtimes | stdlib, `hash`, `errors`, `types` | Child process and Docker CLI |
|
|
70
|
+
| `src/cli.ts` | Composition | `ak-harness` / `ak-verify` command surface | `index`, `metrics`, `errors`, `events`, stdlib, `commander` | Shell/CLI invocation |
|
|
71
|
+
| `src/index.ts` | Composition | Supported package entry point | All supported public modules | Consumer import boundary |
|
|
72
|
+
| `src/adapters/doc-bridge.ts` | Adapter | Deterministic Doc Bridge context provider | stdlib, `hash`, `context` | `.doc-bridge/index.json` |
|
|
73
|
+
| `src/adapters/agent.ts` | Adapter | Structured coding-agent execution with bounded timeout and failure classification | `errors`, `resilience`, `adapter-contract` | Agent/provider callback supplied by caller |
|
|
74
|
+
| `src/adapters/orca.ts` | Adapter | Safe, idempotent Orca dispatch plan | `errors`, `hash`, `preflight` | Orca CLI arguments; no execution |
|
|
75
|
+
| `src/adapters/tracking.ts` | Adapter | Idempotent provider-neutral tracking transition | `errors`, `hash` | User-supplied Linear/GitHub/etc. handler |
|
|
76
|
+
|
|
77
|
+
## Allowed dependency directions and exceptions
|
|
78
|
+
|
|
79
|
+
```text
|
|
80
|
+
consumer -> src/index.ts -> composition
|
|
81
|
+
-> execution support -> kernel
|
|
82
|
+
-> adapters -> kernel
|
|
83
|
+
kernel -----------------> kernel (or Node stdlib)
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The following are intentional exceptions and are type-only or composition
|
|
87
|
+
edges, not provider leakage:
|
|
88
|
+
|
|
89
|
+
1. `context.ts` uses the generic plugin slot from `plugins.ts`; the provider
|
|
90
|
+
implementation remains outside the kernel.
|
|
91
|
+
2. `plugins.ts` references event types and `events.ts` references context and
|
|
92
|
+
runtime types only. These are compile-time contracts, not runtime cycles.
|
|
93
|
+
3. `cli.ts` imports from `index.ts` so the CLI exercises the supported public
|
|
94
|
+
surface rather than private implementation paths.
|
|
95
|
+
4. `bundle.ts` calls reconciliation because a bundle is only exportable from a
|
|
96
|
+
reconciled terminal run; this is a local execution-support dependency.
|
|
97
|
+
5. `runtime.ts` contains process/Docker mechanics because those are execution
|
|
98
|
+
providers. Kernel decisions receive runtime evidence as data and do not
|
|
99
|
+
import the runtime implementation.
|
|
100
|
+
|
|
101
|
+
Any new external provider (Linear, GitHub, Orca, Doc Bridge, MCP, event bridge,
|
|
102
|
+
LLM, or memory backend) must enter through an adapter or plugin slot. A new
|
|
103
|
+
provider import in a kernel module is a boundary violation and requires an ADR.
|
|
104
|
+
|
|
105
|
+
## Public entry point inventory
|
|
106
|
+
|
|
107
|
+
`src/index.ts` is the only supported consumer entry point. Its named exports
|
|
108
|
+
are grouped below; the source file remains authoritative for exact signatures.
|
|
109
|
+
|
|
110
|
+
| Area | Named exports |
|
|
111
|
+
| --- | --- |
|
|
112
|
+
| Lifecycle/config | `STATES`, `LEGAL_TRANSITIONS`, `HarnessError`, `loadConfig`, `validateConfig`, `transition`, `assertHuman`, `approvedDecision`, `loadLatestRun`, `planRun`, `startRun`, `verifyRun`, `reconcileRun`, `approveRun`, `authorizeRun`, `retryRun`, `cancelRun`, `cleanTaskArtifacts` |
|
|
113
|
+
| Events/plugins/context | `EVENT_LOG_GENESIS`, `FileEventStore`, `HARNESS_EVENT_SCHEMA_VERSION`, `HARNESS_EVENT_TYPES`, `inspectEventLogLock`, `recoverEventLogLock`, `createPluginRegistry`, `createPluginSlot`, `HARNESS_PLUGIN_API_VERSION`, `CONTEXT_PROVIDER_SLOT`, `hashContextSnapshot`, `hashContextSnapshots`, `readContextSnapshots`, `validateContextSnapshot`, `validateContextSnapshots` |
|
|
114
|
+
| Discovery and delivery | `assessDiscovery`, `isDiscoveryCurrent`, `assessWip`, `WIP_STATES`, `selectRuntime`, `assessAcceptance`, `assessIntegration`, `assessPreflight`, `assessProduction`, `assessWorktreeCleanup`, `composePullRequest`, `assessPilot`, `IMPROVEMENT_CYCLE_STEPS`, `assessImprovementCycle` |
|
|
115
|
+
| Eval and optimization | `assessAgentEval`, `runAgentEval`, `createLlmCache`, `createLlmCacheKey`, `validateCacheableOperation`, `compareOptimization`, `validateOptimizationObservation`, `MEMORY_SCOPES`, `createInMemoryMemoryAdapter`, `createKvMemoryAdapter`, `validateMemoryRecord`, `runWorkflow`, `BENCHMARK_SCHEMA_VERSION`, `benchmarkRuns`, `loadBenchmarkManifest`, `recordBenchmarkObservation`, `validateBenchmarkManifest` |
|
|
116
|
+
| Agent/runtime controls | `createSessionRecorder`, `createCodingAgentAdapter`, `createPolicyGate`, `createConfiguredToolRuntime`, `createDockerToolRuntime`, `createProcessToolRuntime`, `createToolRuntime`, `adaptiveConcurrency`, `createMachineMonitor`, `sampleMachine`, `summarizeMachine`, `createDispatchLedger`, `classifyFailure`, `recoveryDelayMs`, `runWithRecovery`, `planFilePreflight`, `validateSafeCommand`, `BLOCK_STATUSES`, `assessBlock`, `validateBlockManifest`, `LEARNING_STATUSES`, `parseRetro`, `promoteLearnings`, `createStatusSnapshot`, `validateStatusSnapshot`, `MODEL_ROLES`, `createModelPolicy`, `modelFor`, `PHASE_MODES`, `createPhaseProfile`, `planPhaseProfile`, `executePhaseProfile`, `ARTIFACT_SCHEMA_VERSION`, `createArtifactEnvelope`, `FileArtifactStore`, `artifactIsFresh`, `resumeStateFromArtifacts`, `ASSURANCE_LEVELS`, `validateAdapterMetadata` |
|
|
117
|
+
| Integrations and evidence | `createDocBridgeContextProvider`, `createOrcaDispatchPlan`, `createTrackingAdapter`, `createTrackingTransition`, `EVIDENCE_BUNDLE_SCHEMA_VERSION`, `exportEvidenceBundle`, `readEvidenceTrustStore`, `verifyEvidenceBundle` |
|
|
118
|
+
|
|
119
|
+
The entry point also re-exports the public type surfaces from `types`,
|
|
120
|
+
`events`, `plugins`, `context`, `discovery`, `wip`, `experiment`, `delivery`,
|
|
121
|
+
`pilot`, `cycle`, `metrics`, `agent`, `policy`, `runtime`, `bundle`, and
|
|
122
|
+
`machine`, `phase-executor`, `artifacts`, plus explicit type exports for cache, optimization, memory,
|
|
123
|
+
coordination, resilience, preflight, block, learning, status, model policy,
|
|
124
|
+
Orca, and tracking. No adapter implementation is re-exported wholesale.
|
|
125
|
+
|
|
126
|
+
## External integration inventory
|
|
127
|
+
|
|
128
|
+
| Integration | Current location | Side effects | 0.4.0 boundary |
|
|
129
|
+
| --- | --- | --- | --- |
|
|
130
|
+
| Doc Bridge | `src/adapters/doc-bridge.ts` | Reads a local index | Keep behind `ContextProvider`; measure context hit/quality separately. |
|
|
131
|
+
| Orca | `src/adapters/orca.ts` | None; produces argv and lifecycle projections only | Keep lease/worktree/issue-lock/SHA planning provider-neutral; execution belongs to the orchestrator. |
|
|
132
|
+
| Linear/GitHub/other tracker | `src/adapters/tracking.ts` callback | Caller-owned network mutation | Require idempotency key and explicit tracking authorization. |
|
|
133
|
+
| Process runtime | `src/execution/runtime.ts` | Starts child processes | Execution support; policy and evidence gates remain kernel decisions. |
|
|
134
|
+
| Docker runtime | `src/execution/runtime.ts` | Starts Docker containers | Optional sandbox selected by config, never a mandatory kernel dependency. |
|
|
135
|
+
| LLM provider/model | Caller/plugin | Provider call and token spend | Bind provider/model in experiment metadata; do not embed SDKs in kernel. |
|
|
136
|
+
| Memory backend | Caller/plugin; `memory.ts` contract | Backend reads/writes | Keep record validation in kernel; backend adapter owns persistence. |
|
|
137
|
+
| MCP/event bridge | Not implemented | Future network/event effects | Add as adapters only after a separate ADR and eval coverage. |
|
|
138
|
+
|
|
139
|
+
## Review status
|
|
140
|
+
|
|
141
|
+
This inventory is evidence for H-040. ADR-0026 records the accepted boundary;
|
|
142
|
+
the remaining H-040 work is empirical baseline collection and verification
|
|
143
|
+
evidence, not an unresolved architecture choice.
|
package/docs/ORGANIZATION.md
CHANGED
|
@@ -9,10 +9,17 @@ the repository stays navigable without speculative layers.
|
|
|
9
9
|
├── src/
|
|
10
10
|
│ ├── index.ts # supported package API
|
|
11
11
|
│ ├── cli.ts # ak-harness / ak-verify commands
|
|
12
|
-
│ ├──
|
|
13
|
-
│
|
|
12
|
+
│ ├── kernel/ # deterministic contracts and decisions
|
|
13
|
+
│ ├── execution/ # filesystem, runtime, evidence, and run plumbing
|
|
14
|
+
│ ├── context/ # context contracts and provider slot
|
|
15
|
+
│ ├── delivery/ # delivery gates and PR projection
|
|
16
|
+
│ ├── profiles/ # configuration profiles
|
|
17
|
+
│ └── adapters/ # optional integrations
|
|
14
18
|
├── test/ # deterministic tests and fixtures
|
|
15
19
|
├── scripts/ # real CLI, packaging, and repository checks
|
|
20
|
+
├── capabilities/ # generated public-surface capability manifest
|
|
21
|
+
├── examples/ # runnable consumer examples
|
|
22
|
+
├── compatibility/ # pinned ecosystem manifest and migration/rollback procedures
|
|
16
23
|
├── docs/ # ADRs and this organization contract
|
|
17
24
|
├── .codex/ # verification contract and local run state
|
|
18
25
|
└── .github/ # CI and release workflows
|
|
@@ -22,8 +29,10 @@ the repository stays navigable without speculative layers.
|
|
|
22
29
|
|
|
23
30
|
- Import internal modules with explicit relative paths; only `src/index.ts`
|
|
24
31
|
is supported for consumers.
|
|
25
|
-
-
|
|
26
|
-
|
|
32
|
+
- Keep deterministic decisions in `kernel/`; execution mechanics and external
|
|
33
|
+
effects live behind the other capability boundaries.
|
|
34
|
+
- A single-file capability may use an `index.ts` boundary when it has a
|
|
35
|
+
distinct public contract; do not add speculative layers.
|
|
27
36
|
- Adapters may depend on the kernel; the kernel must not depend on adapters or
|
|
28
37
|
external providers.
|
|
29
38
|
- Tests may use internal modules when exercising a boundary, but consumer
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Troubleshooting
|
|
2
|
+
|
|
3
|
+
## `STALE`
|
|
4
|
+
|
|
5
|
+
The source, contract, or context changed after verification. Run `ak-verify`
|
|
6
|
+
again; do not reuse the old evidence bundle.
|
|
7
|
+
|
|
8
|
+
## `BLOCKED` or `AWAITING_HUMAN_APPROVAL`
|
|
9
|
+
|
|
10
|
+
Inspect the structured run with `ak-harness status --json`. Resolve the listed
|
|
11
|
+
ambiguity, failed gate, missing evidence, or approval, then rerun verification.
|
|
12
|
+
YOLO only removes unnecessary pauses; it does not bypass a required safety or
|
|
13
|
+
provenance gate.
|
|
14
|
+
|
|
15
|
+
## Missing adapter telemetry
|
|
16
|
+
|
|
17
|
+
Return `status: "unknown"` for measurements you cannot observe. Unknown values
|
|
18
|
+
are excluded from improvement claims and can block a configured quality gate.
|
|
19
|
+
|
|
20
|
+
## Runtime failures
|
|
21
|
+
|
|
22
|
+
Use the process runtime for a local shell-free boundary or the Docker runtime
|
|
23
|
+
when isolation is required. Both report timeout, cancellation, output-limit,
|
|
24
|
+
and non-zero-exit failures as structured evidence.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
|
|
2
|
+
import { tmpdir } from 'node:os'
|
|
3
|
+
import { join } from 'node:path'
|
|
4
|
+
import {
|
|
5
|
+
createCodingAgentAdapter,
|
|
6
|
+
createDocBridgeContextProvider,
|
|
7
|
+
createPhaseProfile,
|
|
8
|
+
createTrackingAdapter,
|
|
9
|
+
executePhaseProfile,
|
|
10
|
+
runAdversarialReview,
|
|
11
|
+
} from '../dist/index.js'
|
|
12
|
+
|
|
13
|
+
const root = mkdtempSync(join(tmpdir(), 'agentskit-harness-example-'))
|
|
14
|
+
try {
|
|
15
|
+
writeFileSync(join(root, '.doc-bridge.json'), JSON.stringify({ contentHash: 'example', knowledge: [{ id: 'guide', title: 'Guide', path: 'guide.md', body: 'approved workflow' }] }))
|
|
16
|
+
const profile = createPhaseProfile({ id: 'minimum', mode: 'yolo', phases: [{ id: 'implement', effect: 'write', outputs: ['result'] }], maxConcurrency: 1 })
|
|
17
|
+
const agent = createCodingAgentAdapter({ id: 'fake-agent', version: '1.0.0', execute: () => ({ output: { ok: true }, diff: '', usage: { status: 'measured', inputTokens: 1, outputTokens: 1, totalTokens: 2 } }) })
|
|
18
|
+
const tracking = createTrackingAdapter('fake-tracker', () => undefined, { dryRun: true })
|
|
19
|
+
const docs = createDocBridgeContextProvider({ root, indexPath: '.doc-bridge.json' })
|
|
20
|
+
const review = await runAdversarialReview({ lenses: [{ id: 'contract' }], binding: { candidateRevision: 'example', contractHash: 'contract', configHash: 'config' }, reviewer: () => ({ status: 'pass', evidence: 'example-review' }) })
|
|
21
|
+
const transition = await tracking.transition({ tracker: 'fake', issue: 'EXAMPLE-1', to: 'qa', reason: 'example' })
|
|
22
|
+
const context = await docs.resolve({ query: 'approved' })
|
|
23
|
+
const execution = await executePhaseProfile(profile, { preflight: () => ({ decision: 'pass' }), handlers: { implement: () => ({ decision: 'pass', outputs: { result: 'ok' } }) } })
|
|
24
|
+
console.log(JSON.stringify({ status: 'passed', profile: execution.status, agent: agent.id, review: review.decision, tracking: transition.to, contextReferences: context.references.length }))
|
|
25
|
+
} finally {
|
|
26
|
+
rmSync(root, { recursive: true, force: true })
|
|
27
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agentskit/harness",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Portable, evidence-backed development harness for coding agents.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": { "ak-harness": "dist/cli.js", "ak-verify": "dist/cli.js" },
|
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
"import": "./dist/index.js"
|
|
13
13
|
}
|
|
14
14
|
},
|
|
15
|
-
"files": ["dist", "README.md", "CHANGELOG.md", "CONTRIBUTING.md", "CODE_OF_CONDUCT.md", "SECURITY.md", "MANIFESTO.md", "LICENSE", "docs"],
|
|
15
|
+
"files": ["dist", "README.md", "CHANGELOG.md", "CONTRIBUTING.md", "CODE_OF_CONDUCT.md", "SECURITY.md", "MANIFESTO.md", "LICENSE", "docs", "capabilities", "examples", "compatibility", "release"],
|
|
16
16
|
"engines": { "node": ">=22" },
|
|
17
17
|
"scripts": {
|
|
18
18
|
"build": "tsup && node -e \"console.log(JSON.stringify({status:'passed',criteria:['package']}))\"",
|
|
@@ -43,6 +43,17 @@
|
|
|
43
43
|
"test:optimization": "pnpm typecheck && vitest run --config vitest.config.ts test/optimization.test.ts test/issues-010-014.eval.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['optimization-contracts','eval','cache','parallelism']}))\"",
|
|
44
44
|
"test:source": "pnpm typecheck && vitest run --config vitest.config.ts test/source.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['source-current']}))\"",
|
|
45
45
|
"test:metrics": "pnpm typecheck && vitest run --config vitest.config.ts test/metrics.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['metrics']}))\"",
|
|
46
|
+
"test:capabilities": "node scripts/generate-capability-manifest.mjs --check capabilities/public-surface.json",
|
|
47
|
+
"test:boundaries": "pnpm typecheck && vitest run --config vitest.config.ts test/boundaries.test.ts && node scripts/verify-dependency-directions.mjs",
|
|
48
|
+
"test:phase-executor": "pnpm typecheck && vitest run --config vitest.config.ts test/phase-executor.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['phase-executor']}))\"",
|
|
49
|
+
"test:artifacts": "pnpm typecheck && vitest run --config vitest.config.ts test/artifacts.test.ts && node scripts/verify-artifact-cli.mjs",
|
|
50
|
+
"test:adapters": "pnpm typecheck && vitest run --config vitest.config.ts test/adapters.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['adapters']}))\"",
|
|
51
|
+
"test:review": "pnpm typecheck && vitest run --config vitest.config.ts test/review.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['review-delivery']}))\"",
|
|
52
|
+
"test:quality": "pnpm typecheck && vitest run --config vitest.config.ts test/quality.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['quality-matrix']}))\"",
|
|
53
|
+
"test:eval-battery": "pnpm typecheck && vitest run --config vitest.config.ts test/eval-battery.test.ts test/issues-010-014.eval.test.ts test/optimization.test.ts && pnpm build >/dev/null && node scripts/verify-harness-eval-manifest.mjs && node -e \"console.log(JSON.stringify({status:'passed',criteria:['eval-battery','eval-manifest','eval-coverage']}))\"",
|
|
54
|
+
"test:compatibility": "pnpm typecheck && vitest run --config vitest.config.ts test/compatibility.test.ts && pnpm build >/dev/null && node scripts/verify-harness-compatibility-manifest.mjs && node -e \"console.log(JSON.stringify({status:'passed',criteria:['compatibility-contract','compatibility-manifest','pinned-revisions','real-adapter-boundary']}))\"",
|
|
55
|
+
"test:examples": "pnpm typecheck && pnpm build >/dev/null && node scripts/verify-harness-examples.mjs && node scripts/verify-adoption-docs.mjs",
|
|
56
|
+
"test:release-manifest": "pnpm typecheck && pnpm build >/dev/null && node scripts/verify-release-workflow.mjs && node scripts/verify-release-manifest.mjs",
|
|
46
57
|
"test:benchmark-record": "pnpm typecheck && node scripts/verify-harness-benchmark-record.mjs",
|
|
47
58
|
"test:agent-protocol": "pnpm typecheck && vitest run --config vitest.config.ts test/agent.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['agent-protocol']}))\"",
|
|
48
59
|
"test:policy-gate": "pnpm typecheck && vitest run --config vitest.config.ts test/policy.test.ts test/agent.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['policy-gate','agent-protocol']}))\"",
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
{
|
|
2
|
+
"type": "agentskit-harness-release-manifest",
|
|
3
|
+
"schemaVersion": 1,
|
|
4
|
+
"package": "@agentskit/harness",
|
|
5
|
+
"version": "0.4.0",
|
|
6
|
+
"channel": "latest",
|
|
7
|
+
"sourceRevision": "PENDING_PR_MERGE_SHA",
|
|
8
|
+
"requiredChecks": ["typecheck", "test", "build", "pack", "cli", "consumer", "eval-battery", "pilot-benchmark", "ecosystem-compatibility", "registry-smoke"],
|
|
9
|
+
"publication": { "workflow": ".github/workflows/release-harness.yml", "branch": "main", "trustedPublishing": true, "usesNpmToken": false },
|
|
10
|
+
"evidenceOutputs": ["release/qualification.json", "release/notes.md"],
|
|
11
|
+
"blockedCriteria": ["ecosystem-compatibility", "pilot-benchmark", "published-registry-smoke"],
|
|
12
|
+
"nextBaseline": "benchmarks/harness-0.4.0-baseline.json",
|
|
13
|
+
"digest": "0661969c719964eae2a2ab5b7bddd8b108e8792bb51d4a27c223474430730158"
|
|
14
|
+
}
|
package/release/notes.md
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# 0.4.0 release candidate
|
|
2
|
+
|
|
3
|
+
This release packages the deterministic SDLC kernel, phase quality matrix,
|
|
4
|
+
versioned eval battery, compatibility contract, and consumer examples.
|
|
5
|
+
|
|
6
|
+
Publication is intentionally limited to a merge on `main`. The release
|
|
7
|
+
workflow uses npm Trusted Publishing (`id-token: write`) and no `NPM_TOKEN`.
|
|
8
|
+
|
|
9
|
+
The candidate is not publishable until the pinned upstream AgentsKit tests/evals
|
|
10
|
+
and the post-publication registry/consumer smoke produce current evidence.
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
{
|
|
2
|
+
"type": "agentskit-harness-release-qualification",
|
|
3
|
+
"schemaVersion": 1,
|
|
4
|
+
"package": "@agentskit/harness",
|
|
5
|
+
"version": "0.4.0",
|
|
6
|
+
"sourceRevision": "a18b57d43f037b7a34b134e01baf913ae2341fd7",
|
|
7
|
+
"status": "blocked",
|
|
8
|
+
"harnessVerification": { "runId": "1789063362018-21016-775q93", "decision": "COMPLETE", "checksPassed": 21, "checksTotal": 21 },
|
|
9
|
+
"upstreamCompatibility": { "report": "compatibility/report.json", "status": "blocked" },
|
|
10
|
+
"pilotBenchmark": { "status": "blocked", "reason": "No comparable owner-approved no-Harness cohort; real code-review run has no baseline and did not pass quality matrix." },
|
|
11
|
+
"registrySmoke": { "status": "pending-publication", "reason": "Run only after the main Trusted Publishing workflow succeeds." },
|
|
12
|
+
"publishable": false,
|
|
13
|
+
"blockers": ["ecosystem-compatibility", "pilot-benchmark", "published-registry-smoke"]
|
|
14
|
+
}
|