@agentskit/harness 0.5.0 → 0.7.0

package/docs/ADR-0028-mcp-adapter-boundary.md

@@ -0,0 +1,45 @@
+# ADR-0028: MCP is an adapter-only boundary
+
+## Status
+
+Accepted — 2026-09-11.
+
+## Context
+
+MCP (Model Context Protocol) is a useful integration surface for read-only
+inspection and, later, gated tool calls. The keep-pushing loop and kernel must
+remain usable without any MCP host, SDK, or network dependency. MODULE-BOUNDARIES
+and ADR-0026 already classify MCP as an adapter concern; 0.6.0 needs an
+explicit decision before any public bridge ships.
+
+## Decision
+
+1. **Adapter-only.** MCP lives under `src/adapters/mcp.ts` (and future adapter
+   modules). Kernel modules do not import MCP clients, hosts, or transports.
+2. **Default-deny policy.** Every tool call goes through an allowlist
+   (`allowTools`) and a `PolicyGate` (or compatible `evaluate`). Missing
+   allowlist entry or non-`allow` decision returns `{ status: 'blocked' }`
+   without invoking the underlying call.
+3. **Read-only profile first.** The initial public surface is a policy-gated
+   call bridge intended for read-only / inspection tools. Mutating MCP tools
+   require a later ADR covering authorization, idempotency, and audit evidence.
+4. **Harness works without MCP.** Default loop config keeps `mcp.enabled: false`.
+   No MCP process is started by install, tick, or deliver.
+5. **Not wired into loop tick/deliver in 0.6.0.** The bridge is exported for
+   composition and future CLI/doctor use. Tick and deliver must not call it in
+   this release.
+
+## Consequences
+
+- Consumers can plug MCP behind the same policy gate used by session tool
+  recording without coupling the kernel to a vendor host.
+- Loop automation remains deterministic and offline-testable when MCP is off.
+- A later release may wire a read-only MCP profile into doctor or a dedicated
+  CLI after eval coverage; mutating tools stay out until gated separately.
+
+## Alternatives considered
+
+- **Kernel MCP host:** rejected; violates ADR-0026 and forces an optional
+  network dependency into deterministic gates.
+- **Wire MCP into tick/deliver now:** rejected for 0.6.0; needs allowlist
+  provenance, audit events, and eval coverage first.

package/docs/LOOP.md

@@ -139,6 +139,20 @@ Start from [`loop.config.example.yaml`](../loop.config.example.yaml) at the pack
 | `machine.slots` | `sampleMachine` + `adaptiveConcurrency`, free RAM reserve, WSL cap, running worktrees | no — 0 free slots is a warning, not a failure |
 | `linear.queue` | `orca linear list-issues` per configured state, filtered and ordered locally | yes — an unreachable Linear blocks |
 
+
+## Dynamic model routing
+
+`models.routing.mode` controls how the loop picks `provider/model` for each role:
+
+| Mode | Behaviour |
+|---|---|
+| `tiers` (default) | YAML declaration order; usage only as available/unavailable (0.6 behaviour). |
+| `hybrid` | Keep tier bands (quality policy); **inside a tier** pick the provider with the most remaining Orca usage. |
+| `dynamic` | Flatten all YAML candidates; rank by remaining usage (YAML order is a soft tie-break). |
+| `catalog` | Discover models from CLI lists (`grok models`), builtin catalog, and optional [Artificial Analysis](https://artificialanalysis.ai/) (cached under `stateDir/catalog/`), then rank by usage + quality band (`models.roles.*.quality`). |
+
+Providers still need a `models.providers.<id>` block (`bin` / `tui` / `headless`) — Orca cannot invent argv. The doctor warns when Orca shows an integration that is not declared.
+
 ## Model routing
 
 `models.<role>` is a list of tiers; each tier is a list of `provider/model`. Tiers are tried in order and, inside a
@@ -160,6 +174,87 @@ When a provider runs out of usage the loop records a cooldown in `<stateDir>/pro
 inside a WSL distro (host Defender load is invisible there). The loop never lowers a running worker; it only
 decides whether to start another.
 
+## Integrations used by the loop
+
+The loop talks to the rest of AgentsKit through **adapters and CLI seams**, not by importing those packages into
+the published dependency tree. `@agentskit/harness` stays dependency-light (`commander`, `ink`, `react`, `yaml`,
+`zod`); Doc Bridge and Code Review are optional at runtime.
+
+### Doc Bridge (context for the orchestrator)
+
+| Item | Detail |
+|---|---|
+| Adapter | `src/adapters/doc-bridge.ts` → `createDocBridgeContextProvider` |
+| Loop wiring | `resolveDocContext` in `src/loop/contract.ts` |
+| Trigger | `.doc-bridge/index.json` exists under `project.root` |
+| Knob | `contract.maxContextReferences` (default `6`; `0` disables) |
+| Behaviour | Query is `"<issue id> <title>"`. Up to N deterministic references are appended to the orchestrator prompt. Missing or malformed index → **no refs** (loop continues). |
+| Boundary | No `@agentskit/doc-bridge` import; the adapter only reads the local index contract ([ADR-0003](ADR-0003-doc-bridge-context-binding.md)). Index build/refresh stays with Doc Bridge (`pnpm docs:bridge:index` in repos that use it). |
+
+### Code Review (`agentskit-review`)
+
+| Item | Detail |
+|---|---|
+| Adapter | `src/adapters/code-review.ts` → `runCodeReview` / `buildReviewArgv` |
+| Loop wiring | Deliver stage, after CI is green and the head has not been reviewed |
+| CLI | `delivery.review.cli` (default `agentskit-review` from `@agentskit/code-review`) |
+| Knobs | `mode` (`trusted-local` recommended so CLI logins work), `transport` (`headless` for current Grok CLI), `profile` (`fast` fits a 600 s Orca stage; `full` needs batching), `votes`, `concurrency`, `minSeverity`, `deadlineMs`, `maxCalls`, `post` |
+| Provider/model | First available candidate from `models.reviewer` tiers |
+| Verdicts | exit `0` → clean (merge path); `1` → findings ≥ floor (fix round); `2` / timeout / incomplete → wait and retry |
+| Boundary | Argv + `--result` file only; the harness never embeds the review SDK. |
+
+### Eval battery (package quality, not the live loop)
+
+`evals/manifest.json` plus `runEvalBattery` / `pnpm test:eval-battery` score harness components (`doc-bridge`,
+`code-review`, `orca-worktree`, `memory`, …) under a frozen fixture provider. That is the **library** eval gate.
+The live SDLC loop does **not** call the eval battery on every PR; it calls `agentskit-review` on each head.
+
+Loop-focused Vitest coverage lives under `test/loop-*.test.ts`, `test/review.test.ts`, and `test/doc-bridge.test.ts`.
+
+## AgentsKit ecosystem map (what the harness has vs what the loop uses)
+
+Legend: **Loop** = wired into `ak-harness loop …` today · **Kernel** = public API / contracts available to callers ·
+**Compat** = pinned in `compatibility/manifest.json` · **Absent** = no seam yet.
+
+| Capability | Status | Where | Notes for SDLC |
+|---|---|---|---|
+| **Doc Bridge** | Loop + Kernel + Compat | `adapters/doc-bridge.ts`, contract stage | Optional context on contract freeze. Playbook/docs surfaces appear as Doc Bridge scopes when the index includes them. |
+| **Code Review** | Loop + Compat | `adapters/code-review.ts`, deliver | Live adversarial review before auto-merge. |
+| **Orca / Linear / GitHub** | Loop | `adapters/orca-cli.ts`, `linear-orca.ts`, `github-cli.ts` | Scheduler, worktrees, queue, PR merge. |
+| **Coding-agent CLIs** | Loop | `adapters/providers.ts`, `models.*` tiers | Claude / Codex / Grok / OpenCode via TUI + headless templates — not an AgentsKit agent registry. |
+| **Memory** | Loop + Kernel + Compat | `kernel/memory.ts`, `loop/memory.ts` | Approved learnings (`loop learning promote`) shrink Doc Bridge/issue text in contract + brief. |
+| **Eval** | Kernel + Compat | `kernel/eval.ts`, `evals/manifest.json` | Battery + `assessAgentEval`. **Not run inside tick/deliver**. |
+| **Runtime (process / Docker)** | Kernel | `execution/runtime.ts` | Tool runtimes with attestation for kernel/agent sessions. Loop workers run as Orca terminals + provider TUIs instead. |
+| **Plugin / context registry** | Kernel | `kernel/plugins.ts`, `CONTEXT_PROVIDER_SLOT` | Typed slots so Doc Bridge / Playbook / custom providers plug in without kernel changes. Loop uses Doc Bridge directly today. |
+| **Playbook practices** | Loop (via Doc Bridge scopes) | `contract.briefScopes` → worker brief | Titles/paths for `playbook` / `for-agents` scopes listed in the brief when indexed. |
+| **RAG (`@agentskit/rag` / os-rag)** | Kernel adapter (opt-in) | `adapters/rag-context.ts` | Argv/`ContextProvider` seam; enable via `rag.enabled` + `contract.contextProviders`. No hard dep. |
+| **AgentsKit agent registry** | Composition (opt-in file) | `loop/agent-registry.ts` | Optional `agents.registry.yaml` role→TUI/argv overlay. Not OS marketplace. |
+| **MCP / event bridge** | Kernel adapter + ADR-0028 | `adapters/mcp.ts` | Policy-gated tool bridge; **not** wired into tick/deliver in 0.6.0. |
+| **`@agentskit/core` / `@agentskit/eval` / `@agentskit/memory`** | Compat pins only | `compatibility/manifest.json` | Upstream packages are compatibility-tested; harness does **not** depend on them at runtime. Callers adapt them through the seams above. |
+
+Compatibility report for 0.4.0 was **fail-closed** on code-review quality baselines and the no-Harness pilot cohort; see `compatibility/report.md`. Refresh after each release (qualification + pinned revisions).
+
+## What can be added to help the SDLC
+
+**0.6.0 ships the backlog below** (memory, doctor freshness/review probe, brief scopes, deliver smoke, agent registry, RAG provider, MCP seam+ADR, Docker verify config, weekly retro automation). Remaining work is dogfooding and deeper MCP/OS registry integrations.
+
+Ordered by leverage for a keep-pushing loop (config/adapters first; no kernel redesign required for the early items).
+
+| Priority | Addition | Why it helps | Suggested shape |
+|---|---|---|---|
+| P0 | **Memory in contract + brief** | Stop re-deriving the same repo facts; carry approved decisions across tickets | On contract freeze / worker brief, `recall` from an `AgentMemoryAdapter` (backed by `@agentskit/memory` or a file store under `<stateDir>`). Persist only human-`promoteLearnings` / retro-approved records. |
+| P0 | **Doc Bridge freshness gate** | Orchestrator context goes stale when the index is old | Doctor check: index exists, `contentHash` age, optional `docs:bridge:index` hint when missing. |
+| P1 | **Playbook / for-agents snippets in the brief** | Workers skip repo conventions that already exist as docs | Resolve Doc Bridge (or a Playbook `ContextProvider`) with scopes `playbook` + `for-agents` and render a bounded “must follow” block into `renderWorkerBrief`. |
+| P1 | **Eval smoke on deliver (optional)** | Catch harness/component regressions before merge on harness itself | Config flag to run a **bounded** subset of `evals/manifest.json` (or project `ak-verify`) as an extra deliver gate — not a full battery on every product PR. |
+| P1 | **Review transport/doctor** | Incomplete reviews burned the pilot (login / model id / stage budget) | Doctor probes `agentskit-review --help`, configured transport, and a one-lens trusted-local dry call; surface `review-tool-errors` in retro. |
+| P2 | **RAG context provider** | Large codebases exceed Doc Bridge’s deterministic top-N | New adapter implementing `ContextProvider` over `@agentskit/rag` / os-rag; same freeze-into-contract rules as Doc Bridge (hash + cap). |
+| P2 | **AgentsKit agent/skill registry** | Reuse named agents (reviewer, security, docs) instead of free-form CLI templates | Optional `agents.registry` file or OS dispatcher lookup → map role → argv/TUI. Keep fail-closed when the registry entry is missing. |
+| P2 | **MCP tools behind policy** | Controlled access to issue trackers, browsers, internal APIs | MCP adapter + `createPolicyGate` allowlist; record hashed tool events. Requires ADR (MODULE-BOUNDARIES already flags this). |
+| P3 | **Kernel runtime for untrusted tools** | Sandbox one-off scripts the worker must not run on the host | Offer Docker tool runtime as an opt-in for verify commands; loop default stays Orca worktree. |
+| P3 | **Weekly Linear retro automation** | Close the continuous-improvement loop without a human remembering `loop retro` | Orca schedule → `loop retro --learnings` → comment on a fixed Linear issue (project vs harness sections already split). |
+
+Non-goals for the loop: embedding LLM SDKs, reading API key values from config, or making `@agentskit/*` hard dependencies of the published package. New ecosystem pieces enter as adapters/plugins with argv/timeouts and eval coverage ([ADR-0026](ADR-0026-kernel-adapters-boundary.md), [ADR-0027](ADR-0027-keep-pushing-loop.md)).
+
 ## Boundaries
 
 Adapters (`src/adapters/command.ts`, `orca-cli.ts`, `providers.ts`, `linear-orca.ts`) depend on kernel contracts

package/docs/MODULE-BOUNDARIES.md

@@ -149,7 +149,10 @@ Orca, and tracking. No adapter implementation is re-exported wholesale.
 
 ## External integration inventory
 
-| Integration | Current location | Side effects | 0.4.0 boundary |
+How these seams are wired into the keep-pushing loop (and what is still only a kernel contract) is summarized in
+[`LOOP.md`](LOOP.md#integrations-used-by-the-loop) and the [AgentsKit ecosystem map](LOOP.md#agentskit-ecosystem-map-what-the-harness-has-vs-what-the-loop-uses).
+
+| Integration | Current location | Side effects | Boundary |
 | --- | --- | --- | --- |
 | Doc Bridge | `src/adapters/doc-bridge.ts` | Reads a local index | Keep behind `ContextProvider`; measure context hit/quality separately. |
 | Orca | `src/adapters/orca.ts` | None; produces argv and lifecycle projections only | Keep lease/worktree/issue-lock/SHA planning provider-neutral; execution belongs to the orchestrator. |
@@ -161,8 +164,10 @@ Orca, and tracking. No adapter implementation is re-exported wholesale.
 | Process runtime | `src/execution/runtime.ts` | Starts child processes | Execution support; policy and evidence gates remain kernel decisions. |
 | Docker runtime | `src/execution/runtime.ts` | Starts Docker containers | Optional sandbox selected by config, never a mandatory kernel dependency. |
 | LLM provider/model | Caller/plugin | Provider call and token spend | Bind provider/model in experiment metadata; do not embed SDKs in kernel. |
-| Memory backend | Caller/plugin; `memory.ts` contract | Backend reads/writes | Keep record validation in kernel; backend adapter owns persistence. |
-| MCP/event bridge | Not implemented | Future network/event effects | Add as adapters only after a separate ADR and eval coverage. |
+| Memory backend | Caller/plugin; `memory.ts` + `loop/memory.ts` | Backend reads/writes; loop file store under `stateDir` | Only human-promoted approved records enter loop prompts; prefer-over-DocBridge shrinks tokens. |
+| RAG context | `src/adapters/rag-context.ts` | Optional argv / injected query | No hard `@agentskit/rag` dep; same ContextProvider freeze rules as Doc Bridge. |
+| Agent registry (file) | `src/loop/agent-registry.ts` | Reads YAML | Role→TUI/argv overlay; fail closed only when `agents.requireRegistry`. |
+| MCP/event bridge | `src/adapters/mcp.ts` + [ADR-0028](ADR-0028-mcp-adapter-boundary.md) | Policy-gated tool calls | Adapter-only in 0.6.0; **not** wired into tick/deliver. |
 
 ## Review status
 

package/loop.config.example.yaml

@@ -37,6 +37,15 @@ linear:
   needsInfoLabel: needs-info
 
 models:
+  # routing.mode: tiers (default) | hybrid | dynamic | catalog
+  # hybrid = keep quality tiers, pick highest remaining usage inside each tier
+  # catalog = discover models via CLI/builtin/(optional) Artificial Analysis + usage
+  routing:
+    mode: hybrid
+    usageMetric: max
+    preferKnownUsage: true
+  # roles: { orchestrator: { quality: frontier }, builder: { quality: balanced }, watcher: { quality: fast } }
+  # catalog: { sources: [cli, builtin], artificialAnalysis: { enabled: false } }
   # Tiers, in order. Inside a tier the first provider with usage available wins.
   # Tier 1 = subscriptions, tier 2 = cheap hosted, tier 3 = last resort.
   orchestrator: [[codex/gpt-5.6-sol, claude/opus], [opencode/opencode-go/glm-5.3], [grok/grok-4-fast]]
@@ -90,8 +99,13 @@ delivery:
     profile: fast                    # one bounded pass over required lenses; `full` needs batching to fit a 600 s stage
     votes: 1
     concurrency: 4
-    minSeverity: med                 # nit < med < high < blocker; findings at/above this block auto-merge
+    minSeverity: med
+    # doctorProbe: help                 # nit < med < high < blocker; findings at/above this block auto-merge
     deadlineMs: 600000
+  # smoke:
+  #   enabled: false
+  #   kind: verify-argv
+  #   argv: [pnpm, test]
   merge:
     auto: true
     method: squash
@@ -104,18 +118,49 @@ delivery:
   cleanupWorktree: true              # remove the Orca worktree after merge
   returnState: Todo                  # where a stuck/blocked issue goes back to
 
+
+# --- 0.6.0 optional SDLC seams (defaults keep prior behaviour) ---
+memory:
+  enabled: false                     # set true to recall approved learnings into contract/brief (token reduction)
+  # backend: file
+  # preferOverDocBridge: true        # shrink Doc Bridge refs when memory covers them
+  # maxRecall: 5
+  # maxBlockChars: 1200
+  # categories: [adjustment]         # only these LearningRecord categories become memory on promote
+  # issueCharsWithMemory: 4000
+
+# agents:
+#   registryPath: agents.registry.yaml
+#   requireRegistry: false
+
+# rag:
+#   enabled: false
+#   queryArgv: []                    # argv printing ContextSnapshot JSON on stdout
+#   maxReferences: 4
+
+# mcp:                               # public API / ADR-0028 — not wired into tick/deliver
+#   enabled: false
+#   allowTools: []
+
 contract:
   maxIssueChars: 12000               # issue text rendered into the orchestrator prompt (as untrusted data)
   timeoutMs: 300000
   maxContextReferences: 6            # Doc Bridge hits appended when .doc-bridge/index.json exists
   reuseHours: 72                     # cached contract reused for this long
+  docBridgeMaxAgeHours: 168           # doctor warns when .doc-bridge/index.json is older
+  requireDocBridge: false
+  briefScopes: [playbook, for-agents] # Doc Bridge scopes listed in the worker brief
+  maxBriefReferences: 4
+  contextProviders: [doc-bridge]      # add rag when rag.enabled
 
 schedule:
   tick: "*/5 * * * *"
   deliver: "*/10 * * * *"
   precheckTimeoutSec: 120
   harnessCommand: ak-harness         # how the Orca automation runs the harness inside the workspace
-  namePrefix: loop                   # automations: loop-tick, loop-deliver
+  namePrefix: loop                   # automations: loop-tick, loop-deliver[, loop-retro]
+  # retro: weekly                     # with retroIssue, installs <prefix>-retro
+  # retroIssue: AGE-0                 # Linear issue that receives the digest comment
   runner: precheck                   # the stage runs inside Orca's precheck command; no agent session is ever opened
   stageTimeoutSec: 600               # Orca caps prechecks at 600 s; one tick/deliver run must fit
   # provider: claude                 # Orca agent that executes the automation prompt (default: watcher role)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agentskit/harness",
-  "version": "0.5.0",
+  "version": "0.7.0",
   "description": "Portable, evidence-backed development harness for coding agents.",
   "type": "module",
   "bin": {

package/release/manifest.json

@@ -2,7 +2,7 @@
   "type": "agentskit-harness-release-manifest",
   "schemaVersion": 1,
   "package": "@agentskit/harness",
-  "version": "0.5.0",
+  "version": "0.7.0",
   "channel": "latest",
   "sourceRevision": "b8cfe596e5f30f2c373b89f00f271079b296feaf",
   "requiredChecks": [

package/release/notes.md

@@ -1,3 +1,11 @@
+# 0.7.0 release candidate
+
+Dynamic Orca-aware model routing (hybrid/dynamic/catalog) with remaining-usage ranking and a living model catalog (CLI + builtin + optional Artificial Analysis).
+
+# 0.6.0 release candidate
+
+Keep-pushing loop gains approved memory (token reduction + continuous improvement), Doc Bridge freshness/review doctor probes, brief guidance scopes, optional deliver smoke, agent registry YAML, RAG context provider, MCP adapter seam (ADR-0028), Docker verify config, and weekly Linear retro automation. Config defaults preserve 0.5.0 behaviour.
+
 # 0.5.0 release candidate
 
 This release adds the keep-pushing SDLC loop (`ak-harness loop …`): project config with a per-machine overlay,