@tangle-network/agent-runtime 0.79.3 → 0.79.4

package/README.md

@@ -1,371 +1,94 @@
 # @tangle-network/agent-runtime
 
-The shared task-lifecycle skeleton for agents. It runs an agent (a chat turn, a one-shot task, or a multi-attempt loop), captures every run as a trace, and feeds those traces into eval-gated self-improvement.
+The engine Tangle's AI agents run on. It runs an agent — a **chat turn**, a **one-shot task**, or a **team of agents** working toward a goal — records every run, and uses those records to **measure and improve** agents against real pass/fail checks.
 
-It owns the lifecycle, the loop kernel, and the **optimization suite** — `Environment` + `Strategy` +
-`runBenchmark` + `runStrategyEvolution`, the published surface for measuring and evolving how an agent
-spends compute against a deployable check. It delegates domain behavior (models, tools, knowledge) to
-adapters, scoring statistics and the ship gate to [`@tangle-network/agent-eval`](https://www.npmjs.com/package/@tangle-network/agent-eval), and sandboxed long-running execution to [`@tangle-network/sandbox`](https://www.npmjs.com/package/@tangle-network/sandbox).
+One loop, used three ways. Domain behavior (models, tools, knowledge) plugs in as adapters; the scoring statistics and the ship decision come from [`@tangle-network/agent-eval`](https://www.npmjs.com/package/@tangle-network/agent-eval); sandboxed execution from [`@tangle-network/sandbox`](https://www.npmjs.com/package/@tangle-network/sandbox).
 
 ```bash
 pnpm add @tangle-network/agent-runtime @tangle-network/agent-eval @tangle-network/sandbox
 ```
 
-## The model
-
-One recursive `Agent` atom, run at two timescales, over many tasks. `docs/architecture.md` is the canonical spine. The short version:
-
-1. **One atom.** `driver`, `worker`, `selector`, and `coordinator` are not separate types. They are what a single `Agent` returns from `act`. The recursion bottoms out at execution.
-2. **Two timescales, one machinery.** The same loop runs at inference time (steer a worker over k attempts) and at optimization time (search the steer or the prompt with GEPA, gated on a held-out split).
-3. **A benchmark is an adapter.** A new task is a loader plus a worker plus a judge. The loop, the drivers, the corpus, and the selector are the shared spine, written once.
-4. **The selector is not the judge.** At inference time the selector picks which answer to return without seeing the judge's verdict. The judge is write-only. A steer may read the trace but never the verdict (the firewall that keeps the loop from gaming its own score).
-
-## Getting started
-
-Three entry points, by what you're doing:
+## What you do with it
 
 | You want to… | Call |
 |---|---|
-| Run one **product chat turn** (gtm/legal/tax/creative run this in prod) | `handleChatTurn(...)` |
-| Have a **supervisor drive a team of agents** to a goal — any harness, any number of sandboxes | `supervise(profile, task, { budget, backend? })` |
-| **Self-improve** an agent, certified on a held-out gate | `improve(profile, findings, { surface, gate, … })` |
+| Run a **chat turn** — what every product agent does in production | `handleChatTurn(...)` |
+| Have one agent **supervise a team of agents** toward a goal | `supervise(profile, task, opts)` |
+| **Improve** an agent and prove the gain on fresh tasks | `improve(profile, findings, opts)` |
 
 ### Run a chat turn
 
-Every product agent is a `handleChatTurn` call inside a route — what the gtm, creative, legal, and tax products run in production:
+A product agent is one `handleChatTurn` call inside a route. You give it how to produce the response and how to persist it; it streams, traces, and persists.
 
 ```ts
 import { handleChatTurn } from '@tangle-network/agent-runtime'
 
-export async function POST({ request, env, ctx }: { request: Request; env: Env; ctx: ExecutionContext }) {
-  const { workspaceId, threadId, userMessage } = await request.json()
-  const box = await ensureWorkspaceSandbox(workspaceId)
-
-  const result = handleChatTurn({
-    identity: { tenantId: workspaceId, sessionId: threadId, userId: 'demo', turnIndex: 0 },
-    hooks: {
-      produce: () => ({
-        stream: box.streamPrompt(userMessage),
-        finalText: () => box.lastResponse(),
-      }),
-      persistAssistantMessage: async ({ identity, finalText }) => env.db.insertMessage(identity, finalText),
-      traceFlush: () => env.traceSink.flush(),
-    },
-    waitUntil: ctx.waitUntil.bind(ctx),
-  })
-  return new Response(result.body, { headers: { 'content-type': result.contentType } })
-}
+const result = handleChatTurn({
+  identity: { tenantId, sessionId: threadId, userId, turnIndex: 0 },
+  hooks: {
+    produce: () => ({ stream: box.streamPrompt(userMessage), finalText: () => box.lastResponse() }),
+    persistAssistantMessage: async ({ identity, finalText }) => db.insertMessage(identity, finalText),
+  },
+  waitUntil,
+})
+return new Response(result.body, { headers: { 'content-type': result.contentType } })
 ```
 
-That is the common case for a single product agent. The other two entry points are below.
+### Supervise a team of agents
 
-### Run a supervisor (one call)
-
-A supervisor authors and drives a team of workers to a goal. The brain is resolved from the profile's `harness`: `null` → an in-process router tool-loop; `'claude-code'`/`'opencode'`/`'codex'` → a sandboxed coding harness driving the coordination verbs. The scaffolding (blobs / per-worker budget / journal / executors / depth) is defaulted.
+One supervisor spawns and steers workers toward a goal. Where the workers run (an in-process loop, or a sandboxed coding harness) is one data value; the budget, journaling, and stopping are handled for you.
 
 ```ts
 import { supervise } from '@tangle-network/agent-runtime/loops'
 
 const result = await supervise(
   { name: 'supervisor', harness: null, systemPrompt: 'Delegate to workers; do not solve the task yourself.' },
-  'Implement the feature and make CI green.',
-  { budget, router, backend }, // `backend` = where the workers run (one data value: router-tools | sandbox+harness | bridge)
+  'Implement the feature and make the tests pass.',
+  { budget, router, backend }, // backend = where workers run: router-tools | sandbox+harness | bridge
 )
 ```
 
-See `examples/supervise/` for the full one-call entry; `examples/supervisor-loop/` for the per-backend seams.
-
-### Self-improve an agent
+### Improve an agent
 
-`improve` is the one pluggable RSI verb: it optimizes a `surface` of the profile (prompt / skills / code) with the generator defaulted from the surface (GEPA for prompts, skillOpt for skills, or bring your own), certified on a frozen holdout.
+`improve` optimizes one part of an agent (its prompt, skills, or code) and **only ships a change if it beats the current agent on tasks it never practiced on** — so registering an agent for self-improvement can never make it worse.
 
 ```ts
 import { improve } from '@tangle-network/agent-runtime'
 
 const { profile, shipped, lift } = await improve(baseProfile, findings, {
-  surface: 'prompt',
-  gate: 'holdout',         // certified on a held-out split, never the training set
-  scenarios, judge, agent, // how to MEASURE the profile under a candidate surface
-})
-```
-
-Everything below is the substrate these three sit on: multi-attempt loops, delegation, optimization, and the telemetry that makes them auditable.
-
-### The system in plain language
-
-The internal docs use the project's own vocabulary; this is the same thing without it, for a colleague meeting the project cold. Five sentences:
-
-1. We have tasks with **automatic pass/fail checks** — tests you can run, answer keys you can verify mechanically.
-2. An AI attempts each task a fixed number of times under different **retry policies**: "try 3 times, keep the best", "try, get feedback, try again", and so on.
-3. We compare policies **fairly**: identical tasks, identical attempt budgets, paired statistics, judged on fresh tasks no tuning step ever saw.
-4. The distinctive part: the AI also **writes new retry policies itself**, as short programs, and they enter the same tournament under the same rules as human-written ones.
-5. Every dollar and second is metered, so "better" can also mean "**equally good but cheaper**" — and that claim is statistically testable, not vibes.
-
-The load-bearing core is six pieces: task-with-check · retry policy · the tournament runner · the AI policy-writer · the statistical promotion gate · crash-resume. Everything else is a **fairness rule** or an **experiment on the menu** (a configuration, not a machine part).
-
-| Project term | Plain English | Standard concept |
-|---|---|---|
-| `Environment` | a task domain: open it, act with tools, check the result | RL environment / gym |
-| shot | one attempt | — |
-| steering / `refine` | feedback injected between attempts | self-refinement |
-| `authorStrategy` | the AI writes a new retry policy as a program | program synthesis |
-| evolution / generations | write candidates → tournament → keep the champion | evolutionary search |
-| harness-verified scoring | never trust a policy's self-reported score; recompute it from the attempts actually run | measurement hygiene |
-| selector ≠ judge (the firewall) | the feedback-giver never sees the answer key or the score | no reward leakage |
-| conserved budget pool | every policy gets exactly the same attempt budget; overspending is structurally impossible | compute-matched comparison |
-| holdout / fresh slice | final judging happens on tasks no tuning step ever touched | train/test split |
-| `promotionGate` | a seeded paired bootstrap must show the win is real before anything is "better" | inferential statistics |
-| non-inferiority mode | prove "not worse on quality AND significantly cheaper" | clinical-trials statistics |
-| reproducer certificate | a fresh AI re-builds the winner from a short description; a failed rebuild means the win was memorization, not method | description-length test |
-| waterfall | a per-step timeline of the run: seconds, dollars, tokens per step | distributed tracing |
-
-**Honest weaknesses:** mostly one domain family per claim so far (cross-domain replication is configuration, not new code); small holdouts (12–16 tasks) mean only effects ≳6pp are detectable; and the homegrown vocabulary is heavier than the machine it names — hence this section.
-
-## Which entry point do I reach for?
-
-| You want to | Reach for | Subpath |
-|---|---|---|
-| Run a production chat turn (most products) | `handleChatTurn` | root |
-| Declare an agent (profile, surfaces, adapters) | `defineAgent` | `/agent` |
-| Run a one-shot task with verification and eval | `runAgentTask` | root |
-| Compare optimization strategies on YOUR domain (5 hooks) | `runBenchmark` + `defineStrategy` | `/loops` |
-| Let the system author + evolve its own strategies, gated | `runStrategyEvolution` · `authorStrategy` · `promotionGate` | `/loops` |
-| Run a multi-attempt loop with a custom driver | `runLoop` + an inline `Driver` | `/loops` |
-| Drive one agent profile from another (the canonical driver) | `createCoordinationTools` over `Supervisor` (`/loops`) | `/mcp` |
-| Delegate a disciplined loop by mode (code, research, ...) | `runDelegatedLoop` or `agent-runtime-loop` | root |
-| Build code reliably (reviewed, gated) | `createDefaultCoderDelegate` | `/mcp` |
-| Grow a knowledge base with only grounded facts | `createKbGate` | `/mcp` |
-| Improve a prompt safely (identity-gated) | `selfImprove` | `@tangle-network/agent-eval/contract` |
-| Ship loop traces to a GenAI viewer | `buildLoopOtelSpans` plus `createOtelExporter` | root |
-| Expose delegation as MCP tools to a sandbox agent | `createMcpServer` or `agent-runtime-mcp` | `/mcp` |
-| Persist a run plus its cost ledger | `startRuntimeRun` | root |
-
-## The optimization suite
-
-The canonical surface. A domain is an `Environment` (five hooks: `open`/`tools`/`call`/`score`/`close`);
-a **strategy** is how a compute budget is spent to beat the domain's own deployable check. Two
-built-ins (`sample` = best-of-N, `refine` = critique-and-continue) plus `defineStrategy` to compose
-your own from two steps — and `authorStrategy`, where the system writes new strategies from its own
-per-task losses:
-
-```ts
-import { defineStrategy, runBenchmark, sample, refine } from '@tangle-network/agent-runtime/loops'
-
-const doubleCheck = defineStrategy('double-check', async ({ shot, critique }) => {
-  const first = await shot()
-  const steer = first ? await critique(first.messages) : null
-  const second = steer ? await shot({ messages: first?.messages, steer }) : null
-  const score = Math.max(first?.score ?? 0, second?.score ?? 0)
-  return { score, resolved: score >= 1, completions: 2, progression: [first?.score ?? 0, score], shots: 2 }
-})
-
-const report = await runBenchmark({ environment, tasks, worker, strategies: [sample, refine, doubleCheck], budget: 3 })
-report.perTask // the losses table an author/optimizer consumes
-report.pareto  // the (score, $) frontier
-```
-
-The measurement invariants are structural, not advisory: every strategy spends through a conserved
-budget pool (equal compute by construction), the deliverable score is **harness-verified** from the
-shots actually brokered (a body cannot fabricate a win), and the critic is firewalled from the check
-(selector ≠ judge). `runStrategyEvolution` runs the multi-generation search — populations of authored
-candidates, cost-aware champion selection, a phase ledger with resume, and ONE promotion decision via
-`promotionGate` (seeded paired bootstrap) on a holdout slice the search never touched.
-`createVerifierEnvironment` adapts answer-shaped domains (one `check` function); `createMcpEnvironment`
-adapts any MCP server. The consumer surface — loops as a service with a CLI, detached runner, and MCP
-server — lives in the [`loops`](https://github.com/drewstone/loops) repo; the experiment harness and
-evidence ledger live in [`bench/HARNESS.md`](./bench/HARNESS.md).
-
-## The loop kernel
-
-`runLoop` is a topology-agnostic kernel. Each iteration spawns a sandbox on an `AgentRunSpec`, decodes the output, validates it, and asks a driver what to do next. The driver owns topology. The validator owns scoring. The kernel owns iteration accounting, concurrency, cost and token aggregation, and trace emission.
-
-```ts
-import { runLoop, type Driver } from '@tangle-network/agent-runtime/loops'
-
-const driver: Driver<Task, Output, 'pick-winner' | 'fail'> = {
-  plan: async (task, history) => (history.length === 0 ? [task, task] : []), // fan out, then stop
-  decide: (history) => (history.some((i) => i.verdict?.valid) ? 'pick-winner' : 'fail'),
-}
-
-const result = await runLoop({
-  driver,                                       // the driver owns topology; the kernel owns accounting
-  agentRuns: [claudeSpec, codexSpec, glmSpec], // heterogeneous: one harness per branch
-  output,                                       // events to typed Output
-  validator,                                    // Output to { valid, score }
-  task,
-  ctx: { sandboxClient: sandbox },
+  surface: 'prompt',        // what to optimize: prompt | skills | code
+  gate: 'holdout',          // certified on a held-back exam, never the practice set
+  scenarios, judge, agent,  // how to measure a candidate
 })
-result.winner // highest-scoring valid attempt
-```
-
-A `Driver` is `plan` (emit the round's `Task[]` — `[]` ends the loop) plus `decide` (the terminal
-`Decision` over the history). Topology is orthogonal to harness: the driver never names a backend,
-and the kernel's `agentRuns` decide which harness runs each branch. See `examples/coder-loop` for a
-fixed-shape inline `Driver`. For recursive work prefer the **agent-driver** — an `AgentProfile`
-driving another via `createCoordinationTools` (`/mcp`) over the budget-conserving `Scope`/`Supervisor`
-core (`/runtime`) — plus the `personify` combinators (`fanout`, `loopUntil`, `panel`, `pipeline`) and
-`runPersonified` on that same core.
-
-## Self-improvement
-
-The same machinery, run at the optimization timescale.
-
-The one entry point is agent-eval's **`selfImprove`** (`@tangle-network/agent-eval/contract`). It runs a closed loop over any text/config surface, identity-gated by construction: it evaluates, proposes candidates (default `gepaDriver`), and a held-out gate ships a winner only if it beats the baseline. `result.winner.surface` is the baseline unless `result.gateDecision === 'ship'`, so registering a surface for optimization can never regress it.
-
-```ts
-import { selfImprove } from '@tangle-network/agent-eval/contract'
-
-const result = await selfImprove({
-  baselineSurface: CURRENT_SYSTEM_PROMPT,
-  agent: (surface, scenario, ctx) => runYourThing(surface, scenario),
-  scenarios,
-  judge,
-  budget: { holdoutScenarios, generations: 3 },
-  llm: { baseUrl, apiKey, model: 'claude-sonnet-4-6' },
-})
-// result.winner.surface is the safe one — the baseline unless gateDecision === 'ship'
-```
-
-agent-runtime contributes the runtime-specific pieces: the **CODE-surface `improvementDriver`**
-(`/improvement`) — a git-worktree mutator you pass to `selfImprove` as `driver` to optimize code
-instead of a string — and **`runStrategyEvolution`** (`/loops`), the multi-generation search over
-STRATEGY space: the system reads its own per-task losses, authors candidate strategies as code,
-plays them against the incumbent at equal budget, and a seeded statistical gate decides promotion
-on a never-touched holdout slice.
-
-`runAnalystLoop` (`/analyst-loop`) mines real run traces into findings; `createAnalystDriverHook` feeds those findings to a dynamic-driver planner via `PlannerContext.analyses`, with a firewall (`assertTraceDerivedFindings`) that rejects any finding derived from a judge verdict. Production intake — turning real run traces into the corpus `selfImprove` optimizes against — is agent-eval's `analyzeRuns` / `partitionRunsByAuthoringModel` (`/contract`).
-
-## Delegated loops
-
-`runDelegatedLoop` is one entrypoint a worker agent or a scheduled routine calls to run a disciplined loop in a chosen mode, over the hardened engines below. It fails loud on an unwired mode; a thrown engine is captured as `{ ok: false }`, so unattended runs record rather than crash.
-
-```ts
-import { runDelegatedLoop, worktreeLoopRunner, researchLoopRunner, type DelegatedLoopRegistry } from '@tangle-network/agent-runtime'
-
-const registry: DelegatedLoopRegistry = {
-  code: worktreeLoopRunner({ repoRoot: '/repo', taskPrompt: 'fix the flaky retry test', harnesses, budget }),
-  research: researchLoopRunner({ research, gate: { selfArtifactKinds: ['spec'] }, maxRounds: 3 }),
-}
-const result = await runDelegatedLoop('code', registry)
-```
-
-Modes: `code`, `review`, `research`, `audit`, `self-improve`, `dynamic`. The `agent-runtime-loop` bin runs the registry from a cron or routine and exits 0 (ok), 1 (recorded failure), or 2 (usage or config error).
-
-`worktreeLoopRunner` (`code` mode, the generic recursive path) authors one `AgentProfile` per harness and runs them as a `worktreeFanout` (each leaf `gateOnDeliverable`), winner by the shared valid-only selector. The sandbox-session counterpart is `detachedSessionDelegate` (`/mcp`): it drives the in-box harness over a `SandboxClient` to a mechanically-validated patch, with default-on safety gates — no-op rejection, an always-on secret-path floor (`.env`, keys, wallets), an optional `reviewer` gate, and a `winnerSelection` policy. Its worker profile is a parameter the caller authors (`workerProfile`); omit it for a minimal model-only default.
-
-The knowledge-base gate (`createKbGate`, `/mcp`) is fail-closed: a fact's `verbatimPassage` must appear in its `sourceText`, the asserted value must be in the passage, and citations cannot point at self-generated artifacts. `researchLoopRunner` wraps it with a correct-on-veto loop that re-researches the vetoed gaps up to `maxRounds`, then returns the unverified ones rather than dropping them.
-
-## Tracing
-
-`runLoop` emits a structured event stream. `buildLoopOtelSpans` turns it into a nested, real-duration span tree that any GenAI trace viewer (Phoenix, Langfuse, Grafana Tempo, Tangle Intelligence) renders natively. Attributes follow the current GenAI semantic conventions (`gen_ai.operation.name`, `gen_ai.agent.name`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`) plus a `tangle.loop.*` extension for the topology (move kind and rationale, edge lineage, verdict, placement, cost).
-
-```ts
-import { buildLoopOtelSpans, createOtelExporter } from '@tangle-network/agent-runtime'
-
-const exporter = createOtelExporter() // reads OTEL_EXPORTER_OTLP_ENDPOINT
-for (const span of buildLoopOtelSpans(loopEvents, traceId)) exporter?.exportSpan(span)
-await exporter?.flush()
-```
-
-The shape: `loop` to `loop.round` (move plus rationale) to `loop.iteration` (agent, usage, verdict, cost, parent edge).
-
-## MCP delegation server
-
-Expose the delegation tools to a sandbox coding agent: the generic `delegate` verb (one intent → a supervisor that authors + drives its own worker, returns the delivered output with its real spend) plus the queue-bound `delegate_feedback`, `delegation_status`, `delegation_history` (and `delegate_ui_audit` when a UI-audit runner is wired). Mount the canonical server instead of forking delegation logic.
-
-```ts
-import { createMcpServer } from '@tangle-network/agent-runtime/mcp'
-
-const server = createMcpServer({ delegateSupervisor: { router, backend, deliverable } })
-```
-
-Or mount the `agent-runtime-mcp` stdio bin on a production `AgentProfile.mcp` with `MCP_ENABLE_DELEGATE=1`.
-
-Delegation state is in-memory by default — a server restart drops pending delegations and history. Set `AGENT_RUNTIME_DELEGATION_STATE_FILE=/path/state.json` on the bin (or construct via `DelegationTaskQueue.restore({ store: new FileDelegationStore({ filePath }) })`) to persist records across restarts: `delegation_status`/`delegation_history` keep answering for prior runs, idempotency keys dedupe resubmissions, and in-flight records either resume through the `resumeDelegate` seam (when submitted with a `detachedSessionRef`) or settle as failed with an explicit driver-restart error. A corrupt state file refuses to load (`DelegationStateCorruptError`); `AGENT_RUNTIME_DELEGATION_STATE_RECOVER=1` archives it and starts empty. `AGENT_RUNTIME_DELEGATION_RETAIN_TERMINAL=<n>` caps retained terminal records.
-
-## The experiment harness (bench/)
-
-`bench/` is the internal harness; [`bench/HARNESS.md`](./bench/HARNESS.md) is its map — read that
-first. The canonical path is the optimization suite (`runBenchmark`/`runStrategyEvolution` over real
-domains: the EnterpriseOps gym, commit0, answer-shaped math). The live evidence ledger is
-`.evolve/current.json` — results never live in this README.
-
-The recursive diverse-vs-blind gate runs through the keystone: `gate-cli.mts` →
-`runGate` composes a `Persona` + the generic `fanout` combinator over the budget-conserving
-`Supervisor`, with each child solved via the router and graded by the benchmark's own deployable
-`adapter.judge` (selector ≠ oracle). Each rollout is written to a durable canonical corpus; the deep
-statistics (paired bootstrap with Benjamini-Hochberg correction, selector replay) come from
-`corpus-report.mts` and `corpus-replay.mts` over that corpus, computed once and offline. See
-`bench/HARNESS.md` and `docs/learning-flywheel.md`.
-
-## Defaults
-
-| Knob | Default | Override |
-|---|---|---|
-| Backend model | `gpt-4o-mini` (via `createOpenAICompatibleBackend`) | `model` option or `MODEL_NAME` env |
-| Backend provider | `openai-compat` when `TANGLE_API_KEY`, else `openai` if `OPENAI_API_KEY` | `MODEL_PROVIDER` env |
-| Router base URL | `https://router.tangle.tools/v1` | `TANGLE_ROUTER_BASE_URL` env |
-| Sandbox base URL | `https://sandbox.tangle.tools` | `SANDBOX_API_URL` env |
-| Loop iteration cap | 10 (`runLoop`) | `runLoop({ maxIterations })` |
-| Driver | none, required by `runLoop` | an inline `Driver` (`plan`/`decide`) |
-| Strategy budget (suite) | 3 rollouts/shots per strategy per task | `runBenchmark({ budget })` |
-| Winner selection (coder delegate) | `highest-score` | `winnerSelection` option |
-| KB gate min passage | 12 chars | `createKbGate({ minPassageChars })` |
-| `selfImprove` gate | held-out gate (default) | pass `gate: defaultProductionGate` for red-team hardening |
-| OTEL export | off | set `OTEL_EXPORTER_OTLP_ENDPOINT` |
-| Loop-runner mode failure | recorded as `{ ok: false }` | `runDelegatedLoop` never crashes on a thrown engine |
-
-## Composition with the stack
-
 ```
-agent-runtime   handleChatTurn, runLoop + drivers, runProgram, runDelegatedLoop, createMcpServer,
-                improvementDriver, createKbGate, buildLoopOtelSpans, defineAgent
 
-agent-eval      selfImprove (the optimization entry point), runEvalCampaign,
-                runImprovementLoop (gepaDriver), heldOutGate, runAgentMatrix, analyzeRuns.
-                Consumes runtime traces, scores, gates promotion. agent-runtime depends on it,
-                never the reverse.
+## How it works (the short version)
 
-agent-knowledge proposeKnowledgeWrites, applyKnowledgeWriteBlocks. The analyst loop produces
-                these; the runtime and createKbGate consume them.
+- **One agent, run two ways.** The same agent runs at "do the task" speed and at "get better at the task" speed. "Driver", "worker", and "coordinator" aren't separate types — they're roles one agent plays.
+- **Everything is measured.** Every run is a trace: tokens, dollars, time, and a pass/fail score from a real check. "Better" is a number with a denominator, not a vibe — and "equally good but cheaper" is a result you can prove.
+- **Improvement is gated.** A change ships only after it beats the current agent on fresh tasks no tuning step ever saw, with a statistical test — not a single lucky run.
+- **The grader is honest.** Whatever gives feedback never sees the answer key, and scores are recomputed from the attempts actually run — an agent can't fabricate its own win.
 
-sandbox         AgentProfile, Sandbox.create, streamPrompt, exportTraceBundle. The harness
-                execution surface every loop runs on.
-```
-
-## Subpath exports
+## Examples
 
-Six subpaths — the public surface:
+Runnable, grouped by what they show — copy the one nearest your task:
 
-| Import | Owns |
+| Do this | Example |
 |---|---|
-| `@tangle-network/agent-runtime` | chat turns, delegated loop-runner, OTEL export, errors, model resolution |
-| `.../agent` | `defineAgent` plus surface and outcome adapters |
-| `.../loops` | **the optimization suite** (`Environment`, `defineStrategy`, `runBenchmark`, `runStrategyEvolution`, `authorStrategy`, `promotionGate`) + the recursive atom (`Supervisor`/`Scope`, `createExecutor`), the `runLoop` kernel, the `Driver` type, `loopDispatch` |
-| `.../profiles` | `coderTaskToPrompt` (the coder task formatter), the `uiAuditorProfile` presets + the UI-audit workspace I/O helpers |
-| `.../intelligence` | `withTangleIntelligence`, `createIntelligenceClient` — Observe + the provable-OFF billing boundary |
-| `.../mcp` | `createMcpServer`, `createDefaultCoderDelegate`, `createKbGate`, the `agent-runtime-mcp` bin |
-
-Bins: `agent-runtime-mcp` (delegation MCP server), `agent-runtime-loop` (schedulable delegated loop-runner).
-
-## Teaching an agent to build on this
-
-Two agent-consumable skills live in the [`loops`](https://github.com/drewstone/loops) repo:
-**`skills/loop-builder`** (domain → `Environment` → loop → gate → operator surface, with the
-measured foot-gun list) and **`skills/loop-author`** (authoring a strategy body from losses;
-read the contract with `loops contract`). The runnable on-ramp is [`examples/`](./examples/README.md)
-— a learning progression from the production chat turn through the strategy suite to the recursive
-supervisor. For the broader pipeline (trace sink, analyst loop, scorecard, CI), see the
-`agent-eval-adoption` and `agent-stack-adoption` skills.
-
-## Stability, tests, docs
-
-Every public export is annotated `@stable` or `@experimental`. `@stable` exports do not change shape inside a minor version; `@experimental` ones may, and require a deliberate consumer bump.
-
-```bash
-pnpm test       # kernel, drivers, MCP, delegate hardening, kb-gate, loop-runner, backends
-pnpm typecheck
-pnpm build
-```
-
-Deeper docs: [`docs/architecture.md`](./docs/architecture.md) (the canonical spine), [`docs/canonical-api.md`](./docs/canonical-api.md) (the anti-reinvention decision table), [`docs/learning-flywheel.md`](./docs/learning-flywheel.md) (the self-improvement thesis and the open gate), [`docs/concepts.md`](./docs/concepts.md) (mental model), [`docs/agent-bus-protocol.md`](./docs/agent-bus-protocol.md) (cross-gateway header contract), [`docs/durability-adapters.md`](./docs/durability-adapters.md) (SQL-backed `ConversationJournal`).
+| Run a product chat turn | [`chat-handler`](./examples/chat-handler) |
+| Drive a team of agents to a goal | [`supervise`](./examples/supervise) · [`recursive-supervisor`](./examples/recursive-supervisor) |
+| Benchmark strategies on your own domain | [`coding-benchmark`](./examples/coding-benchmark) |
+| Benchmark **harnesses × models** over a real task suite (the real WebCode dataset) | [`webcode-matrix`](./examples/webcode-matrix) |
+| Render a **multi-profile leaderboard** — ranked board + score matrix + SVG/HTML charts, any domain | `leaderboard(records)` → `renderLeaderboardMarkdown` / `Svg` / `Html` |
+| Trace + bill + effort-gate the WebCode benchmark (the Intelligence SDK) | [`intelligence-webcode`](./examples/intelligence-webcode) |
+| Self-improve an agent, gated on a held-out set | [`improve`](./examples/improve) · [`self-improving-coder`](./examples/self-improving-coder) |
+| Study coordination vs raw compute | [`ablation-suite`](./examples/ablation-suite) |
+
+All 28 live in [`examples/`](./examples).
+
+## Where to go next
+
+- New here? [`docs/concepts.md`](./docs/concepts.md) — the mental model in plain terms.
+- [`docs/canonical-api.md`](./docs/canonical-api.md) — find the primitive: "I want to ___ → use ___".
+- [`docs/architecture.md`](./docs/architecture.md) — the design, end to end.
+- [`bench/HARNESS.md`](./bench/HARNESS.md) — the experiment harness and how to run a benchmark.

package/dist/agent.js

@@ -1,12 +1,12 @@
-import "./chunk-AG335EXG.js";
+import "./chunk-AHZ3YBL6.js";
 import {
   createSandboxForSpec
-} from "./chunk-KRULXIWS.js";
-import "./chunk-VMNEQHJR.js";
-import "./chunk-Z3RRRPRB.js";
+} from "./chunk-44CX5JU6.js";
+import "./chunk-IPEQ3ERC.js";
+import "./chunk-H7IBHAFT.js";
 import {
   mapSandboxEvent
-} from "./chunk-T2HVQVB4.js";
+} from "./chunk-4J6RBI3K.js";
 import "./chunk-5JAUQZQA.js";
 import "./chunk-NBV35BR6.js";
 import "./chunk-WIR4HOOJ.js";

package/dist/analyst-loop.js

@@ -1,8 +1,8 @@
 import {
   iterationsToTraceStore,
   runAnalystLoop
-} from "./chunk-IODKUOBA.js";
-import "./chunk-T2HVQVB4.js";
+} from "./chunk-C2FZ6GR6.js";
+import "./chunk-4J6RBI3K.js";
 import "./chunk-NBV35BR6.js";
 import "./chunk-DGUM43GV.js";
 export {

package/dist/{chunk-KRULXIWS.js → chunk-44CX5JU6.js}

@@ -1,6 +1,6 @@
 import {
   buildLoopSpanNodes
-} from "./chunk-VMNEQHJR.js";
+} from "./chunk-IPEQ3ERC.js";
 import {
   addTokenUsage,
   deleteBoxSafe,
@@ -15,10 +15,10 @@ import {
   throwIfAborted,
   withTimeout,
   zeroTokenUsage
-} from "./chunk-Z3RRRPRB.js";
+} from "./chunk-H7IBHAFT.js";
 import {
   extractLlmCallEvent
-} from "./chunk-T2HVQVB4.js";
+} from "./chunk-4J6RBI3K.js";
 import {
   harnessInvocation,
   runLocalHarness
@@ -6479,4 +6479,4 @@ export {
   createInProcessTransport,
   serveCoordinationMcp
 };
-//# sourceMappingURL=chunk-KRULXIWS.js.map
+//# sourceMappingURL=chunk-44CX5JU6.js.map

package/dist/{chunk-T2HVQVB4.js → chunk-4J6RBI3K.js}

@@ -29,6 +29,19 @@ function extractLlmCallEvent(event, agentRunName) {
   }
   return void 0;
 }
+function sumSandboxUsage(events, agentRunName = "agent") {
+  let input = 0;
+  let output = 0;
+  let costUsd = 0;
+  for (const ev of events) {
+    const call = extractLlmCallEvent(ev, agentRunName);
+    if (!call) continue;
+    input += call.tokensIn ?? 0;
+    output += call.tokensOut ?? 0;
+    costUsd += call.costUsd ?? 0;
+  }
+  return { input, output, costUsd };
+}
 function buildLlmCall(data, agentRunName) {
   const tokensIn = pickFiniteNumber(data, ["tokensIn", "inputTokens", "prompt_tokens"]);
   const tokensOut = pickFiniteNumber(data, ["tokensOut", "outputTokens", "completion_tokens"]);
@@ -73,6 +86,7 @@ function mapSandboxEvent(event, opts = {}) {
 
 export {
   extractLlmCallEvent,
+  sumSandboxUsage,
   mapSandboxEvent
 };
-//# sourceMappingURL=chunk-T2HVQVB4.js.map
+//# sourceMappingURL=chunk-4J6RBI3K.js.map

package/dist/chunk-4J6RBI3K.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/runtime/sandbox-events.ts"],"sourcesContent":["/**\n * Sandbox-event → runtime-event mapping.\n *\n * The sandbox SDK emits a polymorphic `SandboxEvent = { type, data, id? }`\n * whose `type` vocabulary is backend-determined (opencode, etc.) rather than\n * enumerated by the SDK. Two consumers project it:\n *   - the loop kernel's cost ledger (`extractLlmCallEvent`) — sums usage off\n *     every cost-bearing event, regardless of stream shape;\n *   - the `AgentRuntime.act` streaming contract (`mapSandboxEvent`) — projects\n *     incremental events to the `RuntimeStreamEvent` chat-UX vocabulary.\n *\n * Both live here so the empirically-observed `type` vocabulary has one home.\n */\n\nimport type { SandboxEvent } from '@tangle-network/sandbox'\nimport type { RuntimeStreamEvent } from '../types'\n\n/**\n * Extract a `RuntimeStreamEvent`-shaped `llm_call` from a sandbox event when\n * the event carries usage/cost data. Returns `undefined` for non-cost events\n * so the kernel can iterate the full stream without branching.\n *\n * Canonical cost-carrying types observed in the wild:\n *   - `llm_call` — `data: { model, tokensIn, tokensOut, costUsd, ... }`\n *   - `message.completed` / `result` — `data: { usage: { inputTokens,\n *      outputTokens, totalCostUsd? } }`\n *   - `cost.usage` / `usage` — same shape under a dedicated type\n *\n * Numeric coercion is strict: `Number.isFinite` gates every accumulator write\n * so a sentinel `NaN` from a misbehaving backend cannot poison the ledger.\n */\nexport function extractLlmCallEvent(\n  event: SandboxEvent,\n  agentRunName: string,\n): (RuntimeStreamEvent & { type: 'llm_call' }) | undefined {\n  if (!event || typeof event !== 'object') return undefined\n  const type = String(event.type ?? '')\n  const data =\n    event.data && typeof event.data === 'object'\n      ? (event.data as Record<string, unknown>)\n      : ({} as Record<string, unknown>)\n\n  if (type === 'llm_call' || type === 'cost.usage' || type === 'usage') {\n    return buildLlmCall(data, agentRunName)\n  }\n  if (type === 'message.completed' || type === 'result' || type === 'final') {\n    const usage = data.usage as Record<string, unknown> | undefined\n    if (!usage || typeof usage !== 'object') return undefined\n    return buildLlmCall({ ...usage, model: data.model ?? usage.model }, agentRunName)\n  }\n  // sandbox 0.4.0 terminal event: `data = { tokenUsage: { inputTokens, outputTokens,\n  // reasoningTokens, cacheReadInputTokens }, totalCostUsd }`. Usage lives under\n  // `tokenUsage` (not `usage`) and the cost is top-level — neither matched the\n  // branches above, so an in-process loopDispatch run reported {0,0} and the\n  // backend-integrity guard misread a real run as a stub. Reasoning tokens are\n  // billed output (reasoning models), so they fold into the output count.\n  if (type === 'done') {\n    const usage = data.tokenUsage as Record<string, unknown> | undefined\n    if (!usage || typeof usage !== 'object') return undefined\n    const out = pickFiniteNumber(usage, ['outputTokens', 'completion_tokens', 'tokensOut'])\n    const reasoning = pickFiniteNumber(usage, ['reasoningTokens'])\n    const mergedOut =\n      out !== undefined || reasoning !== undefined ? (out ?? 0) + (reasoning ?? 0) : undefined\n    return buildLlmCall(\n      {\n        inputTokens: usage.inputTokens,\n        outputTokens: mergedOut,\n        totalCostUsd: data.totalCostUsd,\n        model: data.model ?? usage.model,\n      },\n      agentRunName,\n    )\n  }\n  return undefined\n}\n\n/**\n * Sum the token usage + USD cost of a sandbox turn's events — the one honest way to meter an\n * `openSandboxRun` cell. Folds `extractLlmCallEvent` over the stream (which reads usage off EVERY backend\n * event shape), so a `runProfileMatrix` dispatch can report it to `ctx.cost`:\n *\n *     const turn = await run.start(prompt)\n *     const u = sumSandboxUsage(turn.events)\n *     if (u.input || u.output) ctx.cost.observeTokens({ input: u.input, output: u.output })\n *     if (u.costUsd) ctx.cost.observe(u.costUsd, 'sandbox-cell')\n *\n * Without this a cell reads `{tokens:0, cost:0}` and the backend-integrity guard correctly aborts the\n * matrix as a stub. `agentRunName` is the fallback model label for cost-only events (default `'agent'`).\n */\nexport function sumSandboxUsage(\n  events: readonly SandboxEvent[],\n  agentRunName = 'agent',\n): { input: number; output: number; costUsd: number } {\n  let input = 0\n  let output = 0\n  let costUsd = 0\n  for (const ev of events) {\n    const call = extractLlmCallEvent(ev, agentRunName)\n    if (!call) continue\n    input += call.tokensIn ?? 0\n    output += call.tokensOut ?? 0\n    costUsd += call.costUsd ?? 0\n  }\n  return { input, output, costUsd }\n}\n\nfunction buildLlmCall(\n  data: Record<string, unknown>,\n  agentRunName: string,\n): (RuntimeStreamEvent & { type: 'llm_call' }) | undefined {\n  const tokensIn = pickFiniteNumber(data, ['tokensIn', 'inputTokens', 'prompt_tokens'])\n  const tokensOut = pickFiniteNumber(data, ['tokensOut', 'outputTokens', 'completion_tokens'])\n  const costUsd = pickFiniteNumber(data, ['costUsd', 'totalCostUsd', 'cost_usd', 'cost'])\n  if (tokensIn === undefined && tokensOut === undefined && costUsd === undefined) {\n    return undefined\n  }\n  const model = typeof data.model === 'string' && data.model.length > 0 ? data.model : agentRunName\n  const event: RuntimeStreamEvent & { type: 'llm_call' } = {\n    type: 'llm_call',\n    model,\n  }\n  if (tokensIn !== undefined) event.tokensIn = tokensIn\n  if (tokensOut !== undefined) event.tokensOut = tokensOut\n  if (costUsd !== undefined) event.costUsd = costUsd\n  return event\n}\n\nfunction pickFiniteNumber(data: Record<string, unknown>, keys: string[]): number | undefined {\n  for (const key of keys) {\n    const value = data[key]\n    if (typeof value === 'number' && Number.isFinite(value)) return value\n  }\n  return undefined\n}\n\n/**\n * Project one `SandboxEvent` onto the `RuntimeStreamEvent` chat-UX vocabulary,\n * for runtimes that bridge a sandbox `streamPrompt` into the\n * `AgentRuntime.act` streaming contract. Returns `undefined` for events that\n * have no faithful projection — the raw stream is preserved separately for the\n * `OutputAdapter`, so an unmapped event never loses data.\n *\n * Mapped (the task-optional incremental variants — no synthesized task\n * lifecycle, no guessed tool-part shapes):\n *   - `message.part.updated` text part → `text_delta`\n *   - `message.part.updated` reasoning/thinking part → `reasoning_delta`\n *   - cost-bearing events → `llm_call` (shared with the ledger extractor)\n *\n * The opencode backend emits incremental text as\n * `{ type: 'message.part.updated', data: { part: { type, text }, delta } }`;\n * `delta` is the increment, `part.text` the running accumulation.\n */\nexport function mapSandboxEvent(\n  event: SandboxEvent,\n  opts: { agentRunName?: string } = {},\n): RuntimeStreamEvent | undefined {\n  if (!event || typeof event !== 'object') return undefined\n  const type = String(event.type ?? '')\n  const data =\n    event.data && typeof event.data === 'object'\n      ? (event.data as Record<string, unknown>)\n      : ({} as Record<string, unknown>)\n\n  if (type === 'message.part.updated') {\n    const part =\n      data.part && typeof data.part === 'object' ? (data.part as Record<string, unknown>) : {}\n    const partType = String(part.type ?? '')\n    const delta = typeof data.delta === 'string' ? data.delta : undefined\n    const text = delta ?? (typeof part.text === 'string' ? part.text : undefined)\n    if (text === undefined) return undefined\n    if (partType === 'text') return { type: 'text_delta', text }\n    if (partType === 'reasoning' || partType === 'thinking')\n      return { type: 'reasoning_delta', text }\n    return undefined\n  }\n\n  return extractLlmCallEvent(event, opts.agentRunName ?? 'agent')\n}\n"],"mappings":";AA+BO,SAAS,oBACd,OACA,cACyD;AACzD,MAAI,CAAC,SAAS,OAAO,UAAU,SAAU,QAAO;AAChD,QAAM,OAAO,OAAO,MAAM,QAAQ,EAAE;AACpC,QAAM,OACJ,MAAM,QAAQ,OAAO,MAAM,SAAS,WAC/B,MAAM,OACN,CAAC;AAER,MAAI,SAAS,cAAc,SAAS,gBAAgB,SAAS,SAAS;AACpE,WAAO,aAAa,MAAM,YAAY;AAAA,EACxC;AACA,MAAI,SAAS,uBAAuB,SAAS,YAAY,SAAS,SAAS;AACzE,UAAM,QAAQ,KAAK;AACnB,QAAI,CAAC,SAAS,OAAO,UAAU,SAAU,QAAO;AAChD,WAAO,aAAa,EAAE,GAAG,OAAO,OAAO,KAAK,SAAS,MAAM,MAAM,GAAG,YAAY;AAAA,EAClF;AAOA,MAAI,SAAS,QAAQ;AACnB,UAAM,QAAQ,KAAK;AACnB,QAAI,CAAC,SAAS,OAAO,UAAU,SAAU,QAAO;AAChD,UAAM,MAAM,iBAAiB,OAAO,CAAC,gBAAgB,qBAAqB,WAAW,CAAC;AACtF,UAAM,YAAY,iBAAiB,OAAO,CAAC,iBAAiB,CAAC;AAC7D,UAAM,YACJ,QAAQ,UAAa,cAAc,UAAa,OAAO,MAAM,aAAa,KAAK;AACjF,WAAO;AAAA,MACL;AAAA,QACE,aAAa,MAAM;AAAA,QACnB,cAAc;AAAA,QACd,cAAc,KAAK;AAAA,QACnB,OAAO,KAAK,SAAS,MAAM;AAAA,MAC7B;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,SAAO;AACT;AAeO,SAAS,gBACd,QACA,eAAe,SACqC;AACpD,MAAI,QAAQ;AACZ,MAAI,SAAS;AACb,MAAI,UAAU;AACd,aAAW,MAAM,QAAQ;AACvB,UAAM,OAAO,oBAAoB,IAAI,YAAY;AACjD,QAAI,CAAC,KAAM;AACX,aAAS,KAAK,YAAY;AAC1B,cAAU,KAAK,aAAa;AAC5B,eAAW,KAAK,WAAW;AAAA,EAC7B;AACA,SAAO,EAAE,OAAO,QAAQ,QAAQ;AAClC;AAEA,SAAS,aACP,MACA,cACyD;AACzD,QAAM,WAAW,iBAAiB,MAAM,CAAC,YAAY,eAAe,eAAe,CAAC;AACpF,QAAM,YAAY,iBAAiB,MAAM,CAAC,aAAa,gBAAgB,mBAAmB,CAAC;AAC3F,QAAM,UAAU,iBAAiB,MAAM,CAAC,WAAW,gBAAgB,YAAY,MAAM,CAAC;AACtF,MAAI,aAAa,UAAa,cAAc,UAAa,YAAY,QAAW;AAC9E,WAAO;AAAA,EACT;AACA,QAAM,QAAQ,OAAO,KAAK,UAAU,YAAY,KAAK,MAAM,SAAS,IAAI,KAAK,QAAQ;AACrF,QAAM,QAAmD;AAAA,IACvD,MAAM;AAAA,IACN;AAAA,EACF;AACA,MAAI,aAAa,OAAW,OAAM,WAAW;AAC7C,MAAI,cAAc,OAAW,OAAM,YAAY;AAC/C,MAAI,YAAY,OAAW,OAAM,UAAU;AAC3C,SAAO;AACT;AAEA,SAAS,iBAAiB,MAA+B,MAAoC;AAC3F,aAAW,OAAO,MAAM;AACtB,UAAM,QAAQ,KAAK,GAAG;AACtB,QAAI,OAAO,UAAU,YAAY,OAAO,SAAS,KAAK,EAAG,QAAO;AAAA,EAClE;AACA,SAAO;AACT;AAmBO,SAAS,gBACd,OACA,OAAkC,CAAC,GACH;AAChC,MAAI,CAAC,SAAS,OAAO,UAAU,SAAU,QAAO;AAChD,QAAM,OAAO,OAAO,MAAM,QAAQ,EAAE;AACpC,QAAM,OACJ,MAAM,QAAQ,OAAO,MAAM,SAAS,WAC/B,MAAM,OACN,CAAC;AAER,MAAI,SAAS,wBAAwB;AACnC,UAAM,OACJ,KAAK,QAAQ,OAAO,KAAK,SAAS,WAAY,KAAK,OAAmC,CAAC;AACzF,UAAM,WAAW,OAAO,KAAK,QAAQ,EAAE;AACvC,UAAM,QAAQ,OAAO,KAAK,UAAU,WAAW,KAAK,QAAQ;AAC5D,UAAM,OAAO,UAAU,OAAO,KAAK,SAAS,WAAW,KAAK,OAAO;AACnE,QAAI,SAAS,OAAW,QAAO;AAC/B,QAAI,aAAa,OAAQ,QAAO,EAAE,MAAM,cAAc,KAAK;AAC3D,QAAI,aAAa,eAAe,aAAa;AAC3C,aAAO,EAAE,MAAM,mBAAmB,KAAK;AACzC,WAAO;AAAA,EACT;AAEA,SAAO,oBAAoB,OAAO,KAAK,gBAAgB,OAAO;AAChE;","names":[]}