@quadiusai/dsh-greenharness 0.0.0-stage → 0.1.0

package/GETTING-STARTED.md

@@ -0,0 +1,76 @@
+# GreenHarness: Getting Started
+
+GreenHarness makes your DSH agent cheaper to run. It shortens huge command output, stops repeated work, and can hand easy steps to a cheap model and hard steps to a strong one. You don't have to configure anything for it to start saving.
+
+## 1. Install (one time)
+
+Open DSH's web page, or a terminal, and run:
+
+```
+dsh plugin --profile web add @quadiusai/dsh-greenharness
+```
+
+Then start `dsh web` as usual. That's it. GreenHarness is now on.
+
+## 2. Add your API keys and pick your models (no files to edit)
+
+Use DSH's own screens:
+
+1. In DSH web, open **Settings → Models**.
+2. Add your provider (for example your NVIDIA/Brev endpoint) and paste your API key. DSH stores it for you.
+3. Use DSH's model chooser to pick the **cheap (small)** model for normal chats.
+4. Tell GreenHarness which model is the **strong (large)** one by typing in the chat box:
+
+```
+/greenharness large <provider> <model>
+```
+
+Example: `/greenharness large nemo-large lightning-30b`
+
+The cheap model is whatever DSH is already using. GreenHarness remembers it automatically. If you want to set it yourself: `/greenharness small <provider> <model>`.
+
+If you skip step 4, GreenHarness still saves tokens. It just won't switch models.
+
+## 3. See it working
+
+Type this in the chat box any time:
+
+```
+/greenharness
+```
+
+You get a live summary like:
+
+```
+GreenHarness is ON
+
+Tokens saved so far: about 1,240
+  (long outputs shortened: 3, repeated reads replaced: 2, loops stopped: 0)
+Work done by the cheap model: [██████████████░░░░░░] 70%
+Model calls: 12. Switches between models: 2.
+```
+
+- **Tokens saved** is an estimate (about 4 characters per token) of text GreenHarness kept out of the model.
+- **The bar** shows how much of the work the cheap model did. A fuller bar means a cheaper run.
+- Type `/greenharness` again to refresh the numbers.
+
+## 4. Handy commands
+
+| Type this | What it does |
+|---|---|
+| `/greenharness` | Show the summary above |
+| `/greenharness off` / `on` | Pause or resume everything |
+| `/greenharness large <provider> <model>` | Pick the strong model (turns switching on) |
+| `/greenharness routing off` | Stop switching models |
+| `/greenharness distill off` | Stop shortening long output |
+| `/greenharness help` | List every command |
+
+Changes last until DSH restarts.
+
+## 5. If something looks wrong
+- **No `/greenharness` command:** run the install line again, then restart `dsh web`.
+- **"Not set" for the strong model:** run `/greenharness large ...` (step 2).
+- **Needs Node 22.19 or newer.**
+
+## Honest limits
+The summary is text in the chat. DSH doesn't offer plugins a way to add their own panel, so there's no separate graphical dashboard. Savings numbers are estimates. The `/greenharness` command in the web chat has been tested only through unit tests and DSH start-up, not by clicking through the live page.

package/README.md

@@ -1,3 +1,148 @@
-# Temporary Holding Version
+# GreenHarness
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A DeepSeek Harness (dsh) plugin that makes multi-step agent runs cheaper. It routes each model call between a small and a large model using the outcome of the project's own tests. It shortens and deduplicates tool output before the model reads it, and it stops an agent from repeating a failing command.
+
+Built for AEGIS 2026, problem 2 (agentic cost and efficiency), team Skyline.
+
+New to terminals or agents? Start with [`GETTING-STARTED.md`](GETTING-STARTED.md).
+
+## Quick install
+
+```bash
+dsh plugin --profile web add @quadiusai/dsh-greenharness
+```
+
+That is the whole setup. The package declares itself as a DSH bundle, so DSH turns it on in that profile by itself, and every setting has a default. Restart `dsh web`, then type `/greenharness` in the chat:
+
+| Command | What it does |
+|---|---|
+| `/greenharness` | Status: what is on, and what it has done in this process. |
+| `/greenharness on` / `off` | Resume or pause every mechanism. |
+| `/greenharness large <provider> <model> [effort]` | Turn on model routing. The small model is whatever DSH already uses; this one is the large model that failing tests escalate to. |
+| `/greenharness small <provider> <model> [effort]` | Set the small model explicitly. |
+| `/greenharness routing on` / `off` | Switch routing without losing the models. |
+| `/greenharness distill` / `dedupe` / `loop` `on` / `off` | Switch one mechanism. |
+
+With no setup, GreenHarness shortens long command output, replaces repeated file reads with a pointer, and blocks a repeated failing command. These need only one model. Routing between two models starts when you set a large model. Command settings last until DSH restarts; to make them permanent, put the same values in the plugin's config (`tiers`, `distill`, and so on, see Configuration).
+
+**Status of this install path.** Checked: the package installs into a fresh web profile with the command above (from a packed tarball), DSH lists it as an active bundle, `dsh web` boots with it and no activation warning, and the command logic has unit tests. **Not checked:** typing `/greenharness` in the web chat itself, and installing from the npm registry (the package is not published yet; publish it with `npm publish --access public`).
+
+## Status
+
+- **Plugin:** four mechanisms wired to DSH 0.2.1-rc.2 headless. 59 unit tests pass on Node 24.22.
+- **Verified end to end against a scripted mock server** (plumbing, not a model). Routing, distillation, read dedupe and loop breaking all reach the model. The thinking-switch request body is correct on both tiers.
+- **Not measured: any real NeMo model.** This session had no GPU and no Brev access, so no quality or cost numbers exist yet. The live runbook is [`bench/BREV.md`](bench/BREV.md).
+
+## How it works
+
+| Mechanism | What it does | DSH hook | Ledger record |
+|---|---|---|---|
+| Cascade | A verifier command (test runner, type checker, linter) that fails moves the next model call to the large tier. Passes twice in a row, after a minimum number of calls, move it back. No judge model is called. | `agent/request` (route), `tools/post-execute` (outcome) | `route`, `request` |
+| Distillation | Bash output keeps failure, summary and safety lines plus nearby context. It drops progress frames and library stack frames, and collapses repeated lines. The full output goes to the spill store, and the model gets a pointer to it. | `tools/post-execute` | `tool` with action `distill` |
+| Read dedupe | A read whose bytes match an earlier read still in context returns a pointer instead of the file again. An edit or write forgets the path, and compaction forgets everything. | `tools/post-execute` | `tool` with action `dedupe` |
+| Loop breaker | A bash command that failed `threshold` times in a row with the same failure, and no file change since, is denied with a reason. The tier goes to large. | `tools/pre-execute` | `loop-break` |
+
+### Hook facts that shaped the design
+
+These come from DSH 0.2.1-rc.2's typings and source.
+
+- **Routing** uses the `agent/request` waterfall. A listener that calls `await next()` receives the config DSH would use, and it returns a replacement to switch the model. DSH seeds each call from the last logged request header, so the route is applied on every call.
+- **Notice removal.** DSH adds a "model changed" notice to the conversation whenever its default selection differs from the last request. Per-step routing triggers that on every large step, and the notice names the wrong model. GreenHarness removes it. Its listener is attached to each agent at `agent/created`, with `prepend`, so it runs outside DSH's own notice listener.
+- **No `agent.model.select()`.** An earlier build called it. The call threw, the error was swallowed, and the unit tests passed only because they used a fake. Plugin failures are now written to the ledger as `error` records.
+
+## Configuration
+
+Settings are validated when the plugin loads. A missing tier, a bad integer or an invalid regex fails loudly.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `tiers.small`, `tiers.large` | required | `provider`, `model`, and `reasoningEffort` (empty means the provider default). |
+| `cascade.escalateAfterFailures` | 1 | Consecutive verifier failures before moving to large. |
+| `cascade.deescalateAfterPasses` | 2 | Consecutive verifier passes before moving back to small. |
+| `cascade.minStepsPerTier` | 2 | Minimum model calls in a tier before it can change. |
+| `verifier.patterns` | test runners, type checkers, linters | Regexes that mark a bash command as a verifier. |
+| `distill.enabled`, `maxChars`, `contextLines` | true, 6000, 2 | Distillation switch, output budget (at least 500), and context lines around failures. |
+| `readDedupe.enabled` | true | Read dedupe switch. |
+| `loopBreaker.enabled`, `threshold` | true, 3 | Loop breaker switch and repeat count (at least 2). |
+| `ledger.path`, `ledger.pricing` | `.greenharness/ledger.jsonl`, none | Where the JSONL ledger goes, and optional per-provider prices in dollars per million tokens. |
+
+The routes themselves (provider URL, model ids, thinking switch) live in the patch overlay. `bench/presets/` shows the format.
+
+## Develop and benchmark
+
+Everything below is for working on the plugin or running the benchmark; ordinary use needs only the quick install above.
+
+Requirements: Node.js 22.19 or newer (the highest minimum any DSH package declares; the mock benchmark ran on Node 24.22) and DSH `0.2.1-rc.2`. Install DSH in its own folder with `npm install @deepseek-ai/dsh@0.2.1-rc.2` (about two minutes here), then point `DSH_BIN` at its `node_modules/.bin/dsh`.
+
+1. `npm install` in this folder. It also installs the plugin's two peer dependencies, `@deepseek-ai/cordis` and `@deepseek-ai/schemastery`, at the versions DSH uses. Run it first: without it the plugin does not load.
+2. Unit tests: `npm test` (59 tests).
+3. Mock suite (every scenario under every arm, about 35 seconds): `DSH_BIN=<dsh> node bench/run-mock-suite.mjs`
+4. One case: `DSH_BIN=<dsh> node bench/run-case.mjs --task subtract-bug --arm greenharness --scenario subtract-fix --out runs/one`
+5. Live NeMo models on Brev: [`bench/BREV.md`](bench/BREV.md).
+
+Each case copies the task to a fresh workspace and starts a fresh DSH home. The verdict comes from re-running `node --test` in the workspace, not from the agent's claim. Each case writes `summary.json`, `ledger.jsonl`, `patch.yml`, `stdout.txt`, `stderr.txt` and the workspace. If something went wrong (DSH did not start or exited with an error, or GreenHarness recorded no model calls), `summary.json` lists it under `problems`, the reason goes to stderr, and the command exits with code 1.
+
+## Metrics
+
+- **Model calls per tier**: `ledger.tiers`, `ledger.models`.
+- **Prompt tokens**: `inputTokens` (uncached) plus `cacheReadTokens` plus `cacheWriteTokens`. Output tokens are reported separately.
+- **Tool output**: `rawChars` against `outChars` on each `tool` record. Distillation and dedupe show up here.
+- **Verdict**: the independent suite re-run (`suite.pass`, `suite.fail`).
+- **Cost**: tokens times the configured price, or GPU-hours times an hourly rate. Not automated for self-hosted GPUs.
+
+## Results so far (mock, plumbing only)
+
+These come from `bench/run-mock-suite.mjs`. The mock replays a fixed script, so these numbers show what the plugin changes in the prompt. They say nothing about a real model's pass rate. Token counts are characters divided by four, over the message history only. Token estimates shift by a few percent between installs and folder layouts, so compare the rows, tier splits and pass/fail results rather than the last digit.
+
+| scenario | arm | agent calls | served by | prompt tokens (est.) | cached (est.) | suite | distilled | deduped | route changes | loop denied | errors |
+|---|---|---:|---|---:|---:|---|---:|---:|---:|---|---:|
+| loop-stuck | greenharness | 6 | 1 small / 5 large | 9269 | 6244 | 301 pass / 1 fail | 3 | 0 | 1 | yes | 0 |
+| loop-stuck | always-large | 6 | 0 small / 6 large | 46520 | 30947 | 301 pass / 1 fail | 0 | 0 | 0 | no | 0 |
+| loop-stuck | always-small | 6 | 6 small / 0 large | 46520 | 30952 | 301 pass / 1 fail | 0 | 0 | 0 | no | 0 |
+| subtract-fix | greenharness | 7 | 2 small / 5 large | 10064 | 5523 | 302 pass / 0 fail | 3 | 1 | 2 | no | 0 |
+| subtract-fix | always-large | 7 | 0 small / 7 large | 32024 | 22068 | 302 pass / 0 fail | 0 | 0 | 0 | no | 0 |
+| subtract-fix | always-small | 7 | 7 small / 0 large | 32023 | 22073 | 302 pass / 0 fail | 0 | 0 | 0 | no | 0 |
+
+In the subtract scenario, GreenHarness sent about 3.2 times fewer estimated prompt tokens than either single-model arm. In the loop scenario, it denied the repeats after three identical failures. The `always-*` arms have no verifier and no loop breaker, so they always run the same script to the end.
+
+## Limitations
+
+- **No real model yet.** Routing and the thinking switch are verified against the mock only. The thinking switch's wire format is confirmed; whether the NeMo server honours it is not (see the toggle test in `bench/BREV.md`).
+- **Escalation on the first failure.** A failing task spends most of its calls on the large tier. Savings come from passing runs staying small and from smaller tool output on every tier. `escalateAfterFailures` and `minStepsPerTier` are the first knobs to tune.
+- **Cold caches after a switch.** The first call on a tier after a switch pays a full prefill, because that model's prefix cache is cold. The mock models caches per model, but real servers will differ.
+- **Side request.** Every run has one tools-free request, sent right after the first agent call. The mock logs it as a side call. The ledger does not record it, so its tokens are missing from the ledger totals. It ran on the default route, before the first escalation in every run. Whether GreenHarness routes it has not been checked.
+- **Distillation can hide a line the model needed.** The full output is one retrieval away, but the model has to ask for it.
+- **One task so far.** `subtract-bug` is one failing test plus 300 passing filler tests. The 20–30 task set from the deck is not built.
+- **Developer preview.** DSH 0.2.1-rc.2 is pre-release. The hook names above come from its typings in this version and may change.
+- **Bolt E4B is unverified.** Its model id, serve command and thinking switch are not confirmed (see `bench/BREV.md`).
+
+## Compared with NeMo Switchyard
+
+As its README describes it, Switchyard is a routing proxy on port 4000. It decides per request and per step with a classifier, and it escalates with an LLM judge.
+
+GreenHarness differs in four ways:
+
+- **Verifier, not judge.** It sits inside the agent and reads the result of the test or compiler run, so escalation costs no extra model call.
+- **Compress once.** Distillation and dedupe act on tool results when they are written. A proxy sees the whole request on every call, so it would have to rewrite history it already sent, which breaks the prompt cache prefix.
+- **Session state.** The loop breaker keys on command fingerprints and resets on file changes. A stateless proxy does not hold that state.
+- **Host-bound.** It runs only inside DSH. Switchyard works with any client that speaks the API.
+
+## Files
+
+- `GETTING-STARTED.md`: step by step, for someone new to terminals and agents.
+- `src/index.mjs`: the Cordis plugin. Config schema, hook wiring, error reporting.
+- `src/control.mjs`: the `/greenharness` command (parsing and replies).
+- `src/runtime.mjs`: cascade, distillation, dedupe, loop breaker and ledger records. No DSH imports.
+- `src/core/`: pure logic. `cascade`, `verify`, `distill`, `patterns`, `read-index`, `loop-guard`, `route`, `ledger`, `pricing`.
+- `src/config.mjs`: settings normalization and validation.
+- `test/`: 59 unit tests.
+- `bench/run-case.mjs`: one case (task, arm, preset) through DSH headless, with the independent re-run and `summary.json`.
+- `bench/run-mock-suite.mjs`: every scenario under every arm, as one table.
+- `bench/mock-openai-server.mjs`: scripted OpenAI-compatible server. It logs what each request contained.
+- `bench/presets/`: `mock`, `mock-chat-template`, `nemo-brev`.
+- `bench/scenarios/`: scripts for the mock. `bench/tasks/subtract-bug/`: the fixture.
+- `bench/BREV.md`: live runbook, with what is verified and what is not.
+
+## License
+
+MIT. Add a LICENSE file before publishing.

package/bench/BREV.md

@@ -0,0 +1,127 @@
+# Running GreenHarness against NeMo models on Brev
+
+This runbook is the live half of the benchmark. The mock runs in `bench/run-mock-suite.mjs` check plumbing only.
+
+## What is verified and what is not
+
+| Item | Status |
+|---|---|
+| Plugin routing, distillation, read dedupe, loop breaker | Verified against the scripted mock (`npm run bench:mock`, 47 unit tests). |
+| Thinking switch sent to the server | Verified in the request body the mock received: `chat_template_kwargs.enable_thinking` is `false` on the small tier and `true` on the large tier. Whether the NeMo server honours it is **not** verified (step 4). |
+| Lightning 30B-A3B serve command | Taken from the vLLM Day-0 post (2026-08-10). The post says it assumes one H100. Not run here. |
+| Bolt E4B model id, serve command, thinking switch | **Not verified.** One source we read (DataLearner, dated 2026-10-19) says the model was not yet released, names no repo or serve command, and lists no thinking mode. Another note from earlier research says its thinking budget is configurable. Check the model card before you trust the small tier's numbers. |
+| Prices | Not set. The ledger records tokens; cost needs prices (see step 6). |
+
+No NeMo model was run in this session. Nothing below has been measured on NeMo.
+
+## 1. Instance
+
+Start a Brev instance with one H100 80 GB, or an equivalent card, for the large tier. Connect:
+
+```bash
+brev shell <instance>
+```
+
+Check the current CLI syntax with `brev --help` and `brev port-forward --help`.
+
+## 2. Large tier (port 8000): Nemotron 3.5 Lightning 30B-A3B, BF16
+
+Serve command, copied from the vLLM Day-0 post:
+
+```bash
+vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
+  --max-num-seqs 256 \
+  --max-num-batched-tokens 32768 \
+  --enable-prefix-caching \
+  --async-scheduling \
+  --mamba-backend flashinfer \
+  --moe-backend humming \
+  --linear-backend humming \
+  --mamba-ssu-algorithm horizontal \
+  --mamba-cache-mode align \
+  --mamba-ssm-cache-dtype float16 \
+  --enable-mamba-cache-stochastic-rounding \
+  --mamba-cache-philox-rounds 5 \
+  --reasoning-parser nemotron_v3 \
+  --tool-call-parser qwen3_coder \
+  --enable-auto-tool-choice \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+Use a vLLM release from 2026-08-10 or later. The NVFP4 build (`nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4`) is smaller, but check that your card supports its kernels before you use it.
+
+## 3. Small tier (port 8001): Nemotron 3.5 Bolt E4B
+
+**Not verified.** Get the model id and serve command from the model card once it is published, then run it on port 8001. The preset assumes:
+
+- model id `nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB`
+- an OpenAI-compatible endpoint at `http://localhost:8001/v1`
+
+If the card gives a vLLM command, use the same pattern as step 2 with `--port 8001`. If it ships only as GGUF, serve it with llama.cpp's OpenAI-compatible server (`llama-server`) on port 8001 and check the model name it reports.
+
+Co-hosting both on one 80 GB card is possible only if you cap each server's memory (`--gpu-memory-utilization`, for example 0.80 for the large tier and what is left for the small one). Test it; otherwise use a second instance.
+
+## 4. Toggle test (do this before any benchmark)
+
+The preset turns the thinking switch on or off per tier with `chat_template_kwargs.enable_thinking`. Confirm the server honours it on each endpoint: the same prompt should produce a reasoning trace only when the flag is `true`.
+
+```bash
+for flag in true false; do
+  curl -s http://localhost:8000/v1/chat/completions -H 'content-type: application/json' -d "{
+    \"model\": \"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16\",
+    \"messages\": [{\"role\": \"user\", \"content\": \"What is 17 times 23? Answer with the number only.\"}],
+    \"chat_template_kwargs\": {\"enable_thinking\": $flag}, \"max_tokens\": 800
+  }" | head -c 1500; echo; echo ---
+done
+```
+
+Compare `completion_tokens` and any reasoning field in the two replies. If the flag makes no difference, the server ignores it: change the `compat` block in `bench/presets/nemo-brev.json` before you benchmark. Repeat the test on port 8001 for the small model.
+
+## 5. Port-forward to this machine
+
+```bash
+brev port-forward <instance> --port 8000:8000
+brev port-forward <instance> --port 8001:8001
+```
+
+Check that `curl http://127.0.0.1:8000/v1/models` lists the large model.
+
+## 6. Run the three arms
+
+Install the plugin's dependencies and DSH first (see the README), then:
+
+```bash
+export DSH_BIN=dsh            # or the path to your dsh binary
+export NEMO_API_KEY=unused    # local vLLM does not check keys
+node bench/run-case.mjs --task subtract-bug --arm greenharness --preset nemo-brev --out runs/brev/greenharness
+node bench/run-case.mjs --task subtract-bug --arm always-large --preset nemo-brev --out runs/brev/always-large
+node bench/run-case.mjs --task subtract-bug --arm always-small --preset nemo-brev --out runs/brev/always-small
+```
+
+Each run writes `summary.json`. Read these fields:
+
+- `suite.pass` / `suite.fail`: an independent `node --test` re-run in the workspace. This is the verdict.
+- `ledger.tiers` and `ledger.models`: calls per tier.
+- `ledger.usage`: `inputTokens` (uncached), `cacheReadTokens`, `cacheWriteTokens`, `outputTokens`. Prompt tokens are the sum of the first three.
+- `ledger.tool_actions`: how often distillation and dedupe fired.
+- `ledger.route_changes`: each tier change and its reason.
+- `problems`: anything that went wrong (DSH failed, or the plugin recorded no model calls). Read it first; an empty list means the run is usable.
+- `wallSeconds`: whole-run wall time.
+
+Run each arm at least three times. Model output varies between runs.
+
+## 7. Cost
+
+The ledger records tokens, not money. You can get money in one of two ways:
+
+- Per-token prices: set `ledger.pricing` in the plugin config (`inputPerMTok`, `cachedInputPerMTok`, `outputPerMTok`, keyed by provider name). `costUsd` in each request record then follows.
+- Self-hosted GPU time: multiply each arm's wall time by your instance's hourly rate, then compare arms. This does not split time by tier. Do that by reading the vLLM request log, which is not automated here.
+
+Report which of the two you used. Neither price source has been checked for these models.
+
+## Troubleshooting
+
+- `TRANSPORT: Connection error` in `stderr.txt`: the port-forward is down, or the server is not up.
+- A `greenharness:` error in `ledger.jsonl` (`"type":"error"`): the plugin caught a failure in one hook. The message says which.
+- Suite fails with `not run` in the table: the summary was not written. Read `stderr.txt` in the run folder.

package/bench/lib/patch.mjs

@@ -0,0 +1,60 @@
+/**
+ * Renders the DSH patch overlay for one benchmark variant. The overlay is JSON, which is valid
+ * YAML, so DSH's loader reads it with `--patch`. It mounts the OpenAI-compatible model routes
+ * (`@deepseek-ai/dsh-llm-pi-ai`), points the default agent at the small tier, and mounts
+ * GreenHarness with the tier map. A baseline variant omits the GreenHarness row, so the same
+ * routes serve both arms of the comparison.
+ */
+
+/**
+ * @typedef {{ id: string, name?: string, contextWindow?: number, reasoningEfforts?: Record<string, string | null> }} RouteModel
+ * @typedef {{ baseURL: string, apiKeyEnv?: string, api?: string, compat?: Record<string, unknown>, defaultMaxTokens?: number, displayName?: string, models: RouteModel[] }} RouteSpec
+ * @typedef {{ provider: string, model: string, reasoningEffort?: string }} TierSpec
+ *
+ * @param {object} options
+ * @param {Record<string, RouteSpec>} options.routes - model routes keyed by route name.
+ * @param {{ small: TierSpec, large: TierSpec }} options.tiers - the cascade's two tiers.
+ * @param {boolean} options.greenharness - mount the plugin (false for the baseline arm).
+ * @param {string} [options.pluginEntry] - absolute path of the plugin's src/index.mjs (required when greenharness is true).
+ * @param {string} [options.ledgerPath] - absolute path of the JSONL ledger for this run.
+ * @param {Record<string, unknown>} [options.pluginOptions] - extra GreenHarness config (cascade, distill, ...).
+ * @returns {string} the overlay text.
+ */
+export function renderPatch({ routes, tiers, greenharness, pluginEntry, ledgerPath, pluginOptions = {} }) {
+  const rows = [
+    // `llm-pi-ai` is already mounted by the base bundle; this row configures that instance.
+    // Self-hosted OpenAI-compatible servers ignore the key unless one is configured, but pi-ai
+    // requires a key reference on hand-declared routes, so each route names one.
+    { id: 'llm-pi-ai', config: { providers: Object.fromEntries(Object.entries(routes).map(([name, route]) => [name, { apiKeyEnv: 'NEMO_API_KEY', ...route }])) } },
+    {
+      id: 'agent-default-model',
+      config: { provider: tiers.small.provider, model: tiers.small.model },
+    },
+  ]
+  if (greenharness) {
+    if (!pluginEntry) throw new Error('renderPatch: pluginEntry (absolute path to src/index.mjs) is required when greenharness is true')
+    rows.push({
+      insert: [{
+        id: 'greenharness',
+        name: pluginEntry,
+        config: {
+          tiers: { small: tiers.small, large: tiers.large },
+          ledger: { path: ledgerPath ?? '.greenharness/ledger.jsonl', ...(pluginOptions.ledger ?? {}) },
+          ...withoutKey(pluginOptions, 'ledger'),
+        },
+      }],
+    })
+  }
+  return `${JSON.stringify(rows, null, 2)}\n`
+}
+
+/**
+ * @template T
+ * @param {Record<string, T>} object
+ * @param {string} key
+ */
+function withoutKey(object, key) {
+  const copy = { ...object }
+  delete copy[key]
+  return copy
+}

package/bench/mock-openai-server.mjs

@@ -0,0 +1,210 @@
+#!/usr/bin/env node
+/**
+ * Scripted OpenAI-compatible chat-completions server for offline tests.
+ *
+ * It replays a fixed list of assistant steps (tool calls or text), one per agent call, and logs
+ * every request so a test can check which model served it, what tool results the model saw,
+ * and whether the shared prompt prefix would have hit a KV/prompt cache. Calls without tools
+ * (side calls such as title generation) get a fixed reply and do not consume a step. Usage is
+ * estimated from the serialized message history (about four characters per token). It is a
+ * plumbing double, not a model: use it to verify routing and context edits, never to measure
+ * quality.
+ *
+ * Usage: node bench/mock-openai-server.mjs --script <steps.json> --log <requests.jsonl> [--port 8123]
+ */
+import { createServer } from 'node:http'
+import { appendFileSync, mkdirSync, readFileSync } from 'node:fs'
+import { dirname } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+/**
+ * @typedef {{ tool_calls?: Array<{ name: string, arguments: Record<string, unknown> }>, content?: string }} Step
+ * @typedef {{ steps: Step[] }} Script
+ */
+
+/**
+ * Start the server.
+ * @param {{ port?: number, steps: Step[], logPath?: string }} options
+ * @returns {Promise<{ url: string, close(): Promise<void>, requests: number }>}
+ */
+export function startMockServer({ port = 0, steps, logPath }) {
+  let requestCount = 0
+  let agentCalls = 0
+  const previousMessages = new Map()
+
+  const server = createServer(async (req, res) => {
+    if (req.method !== 'POST' || !req.url?.endsWith('/chat/completions')) {
+      res.writeHead(404, { 'content-type': 'application/json' })
+      res.end(JSON.stringify({ error: 'mock only serves POST /v1/chat/completions' }))
+      return
+    }
+    const body = JSON.parse(await readBody(req))
+    requestCount += 1
+    const index = requestCount - 1
+    // Agent calls declare tools; side calls (for example, title generation) do not. Only agent
+    // calls consume scripted steps, so a side call cannot shift the script.
+    const agentCall = Array.isArray(body.tools) && body.tools.length > 0
+    const agentStep = agentCall ? agentCalls : null
+    if (agentCall) agentCalls += 1
+    const step = agentCall
+      ? (agentStep < steps.length ? steps[agentStep] : { content: 'Done.' })
+      : { content: 'ok' }
+    const toolCalls = (step.tool_calls ?? []).map((call, i) => ({
+      id: `call_${index}_${i}`,
+      name: call.name,
+      arguments: JSON.stringify(call.arguments ?? {}),
+    }))
+
+    const serialized = JSON.stringify(body.messages ?? [])
+    const model = String(body.model ?? 'unknown')
+    const previous = previousMessages.get(model) ?? ''
+    const promptTokens = Math.ceil(serialized.length / 4)
+    const cachedTokens = Math.ceil(commonPrefixLength(previous, serialized) / 4)
+    previousMessages.set(model, serialized)
+    const completionText = toolCalls.length > 0 ? JSON.stringify(toolCalls.map(c => c.name)) : (step.content ?? '')
+    const completionTokens = Math.max(1, Math.ceil(completionText.length / 4))
+    const usage = {
+      prompt_tokens: promptTokens,
+      completion_tokens: completionTokens,
+      total_tokens: promptTokens + completionTokens,
+      prompt_tokens_details: { cached_tokens: cachedTokens },
+    }
+
+    const messages = body.messages ?? []
+    logRequest(logPath, {
+      n: requestCount, agentCall, agentStep: agentCall ? agentStep + 1 : null,
+      model, stream: body.stream === true, messageCount: messages.length,
+      roles: messages.map(m => m.role),
+      noticeCount: messages.filter(m => contentToText(m.content).includes('[model changed:')).length,
+      promptTokens, cachedTokens, bodyKeys: Object.keys(body).sort(),
+      chatTemplateKwargs: body.chat_template_kwargs ?? null,
+      reasoningEffort: body.reasoning_effort ?? body.reasoning?.effort ?? null,
+      thinkingBudget: body.thinking_token_budget ?? body.thinking_budget ?? null,
+      toolMessages: messages.filter(m => m.role === 'tool').map(m => contentToText(m.content)),
+    })
+
+    const finish = toolCalls.length > 0 ? 'tool_calls' : 'stop'
+    if (body.stream !== true) {
+      res.writeHead(200, { 'content-type': 'application/json' })
+      res.end(JSON.stringify({
+        id: `chatcmpl-${index}`, object: 'chat.completion', created: now(), model,
+        choices: [{
+          index: 0, finish_reason: finish,
+          message: {
+            role: 'assistant', content: step.content ?? null,
+            ...(toolCalls.length > 0 ? { tool_calls: toolCalls.map(toOpenAiToolCall) } : {}),
+          },
+        }],
+        usage,
+      }))
+      return
+    }
+
+    const base = { id: `chatcmpl-${index}`, object: 'chat.completion.chunk', created: now(), model }
+    const chunks = [{ ...base, choices: [{ index: 0, delta: { role: 'assistant', content: '' }, finish_reason: null }] }]
+    if (step.content) chunks.push({ ...base, choices: [{ index: 0, delta: { content: step.content }, finish_reason: null }] })
+    toolCalls.forEach((call, i) => chunks.push({
+      ...base,
+      choices: [{ index: 0, delta: { tool_calls: [{ index: i, ...toOpenAiToolCall(call) }] }, finish_reason: null }],
+    }))
+    chunks.push({ ...base, choices: [{ index: 0, delta: {}, finish_reason: finish }] })
+    chunks.push({ ...base, choices: [], usage })
+
+    res.writeHead(200, { 'content-type': 'text/event-stream', 'cache-control': 'no-cache', connection: 'keep-alive' })
+    for (const chunk of chunks) res.write(`data: ${JSON.stringify(chunk)}\n\n`)
+    res.write('data: [DONE]\n\n')
+    res.end()
+  })
+
+  return new Promise((resolve) => {
+    server.listen(port, '127.0.0.1', () => {
+      const address = server.address()
+      const actualPort = typeof address === 'object' && address !== null ? address.port : port
+      resolve({
+        url: `http://127.0.0.1:${actualPort}/v1`,
+        get requests() { return requestCount },
+        close: () => new Promise((done) => {
+          server.closeAllConnections?.()
+          server.close(() => done())
+        }),
+      })
+    })
+  })
+}
+
+/**
+ * @param {{ id: string, name: string, arguments: string }} call
+ */
+function toOpenAiToolCall(call) {
+  return { id: call.id, type: 'function', function: { name: call.name, arguments: call.arguments } }
+}
+
+/**
+ * @param {import('node:http').IncomingMessage} req
+ * @returns {Promise<string>}
+ */
+function readBody(req) {
+  return new Promise((resolve, reject) => {
+    let data = ''
+    req.setEncoding('utf8')
+    req.on('data', (chunk) => { data += chunk })
+    req.on('end', () => resolve(data))
+    req.on('error', reject)
+  })
+}
+
+/**
+ * @param {string} a
+ * @param {string} b
+ */
+function commonPrefixLength(a, b) {
+  const limit = Math.min(a.length, b.length)
+  let i = 0
+  while (i < limit && a.charCodeAt(i) === b.charCodeAt(i)) i += 1
+  return i
+}
+
+/**
+ * @param {unknown} content
+ */
+function contentToText(content) {
+  if (typeof content === 'string') return content
+  if (Array.isArray(content)) return content.map(part => (part && typeof part === 'object' && 'text' in part ? String(part.text) : '')).join('')
+  return ''
+}
+
+/**
+ * @param {string | undefined} path
+ * @param {Record<string, unknown>} record
+ */
+function logRequest(path, record) {
+  if (!path) return
+  mkdirSync(dirname(path), { recursive: true })
+  appendFileSync(path, `${JSON.stringify({ ts: new Date().toISOString(), ...record })}\n`)
+}
+
+function now() {
+  return Math.floor(Date.now() / 1000)
+}
+
+/**
+ * @param {string[]} argv
+ */
+function parseArgs(argv) {
+  const options = { port: 8123, script: undefined, log: undefined }
+  for (let i = 0; i < argv.length; i += 1) {
+    if (argv[i] === '--port') options.port = Number(argv[++i])
+    else if (argv[i] === '--script') options.script = argv[++i]
+    else if (argv[i] === '--log') options.log = argv[++i]
+  }
+  return options
+}
+
+const isMain = process.argv[1] !== undefined && fileURLToPath(import.meta.url) === process.argv[1]
+if (isMain) {
+  const options = parseArgs(process.argv.slice(2))
+  /** @type {Script} */
+  const script = options.script ? JSON.parse(readFileSync(options.script, 'utf8')) : { steps: [] }
+  const server = await startMockServer({ port: options.port, steps: script.steps, logPath: options.log })
+  console.log(`mock OpenAI server on ${server.url} with ${script.steps.length} scripted steps`)
+}