toolgz 0.1.2 → 0.2.1

package/README.md

@@ -1,6 +1,8 @@
 <h1>toolgz</h1>
 
-<p><strong>Your agent spends 30–50k tokens of context on tool definitions before the user types a word. toolgz gets ~80% of it back.</strong></p>
+<p><strong>Your agent spends 30–70k tokens of context on tool definitions before the user types a word. toolgz gets up to ~85% of it back.</strong></p>
+
+<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–32% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
 
 <p>
 <a href="#measured-results">420-run cross-provider sweep</a> ·
@@ -29,12 +31,18 @@ Reclaiming the room is what this does.
 ## The fix, in three lines
 
 ```ts
-import { compress, forAnthropic } from "toolgz";
+import { compress, recommendLevel, forAnthropic } from "toolgz";
 
-const c = compress(myTools);                  // your existing MCP/SDK tool array
+const { level } = recommendLevel(myTools);    // 1 for small sets, 3 for large ones
+const c = compress(myTools, { level });       // your existing MCP/SDK tool array
 const { tools, system } = forAnthropic(c);    // send these instead
 ```
 
+`compress(myTools)` with no `level` gives you **level 1** — safe, native tool calling,
+provider schema enforcement intact, and 13–32% smaller. The 71–85% figures below are
+**level 3**, which is what `recommendLevel` returns once a tool set is big enough to
+amortise the dispatcher. Ask for it explicitly with `{ level: 3 }` if you prefer.
+
 Then translate the model's call back before you dispatch:
 
 ```ts
@@ -51,24 +59,80 @@ if (r.kind === "call") await myDispatch(r.name, r.args);   // real name, real ar
 Four frontier models, seven strategies, five tool-selection tasks, 3 reps —
 **420 runs** on the current sweep, 1,200+ across all rounds. Every raw per-run record is
 committed in [`bench/results/`](bench/results/); recompute any figure with
-`npx tsx bench/analyze-multi.ts`.
+`npx tsx bench/analyze-multi.ts --sweep=<timestamp>`.
+
+**The table below uses a synthetic catalogue** — 100 realistic-but-invented MCP-style
+tools across 9 namespaces — because it lets us build deliberately confusable clusters
+that a real catalogue may not contain. Two things follow from that, and the first is
+uncomfortable:
+
+- Synthetic naming can flatter a compression style. One map style measured −21% on
+  this fixture because every tool name carried a `namespace_op` prefix to factor out.
+  On real MCP tools, which mostly do not, the same style was worth −1%.
+- Real catalogues are **bigger**, so these numbers understate the problem. A corpus of
+  **149 tools harvested from 14 live MCP servers** measures **68,494 prompt tokens**
+  uncompressed on `claude-opus-5` — more than twice the synthetic fixture, and about a
+  third of a 200k context window before the user types anything.
+
+The real corpus is committed at [`bench/fixtures/real-mcp-tools.json`](bench/fixtures/real-mcp-tools.json)
+with its own scenario suite (`--suite=real`), and it is the corpus of record for any
+claim about real deployments.
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="docs/img/savings-dark.svg">
   <img src="docs/img/savings-light.svg" alt="Prompt tokens saved versus uncompressed tool definitions, by compression level, for each of four providers">
 </picture>
 
-| Provider | Model | Tool block | Prompt tokens | Cost | Latency | Tasks |
-|---|---|---:|---:|---:|---:|:-:|
-| Anthropic | `claude-opus-5` | 9,242 → **1,284** | 30,817 → **4,628** (−85%) | **−78%** | 15.0s → **12.1s** | 15/15 |
-| xAI | `grok-4.5` | 6,421 → **775** | 17,522 → **2,663** (−85%) | **−70%** | 6.1s → **4.6s** | 15/15 |
-| Google | `gemini-3.1-pro-preview` | 5,264 → **732** | 10,948 → **2,302** (−79%) | **−62%** | 5.6s → **5.5s** | 15/15 |
-| OpenAI | `gpt-5.6-sol` | 2,752 → **573** | 7,694 → **2,196** (−71%) | **−7%** | 6.8s → **5.6s** | 15/15 |
+**All figures below are level 3** (`minified-plus`, the shipped default map style).
+Level 1 on the same sweep saves 13–32%; level 2 is dominated by level 3.
+
+| Provider | Model | Tool block | Prompt tokens | Latency | Tasks |
+|---|---|---:|---:|---:|:-:|
+| Anthropic | `claude-opus-5` | 9,242 → **1,284** | 30,817 → **4,628** (**−85%**) | 15.0s → **12.1s** | 15/15 |
+| xAI | `grok-4.5` | 6,421 → **775** | 17,522 → **2,663** (**−85%**) | 6.1s → **4.6s** | 15/15 |
+| Google | `gemini-3.1-pro-preview` | 5,264 → **732** | 10,948 → **2,302** (**−79%**) | 5.6s → **5.5s** | 15/15 |
+| OpenAI | `gpt-5.6-sol` | 2,752 → **573** | 7,694 → **2,196** (**−71%**) | 6.8s → **5.6s** | 15/15 |
 
 Reasoning is enabled on all four at high effort, so this is a like-for-like frontier
 comparison. **60/60 tasks completed, zero hallucinated tool names, zero malformed
 arguments** — and it is faster than uncompressed on every provider.
 
+Recompute any figure with
+`npx tsx bench/analyze-multi.ts --sweep=2026-07-25T19-19` against the raw per-run
+records in [`bench/results/`](bench/results).
+
+### What about cost?
+
+**Cost is not the claim, and we deliberately do not lead with it.** Prompt caching
+already makes tool tokens cheap. What caching does not do is give you the *room* back,
+and the room is what you run out of.
+
+Cost does usually fall as a side effect, by an amount that depends on your provider
+and reasoning settings — and on one of four providers we measured, it does not fall at
+all. On `gpt-5.6-sol` the uncompressed cost distribution is heavily right-skewed (mean
+$0.0172, median $0.0052), so a few expensive runs make compression look break-even
+while the *typical* run gets about 2.5× dearer. We used to publish a "−7% on OpenAI"
+figure. It was a mean over a skewed distribution and we withdrew it.
+
+If you do want to optimise the bill, it is one option — and it is a trade, not a
+freebie:
+
+```ts
+compress(myTools, { level: 3, model: "gpt-5.6-sol", objective: "cost" });
+```
+
+| Model | Median cost vs default |
+|---|---:|
+| `gpt-5.6-sol` | **−20.7%** |
+| `gemini-3.1-pro-preview` | **−15.4%** |
+| `claude-opus-5` | **−9.0%** |
+| `grok-4.5` | **+13.2%** — so it is not enabled there |
+
+From 432 runs, 36 per style per provider, on the real 149-tool corpus. **What you give
+up:** a slightly larger cached map (+275 characters), and on `grok-4.5` a worse bill —
+which is why that row keeps the default rather than the cost-optimised style. Omit
+`objective` and you get the conservative default everywhere, unchanged.
+
 ### It does not make the model worse
 
 That was the thing to disprove, and we tried hard to. The task suite is built from
@@ -86,26 +150,28 @@ problem and looks up what it needs. The default map style exists because of the
 bare tool names failed on `grok-4.5` **deterministically**, 3 of 3 attempts on one scenario,
 answering with zero tool calls and no error raised. Naming the required arguments fixed it.
 
-### Cost, and why it took two rounds to get right
+### How we found the cost story, and got it wrong twice
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="docs/img/cost-dark.svg">
-  <img src="docs/img/cost-light.svg" alt="Prompt tokens and cost both reduced on all four providers">
+  <img src="docs/img/cost-light.svg" alt="Prompt tokens and cost, by compression level, for each of four providers">
 </picture>
 
-The first cross-provider sweep found cost going **up 15% on OpenAI** even while context fell
-69%. The dispatcher was spending extra turns, and on a reasoning model every turn pays for a
-fresh round of thinking.
+The first cross-provider sweep found cost going **up 15% on OpenAI** even while context
+fell 69%. The dispatcher was spending extra turns, and on a reasoning model every turn
+pays for a fresh round of thinking.
 
-So we captured the calls that were being rejected instead of guessing, and found three bugs
-in *this library*: models pass `query` to a parameter named `q` (14 of 18 rejections), they
-sometimes call the map code as the tool name, and they sometimes pass arguments flat instead
-of nested. Fixing all three took OpenAI from **+15% to −7%** and drove malformed arguments to
-**zero on every provider**.
+So we captured the calls being rejected instead of guessing, and found three bugs in
+*this library*: models pass `query` to a parameter named `q` (14 of 18 rejections), they
+sometimes call the map code as the tool name, and they sometimes pass arguments flat
+instead of nested. Fixing all three drove malformed arguments to **zero on every
+provider**. Later rounds found three more of the same kind — a namespace joined with a
+dot, the lookup tool routed through the dispatcher — all shipped in 0.1.2.
 
-OpenAI's −7% is still the smallest saving, and honestly so: reasoning output dominates its
-bill, so a smaller prompt moves the total less. **Context-window occupancy remains the
-primary claim** — cost follows from it, by an amount that depends on your reasoning settings.
+That is the honest shape of this work: **most of the wins came from accepting what
+models actually send, not from making the map smaller.** One extra turn is worth ~3,300
+prompt tokens; the best encoding change available was worth ~550. Six map styles were
+tried and removed in 0.2.0 because they were smaller and still worse.
 
 ---
 
@@ -168,15 +234,576 @@ trip. A test asserts that file matches the code, so it cannot drift.
 
 ---
 
-## Documentation
+## Optional: let the library pick the map style for your model
+
+Level 3 has several map styles. Which one is cheapest turns out to depend on the
+model, so you can hand `compress()` a model id and let it use what was actually
+measured:
+
+```ts
+compress(myTools, { level: 3, model: "gpt-5.6-sol", objective: "cost" });
+```
+
+| Model | Style chosen for `cost` | Measured against the default |
+|---|---|---:|
+| `gpt-5.6-sol` | `explicit` | **−20.7%** |
+| `gemini-3.1-pro-preview` | `explicit` | **−15.4%** |
+| `claude-opus-5` | `explicit` | **−9.0%** |
+| `grok-4.5` | *default* | `explicit` measured **+13.2%** there |
+
+From a 432-run sweep, 36 runs per style per provider, on the real 149-tool corpus.
+`explicit` completed 144/144 tasks and cut lookups on all four providers; only the
+*cost* consequence differs by model. The table lives in
+[`src/policy.generated.ts`](src/policy.generated.ts), is generated from the committed
+results, and a test fails if it drifts from them.
+
+Four things worth knowing:
+
+- **Omitting `model` changes nothing.** Existing behaviour is byte-identical; there
+  is a test asserting that.
+- **`objective` defaults to `occupancy`, which has no table.** Every style we
+  measured landed within ±3.1% of the default on context occupancy — under our 5%
+  effect-size floor — so there is nothing to select. Only `cost` has entries.
+- **An absent model gets the default.** That is an absence of evidence, not a
+  prediction. `gpt-5.6-sol` behaving one way says nothing certain about `gpt-5.7`.
+- **`stats` always tells you what was actually used**, so nothing is substituted
+  silently:
+
+```ts
+const c = compress(myTools, { level: 3, model: "gpt-5.6-sol", objective: "cost" });
+c.stats.mapStyle;          // "explicit" — what was used
+c.stats.requestedMapStyle; // undefined — you did not ask for a specific style
+c.stats.fallbackReason;    // undefined — nothing was substituted
+```
+
+There is also a safety valve: if a future sweep finds a `(model, style)` pair that
+fails, it is refused and `stats.fallbackReason` says why. **That table is currently
+empty** — the one pair ever measured unsafe was `nocode` on `grok-4.5` (19% of runs
+answered with no tool call at all), and rather than document a footgun we deleted the
+style in 0.2.0.
+
+---
+
+## Using it: the full guide
+
+Everything below was a separate `docs/GUIDE.md`. It is inline now, deliberately: the
+same claims lived in both files and drifted apart — the README advertised level 3's
+savings while its own quick-start example used level 1. One document cannot contradict
+itself.
+
+Reference material that is generated or historical stays separate, because it is not
+hand-maintained prose:
+
+- **[docs/RESULTS.md](docs/RESULTS.md)** — every benchmark round, the raw evidence log
+- **[docs/BEFORE-AFTER.md](docs/BEFORE-AFTER.md)** — generated by executing the library
+- **[docs/RELEASING.md](docs/RELEASING.md)** — release process
+
+---
+
+## 1. The agent loop, in full
+
+A complete, working Anthropic loop. The only additions to a normal loop are the
+`compress()` call at the top and the `resolve()` call before dispatch.
+
+```ts
+import Anthropic from "@anthropic-ai/sdk";
+import { compress, forAnthropic } from "toolgz";
+
+const client = new Anthropic();
+const SYSTEM = "You are a helpful operations agent. Use the tools available.";
+
+// 1. Compress once, outside the loop. It is pure and deterministic.
+const c = compress(myTools, { level: 3 });
+const { tools, system } = forAnthropic(c);
+
+export async function run(userMessage: string) {
+  const messages: any[] = [{ role: "user", content: userMessage }];
+
+  for (let turn = 0; turn < 12; turn++) {
+    const res = await client.messages.create({
+      model: "claude-opus-5",
+      max_tokens: 8000,
+      system: [{ type: "text", text: SYSTEM }, ...(system ?? [])],
+      tools: tools as any,
+      messages,
+    });
+
+    const calls = res.content.filter((b: any) => b.type === "tool_use");
+    if (!calls.length) return res;          // model is done
+
+    // 2. Echo the assistant turn back verbatim. On Anthropic this matters:
+    //    thinking blocks must be returned unchanged or the model re-reasons.
+    messages.push({ role: "assistant", content: res.content });
+
+    const results: any[] = [];
+    for (const call of calls) {
+      const r = c.resolve(call.name, (call as any).input);
+
+      if (r.kind === "call") {
+        const output = await myDispatch(r.name, r.args);   // your real handler
+        results.push({
+          type: "tool_result",
+          tool_use_id: call.id,
+          content: JSON.stringify(output),
+        });
+      } else if (r.kind === "meta") {
+        // The model asked toolgz a lookup question. Answer it; don't dispatch.
+        results.push({
+          type: "tool_result",
+          tool_use_id: call.id,
+          content: r.result,
+        });
+      } else {
+        // Bad arguments or an unknown code. The message is written for the
+        // model to read — hand it straight back and let it retry.
+        results.push({
+          type: "tool_result",
+          tool_use_id: call.id,
+          content: `Error: ${r.message}`,
+          is_error: true,
+        });
+      }
+    }
+
+    messages.push({ role: "user", content: results });
+  }
+}
+```
+
+Three rules that matter:
+
+1. **Call `compress()` once**, outside the loop. Calling it per turn is wasteful
+   and, if you vary the options, breaks prompt caching.
+2. **Never dispatch on `meta` or `error`.** Only `kind === "call"` is a real
+   tool invocation.
+3. **Feed errors back rather than throwing.** Recovery is the design; the error
+   strings are written for the model.
+
+---
+
+## 2. Handling the three outcomes
+
+`resolve()` returns a discriminated union. Handle all three.
+
+```ts
+const r = c.resolve(rawName, rawArgs);
+
+switch (r.kind) {
+  case "call":
+    // r.name  → the original tool name, e.g. "github_create_issue"
+    // r.args  → validated against your original schema
+    await myDispatch(r.name, r.args);
+    break;
+
+  case "meta":
+    // The model used a toolgz lookup tool (q / describe_op).
+    // r.result is text to return as the tool result. Do not dispatch.
+    reply(r.result);
+    break;
+
+  case "error":
+    // r.message    → model-readable: names the tool, the parameter, the fix
+    // r.recoverable → true when retrying can plausibly succeed
+    reply(r.message, { isError: true });
+    break;
+}
+```
+
+What produces each:
+
+| Outcome | Cause | You should |
+|---|---|---|
+| `call` | valid invocation | dispatch it |
+| `meta` | model asked for a definition or searched the map | return `r.result` as the tool result |
+| `error` | missing/unknown/wrong-typed argument, or an unknown code | return `r.message` with an error flag |
+
+Argument validation runs against your **original** schema, not the compressed
+one, so an `error` here means the call genuinely would not have worked.
+
+---
+
+## 3. Provider setup
+
+The core is provider-neutral. Adapters handle wire shape and cache placement.
+They are pure functions and never mutate what you pass them.
+
+### Anthropic
+
+```ts
+import { compress, forAnthropic } from "toolgz";
+
+const c = compress(myTools, { level: 3 });
+const { tools, system } = forAnthropic(c);          // or { ttl: "1h" }
+
+await client.messages.create({
+  model: "claude-opus-5",
+  max_tokens: 8000,
+  system: [{ type: "text", text: SYSTEM }, ...(system ?? [])],
+  tools,
+  messages,
+});
+```
+
+`forAnthropic` places exactly one `cache_control` breakpoint on the last
+eligible tool, and skips any tool carrying `defer_loading` because the API
+rejects that combination.
+
+### OpenAI
+
+OpenAI has **two endpoints with two different tool shapes**, and toolgz has an
+adapter for each. Pick by whether you want reasoning.
+
+**If you want tools *and* reasoning — use `/v1/responses`.** On the GPT-5.x
+line, `/v1/chat/completions` refuses the combination outright:
+
+```
+Function tools with reasoning_effort are not supported for gpt-5.6-sol in
+/v1/chat/completions. To use function tools, use /v1/responses or set
+reasoning_effort to 'none'.
+```
+
+```ts
+import { compress, forOpenAIResponses } from "toolgz";
+
+const c = compress(myTools, { level: 3 });
+const { tools, systemPreamble } = forOpenAIResponses(c);   // flat tool shape
+
+const res = await client.responses.create({
+  model: "gpt-5.6-sol",
+  reasoning: { effort: "high" },
+  max_output_tokens: 8000,
+  tools,
+  input: [
+    { type: "message", role: "developer",
+      content: SYSTEM + (systemPreamble ? "\n\n" + systemPreamble : "") },
+    { type: "message", role: "user", content: userMessage },
+  ],
+});
+```
+
+Two things about `/v1/responses` that will bite you otherwise:
+
+- History is a flat `input[]` list, not `messages`. A tool call round-trip is
+  `{type:"function_call", call_id, name, arguments}` followed by
+  `{type:"function_call_output", call_id, output}`.
+- **Reasoning items must be echoed back** alongside tool outputs on the next
+  turn, or the model re-reasons from scratch every turn. Push the whole
+  `response.output` array back verbatim.
+
+**If you don't need reasoning**, chat completions is fine and the tool shape is
+the nested one:
+
+```ts
+import { compress, forOpenAI } from "toolgz";
+
+const { tools, systemPreamble } = forOpenAI(compress(myTools, { level: 3 }));
+
+await client.chat.completions.create({
+  model: "gpt-5.6-sol",
+  messages: [
+    { role: "system", content: SYSTEM + (systemPreamble ? "\n\n" + systemPreamble : "") },
+    ...messages,
+  ],
+  tools,
+});
+```
+
+Either way, OpenAI's prefix caching is automatic with a ~1024-token floor, so
+there is no breakpoint to place — just keep your prefix stable. Cached input
+bills at roughly a tenth of the normal rate.
+
+`forOpenAI` emits `{type:"function", function:{…}}`; `forOpenAIResponses` emits
+the flat `{type:"function", name, …}`. Sending one shape to the other endpoint
+is a validation error, so match them.
+
+### Google Gemini
+
+```ts
+import { compress, forGemini } from "toolgz";
+
+const c = compress(myTools, { level: 3 });
+const { tools, systemPreamble } = forGemini(c);
+
+await ai.models.generateContent({
+  model: "gemini-3.1-pro-preview",
+  contents,
+  config: {
+    systemInstruction: SYSTEM + (systemPreamble ? "\n\n" + systemPreamble : ""),
+    tools,
+  },
+});
+```
+
+`forGemini` strips the JSON Schema keywords Gemini rejects
+(`additionalProperties`, `$schema`, `default`, `examples`) recursively.
+
+### xAI
+
+xAI's API is OpenAI-compatible, so use `forOpenAI`:
+
+```ts
+import OpenAI from "openai";
+import { compress, forOpenAI } from "toolgz";
+
+const client = new OpenAI({
+  apiKey: process.env.XAI_API_KEY,
+  baseURL: "https://api.x.ai/v1",
+});
+const { tools, systemPreamble } = forOpenAI(compress(myTools, { level: 3 }));
+```
 
-| | |
+> **Note:** xAI excludes reasoning tokens from `completion_tokens`. If you're
+> tracking spend, add `usage.completion_tokens_details.reasoning_tokens`.
+
+---
+
+## 4. Prompt caching
+
+**Read this even if you skip everything else.** Prompt caching is what makes the
+per-turn cost of your tool block collapse, and it is easy to break by accident.
+
+Caching is a **prefix match**. Any byte that changes invalidates everything after
+it. Tool definitions render first, so a single reordered tool re-bills your whole
+prompt.
+
+`compress()` is built for this: it sorts tools by name and emits byte-identical
+output for identical input. There is a test asserting it, and that test does not
+get deleted.
+
+What breaks caching — check your own code for these:
+
+| Mistake | Why it breaks |
 |---|---|
-| **[Complete guide](docs/GUIDE.md)** | Install → working agent loop. Per-provider setup for all four, prompt caching, MCP aggregation, troubleshooting. Start here. |
-| **[Before / after](docs/BEFORE-AFTER.md)** | Generated, not illustrated. Both artifacts toolgz modifies, at every level. |
-| **[Full results](docs/RESULTS.md)** | Every number, the methodology, and what it does **not** establish. |
-| **[Announcement drafts](docs/ANNOUNCEMENT.md)** | Ready-to-post write-ups for HN and LinkedIn, plus claims *not* to make. |
-| **[Releasing](docs/RELEASING.md)** | Publishing to npm over OIDC, with no long-lived token. |
+| `new Date().toISOString()` in the system prompt | prefix differs every request |
+| a request id or UUID near the top | same |
+| building tools from an unsorted `Set`/`Map` | serialization order drifts |
+| calling `compress()` with different options per turn | different payload |
+| adding or removing a tool mid-conversation | tool block is position 0 |
+
+Verify it's working — don't assume:
+
+```ts
+const res = await client.messages.create({ /* … */ });
+console.log(res.usage.cache_read_input_tokens);   // should be > 0 after turn 1
+```
+
+If that stays `0` across turns with an identical prefix, something upstream is
+varying. Diff two rendered request bodies to find it.
+
+---
+
+## 5. Composing with Anthropic's native tool search
+
+Anthropic ships server-side tool search (`tool_search_tool_regex_20251119` plus
+`defer_loading: true`). It defers *whole schemas*; toolgz shrinks *each schema*.
+They're orthogonal and compose:
+
+```ts
+const c = compress(myTools, { level: 1 });
+const tools = [
+  { type: "tool_search_tool_regex_20251119", name: "tool_search_tool_regex" },
+  ...(c.tools as any[]).map((t, i) => (i < 5 ? t : { ...t, defer_loading: true })),
+];
+```
+
+Two API constraints: at least one tool must stay non-deferred, and
+`cache_control` cannot go on a deferred tool (`forAnthropic` handles the second).
+
+**One measured caveat.** `defer_loading` hides tools until the model *elects to
+search*. On Claude Opus 5 it elects reliably — 20/20 tasks. On Haiku 4.5 it
+often does not: **6 of 30 tasks**, with four of five scenarios answered in a
+single turn and zero tool calls. No error is raised; the request succeeds and the
+answer is simply unaided.
+
+A dispatcher doesn't share that failure mode, for a structural reason: `t` and
+`q` are ordinary always-visible tools, so the model cannot forget to search —
+searching is the only thing on offer. Deferred loading makes discovery
+*optional*; a dispatcher makes it *the entry point*.
+
+Rule of thumb: compose with native search on frontier models; prefer level 3
+alone below that tier.
+
+---
+
+## 6. MCP servers
+
+MCP tool definitions already match the input shape, so there's nothing to
+convert:
+
+```ts
+const { tools: mcpTools } = await mcpClient.listTools();
+const c = compress(mcpTools, { level: 3 });
+```
+
+Aggregating several servers is where the payoff is largest. Namespace the names
+so the grouping is meaningful:
+
+```ts
+const all = [];
+for (const [serverName, client] of servers) {
+  const { tools } = await client.listTools();
+  all.push(...tools.map((t) => ({ ...t, name: `${serverName}_${t.name}` })));
+}
+
+const c = compress(all, { level: 3 });
+```
+
+Default namespacing splits on the first `_` or `.`. Override it if your names
+don't follow that convention:
+
+```ts
+compress(all, {
+  level: 3,
+  namespaceOf: (name) => {
+    const [ns, ...rest] = name.split("::");
+    return { ns, op: rest.join("::") };
+  },
+});
+```
+
+---
+
+## 7. Troubleshooting
+
+**The model calls `t` or `q` and my dispatcher explodes.**
+You're dispatching on every tool call. Only `kind === "call"` is real; `meta`
+and `error` are toolgz's own tools and must be answered, not dispatched. See
+[§5](#5-handling-the-three-outcomes).
+
+**`resolve()` returns `error: unknown parameter "x"`.**
+The model invented a parameter. Feed `r.message` back as an error tool result —
+it names the accepted parameters and the model will retry. This is expected on
+levels 2 and 3 and is exactly what validation is for.
+
+**Lots of malformed arguments.**
+First check you have not overridden `mapStyle` away from the default
+`"name+required"` — on the current sweep that default produced **zero**
+malformed arguments on all four providers, while the bare-name map still
+produced them. If you are on the default and still seeing errors, try
+`mapStyle: "signature"` (the model then has the optional parameters too), or
+drop to level 1, which keeps provider-side schema enforcement.
+
+The error messages are written to be actionable: a near-miss parameter name is
+named explicitly ("You passed \"query\" — did you mean \"q\"? Rename it."), and a
+wrong-case enum value shows the exact accepted spelling. Feed them straight back
+and the retry usually lands first time.
+
+**`cache_read_input_tokens` is always 0.**
+Something in your prefix varies per request. See [§7](#7-prompt-caching).
+
+**Gemini rejects my schema.**
+Use `forGemini`, which strips the keywords Gemini won't accept. If a new one
+appears, it's a one-line addition to the adapter.
+
+**Savings look small.**
+Check tool count and shape with `recommendLevel(myTools)`. Under ~15 tools
+there's little to reclaim. Also check `c.stats`:
+
+```ts
+console.log(c.stats);
+// { level: 3, toolCount: 100, wireToolCount: 2,
+//   originalChars: 61461, compressedChars: 2211, savedPct: 96.4 }
+```
+
+`savedPct` is a character-count proxy, not a token count — useful for a quick
+sanity check, not for billing.
+
+**I need the real token numbers.**
+Count them with the provider's own endpoint. Never use `tiktoken` for Claude —
+it's OpenAI's tokenizer and is wrong for Claude by 15–20%+.
+
+```ts
+const { input_tokens } = await client.messages.countTokens({
+  model: "claude-opus-5",
+  tools: tools as any,
+  system: SYSTEM,
+  messages: [{ role: "user", content: "x" }],
+});
+```
+
+---
+
+## 8. Benchmarking your own tools
+
+Don't take our numbers for your workload — especially at level 3. The repo ships
+the harness:
+
+```bash
+git clone https://github.com/dperussina/toolgz
+cd toolgz && npm install
+cp .env.example .env          # add your keys
+
+npm test                      # 107 offline tests, no network, no cost
+
+# one provider, one scenario, cheap
+npx tsx bench/harness/run-multi.ts --provider=openai \
+  --scenario=acc-search-vs-list --reps=1
+
+# the full comparison (costs money)
+npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants
+
+npx tsx bench/analyze-multi.ts   # per-arm table + ranking stability
+```
+
+To use your own tools and tasks, edit `bench/fixtures/tools.ts` and
+`bench/scenarios-accuracy.ts`. A scenario declares its tools, the prompt, and
+the expected call, so grading is mechanical rather than model-judged.
+
+Raw per-run JSONL lands in `bench/results/` and is committed, so every published
+number can be recomputed rather than trusted.
+
+---
+
+## 9. API reference
+
+### `compress(tools, options?) → CompressResult`
+
+| Option | Type | Default | Notes |
+|---|---|---|---|
+| `level` | `0 \| 1 \| 2 \| 3` | `1` | see [§3](#3-which-level-should-i-use) |
+| `mapStyle` | `"name+required" \| "explicit" \| "signature"` | `"name+required"` | level 3 only |
+| `namespaceOf` | `(name) => {ns, op}` | split on first `_`/`.` | levels 2–3 grouping |
+| `aliasOf` | `(ns) => string` | identity | level 2 tool naming |
+| `searchLimit` | `number` | `8` | max results from a `q` search |
+| `validate` | `boolean` | `true` | **leave this on** |
+
+Returns:
+
+| Field | Type | Notes |
+|---|---|---|
+| `tools` | `unknown[]` | send these; Anthropic shape |
+| `systemPreamble` | `string` | append to your system prompt; `""` at levels 0–1 |
+| `cachePreamble` | `boolean` | whether the preamble should sit behind a breakpoint |
+| `resolve(name, args)` | `→ Resolution` | translate a model call back |
+| `codeFor(name)` | `→ string` | real name → level-3 code; throws below level 3 |
+| `stats` | `CompressStats` | `level`, `toolCount`, `wireToolCount`, `savedPct`, … |
+
+### `recommendLevel(tools, namespaceOf?) → Recommendation`
+
+`{ level, reason, toolCount, namespaceCount, opsPerNamespace }`. Returns 1 or 3.
+
+### Provider adapters
+
+| Adapter | Endpoint | Tool shape |
+|---|---|---|
+| `forAnthropic(c, { cache?, ttl? })` → `{ tools, system }` | Messages API | Anthropic native; places one `cache_control` breakpoint |
+| `forOpenAI(c)` → `{ tools, systemPreamble }` | `/v1/chat/completions` | nested `{type, function:{…}}` |
+| `forOpenAIResponses(c)` → `{ tools, systemPreamble }` | `/v1/responses` | **flat** `{type, name, …}` — required for tools + reasoning |
+| `forGemini(c)` → `{ tools, systemPreamble }` | `generateContent` | one `functionDeclarations` array |
+
+### Renderers (exported for tooling)
+
+- `signatureLine(tool, nameOverride?)` → `"name(a,b?:x|y)"`
+- `flattenSchema(schema)` → schema with prose and boilerplate removed
+- `countSchemaTokensApprox(value)` → character-count proxy, **not** a token count
+
+---
+
+Measured results and their limits: [RESULTS.md](RESULTS.md).
+Methodology and repo conventions: [../AGENTS.md](../AGENTS.md).
+
+
+---
 
 ## Providers
 
@@ -224,7 +851,7 @@ and it does not get deleted.
 ## Development
 
 ```bash
-npm test        # 131 tests, offline, no cost
+npm test        # 245 tests, offline, no cost
 npm run build   # tsc → dist/ with .d.ts
 
 npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants   # costs money

package/dist/index.d.ts

@@ -13,9 +13,28 @@
  *     if (r.kind === "call") await myDispatch(r.name, r.args);
  *   }
  */
-import type { CompressOptions, CompressResult, Tool } from "./types.js";
+import type { CompressOptions, CompressResult, Tool, MapStyle } from "./types.js";
+import type { PolicyEntry, BrokenEntry } from "./policy.generated.js";
 export * from "./types.js";
 export { flattenSchema, signatureLine, countSchemaTokensApprox } from "./render/index.js";
+export { POLICY, BROKEN, CONSERVATIVE_DEFAULT } from "./policy.generated.js";
+export type { PolicyEntry, BrokenEntry, Objective } from "./policy.generated.js";
+/**
+ * Choose the level-3 map style, and report anything it had to substitute.
+ *
+ * Pure and exported so the disallow path is testable against synthetic tables. It
+ * used to be inline in `compress()`, which meant it could only be exercised through
+ * a real BROKEN entry — and once the one unsafe style was removed from the library
+ * there were none, leaving the safety valve untested.
+ *
+ * Order: an explicit request wins unless measured unsafe for that model; then the
+ * measured policy for (model, objective); then the conservative default.
+ */
+export declare function selectMapStyle(options: Pick<CompressOptions, "mapStyle" | "model" | "objective">, policy?: readonly PolicyEntry[], broken?: readonly BrokenEntry[]): {
+    mapStyle: MapStyle;
+    requestedMapStyle?: MapStyle;
+    fallbackReason?: string;
+};
 export declare function compress(input: Tool[], options?: CompressOptions): CompressResult;
 export { recommendLevel } from "./recommend.js";
 export type { Recommendation } from "./recommend.js";

package/dist/index.js

@@ -1,11 +1,16 @@
-import { countSchemaTokensApprox, defaultNamespaceOf, firstSentence, flattenSchema, normalize, signatureLine, terseDescriptor, } from "./render/index.js";
+import { countSchemaTokensApprox, defaultNamespaceOf, firstSentence, flattenSchema, normalize, signatureLine, } from "./render/index.js";
 import { validateArgs } from "./runtime/validate.js";
 import { nearest } from "./runtime/similar.js";
+import { POLICY, BROKEN, CONSERVATIVE_DEFAULT } from "./policy.generated.js";
 export * from "./types.js";
 export { flattenSchema, signatureLine, countSchemaTokensApprox } from "./render/index.js";
+// Exported so consumers can see WHY a style was chosen. The generated file is not in
+// the published tarball (only dist/ ships), so without this the policy is a black box
+// and the README could point at something a user cannot read.
+export { POLICY, BROKEN, CONSERVATIVE_DEFAULT } from "./policy.generated.js";
 const CODE_CHARS = "abcdefghijklmnopqrstuvwxyz";
 const LEVELS = [0, 1, 2, 3];
-const MAP_STYLES = ["name", "name+required", "signature", "terse"];
+const MAP_STYLES = ["name+required", "explicit", "signature"];
 const err = (message, recoverable = true) => ({
     kind: "error",
     message,
@@ -30,6 +35,41 @@ function asObject(v) {
 function defaultAlias(ns) {
     return ns;
 }
+/**
+ * Choose the level-3 map style, and report anything it had to substitute.
+ *
+ * Pure and exported so the disallow path is testable against synthetic tables. It
+ * used to be inline in `compress()`, which meant it could only be exercised through
+ * a real BROKEN entry — and once the one unsafe style was removed from the library
+ * there were none, leaving the safety valve untested.
+ *
+ * Order: an explicit request wins unless measured unsafe for that model; then the
+ * measured policy for (model, objective); then the conservative default.
+ */
+export function selectMapStyle(options, policy = POLICY, broken = BROKEN) {
+    const objective = options.objective ?? "occupancy";
+    if (options.mapStyle) {
+        const unsafe = options.model
+            ? broken.find((b) => b.model === options.model && b.mapStyle === options.mapStyle)
+            : undefined;
+        if (!unsafe)
+            return { mapStyle: options.mapStyle };
+        // Owner decision: disallow a measured-unsafe pair and fall back, rather than
+        // honour it with a warning. Reported, never silent.
+        return {
+            mapStyle: CONSERVATIVE_DEFAULT,
+            requestedMapStyle: options.mapStyle,
+            fallbackReason: `mapStyle "${options.mapStyle}" is measured unsafe on ${unsafe.model}: ` +
+                `${unsafe.reason} (n=${unsafe.n}, sweep ${unsafe.sweep}). ` +
+                `Using "${CONSERVATIVE_DEFAULT}" instead.`,
+        };
+    }
+    if (options.model) {
+        const hit = policy.find((e) => e.model === options.model && e.objective === objective);
+        return { mapStyle: hit ? hit.mapStyle : CONSERVATIVE_DEFAULT };
+    }
+    return { mapStyle: CONSERVATIVE_DEFAULT };
+}
 export function compress(input, options = {}) {
     const level = (options.level ?? 1);
     if (!LEVELS.includes(level)) {
@@ -44,7 +84,7 @@ export function compress(input, options = {}) {
     // 60/60 tasks, fewer malformed arguments, fewer lookup round-trips, and
     // faster wall-clock than uncompressed on every provider. The larger map pays
     // for itself by removing a discovery turn.
-    const mapStyle = options.mapStyle ?? "name+required";
+    const { mapStyle, requestedMapStyle, fallbackReason, } = selectMapStyle(options, POLICY, BROKEN);
     if (!MAP_STYLES.includes(mapStyle)) {
         throw new Error(`unsupported mapStyle: ${mapStyle} (expected ${MAP_STYLES.join(", ")})`);
     }
@@ -134,6 +174,9 @@ export function compress(input, options = {}) {
             encodeCallForTest: encode,
             stats: {
                 level,
+                mapStyle,
+                requestedMapStyle: requestedMapStyle && requestedMapStyle !== mapStyle ? requestedMapStyle : undefined,
+                fallbackReason,
                 toolCount: tools.length,
                 wireToolCount: wire.length,
                 originalChars,
@@ -256,16 +299,16 @@ export function compress(input, options = {}) {
     // against a full schema's ~400 — to reduce malformed arguments on models that
     // fill the generic argument bag poorly.
     const renderLine = (code, t) => {
-        if (mapStyle === "terse")
-            return `${code} ${terseDescriptor(t.description) || t.name}`;
+        const req = t.schema.required ?? [];
         // Full signature: optional params included, so the model rarely needs q().
         if (mapStyle === "signature")
             return `${code} ${signatureLine(t)}`;
-        if (mapStyle === "name+required") {
-            const req = t.schema.required ?? [];
-            return req.length ? `${code} ${t.name} ${req.join(",")}` : `${code} ${t.name}`;
-        }
-        return `${code} ${t.name}`;
+        if (req.length)
+            return `${code} ${t.name} ${req.join(",")}`;
+        // No required parameters. `explicit` says so, because a bare name is
+        // indistinguishable from a tool whose parameters were omitted, and the model
+        // spends a lookup finding out it could have just called it.
+        return mapStyle === "explicit" ? `${code} ${t.name} ()` : `${code} ${t.name}`;
     };
     const lines = [...codeToTool.entries()].map(([code, t]) => renderLine(code, t));
     const wire = [
@@ -290,12 +333,13 @@ export function compress(input, options = {}) {
             },
         },
     ];
-    const mapLegend = mapStyle === "name+required"
-        ? "Each line is: code name required-args. "
-        : mapStyle === "signature"
-            ? "Each line is: code name(args), where ? marks optional. "
-            : "";
-    const systemPreamble = `<toolmap>\n${lines.join("\n")}\n</toolmap>\n${mapLegend}Invoke with t(f=<code>, a={…}). Use q to expand a code before calling if you are unsure of its parameters.`;
+    const mapLegend = mapStyle === "signature"
+        ? "Each line is: code name(args), where ? marks optional. "
+        : mapStyle === "explicit"
+            ? "Each line is: code name required-args. A line ending in () takes no required arguments and can be called with none. "
+            : "Each line is: code name required-args. ";
+    const invokeHint = "Invoke with t(f=<code>, a={…}). Use q to expand a code before calling if you are unsure of its parameters.";
+    const systemPreamble = `<toolmap>\n${lines.join("\n")}\n</toolmap>\n${mapLegend}${invokeHint}`;
     return finish(wire, systemPreamble, true, (rawName, rawArgs) => {
         let name = rawName;
         let args = asObject(rawArgs);

package/dist/policy.generated.d.ts

@@ -0,0 +1,64 @@
+/**
+ * GENERATED — do not edit by hand. Regenerate with:
+ *   npx tsx bench/generate-policy.ts --sweep=<timestamp>
+ *
+ * A record of measurements, not a theory of models. Every row was produced by
+ * `bench/analyze-multi.ts` scoped to one sweep, compared within a provider, and
+ * filtered to effects of at least 5% — same-arm figures moved by more than that
+ * between n=4 and n=96 during development, so smaller effects are noise.
+ *
+ * Hand-maintaining this would rot exactly the way `bench/` and `src/` diverged
+ * before. A drift test (tests/policy.test.ts) fails if it disagrees with the
+ * committed results.
+ *
+ * There is deliberately NO occupancy table. Every measured difference on that axis
+ * was within ±3.1%, under the floor — so occupancy gets the default, and this file
+ * exists only for the `cost` objective. That is a smaller feature than spec 002
+ * originally described, and it is what the data supports.
+ */
+import type { MapStyle } from "./types.js";
+export type Objective = "occupancy" | "cost";
+export type PolicyEntry = {
+    /** Exact model id. Never a family: gpt-5.6-sol says nothing about gpt-5.7. */
+    model: string;
+    objective: Objective;
+    mapStyle: MapStyle;
+    /** Percent change against the conservative default. Negative is better. */
+    effectPct: number;
+    /** Runs behind this row, per arm. */
+    n: number;
+    /** The sweep it came from, so any row can be re-derived. */
+    sweep: string;
+    measured: string;
+};
+/**
+ * Tier-3 sweep, 432 runs, 36 per arm per provider, 12 real MCP scenarios × 3 reps.
+ * `explicit` was 144/144 tasks; turns and lookups fell on all four providers.
+ * Only rows clearing the 5% floor appear.
+ */
+export declare const POLICY: readonly PolicyEntry[];
+/**
+ * (model, mapStyle) pairs measured as unsafe. A caller asking for one is refused and
+ * falls back, per the owner's decision (#15) to disallow rather than warn.
+ */
+export type BrokenEntry = {
+    model: string;
+    mapStyle: MapStyle;
+    reason: string;
+    n: number;
+    sweep: string;
+};
+/**
+ * Empty, and that is the current truth rather than an oversight.
+ *
+ * The one pair ever measured unsafe was `nocode` on grok-4.5 — a 19% silent
+ * failure rate, the model answering with no tool call at all. In 0.2.0 that style
+ * was removed from the library outright, so the pair can no longer be requested.
+ * Deleting a footgun beats documenting it.
+ *
+ * The mechanism stays because it is the safety valve: if a future sweep finds a
+ * (model, style) pair that fails, it goes here and is refused at the boundary.
+ */
+export declare const BROKEN: readonly BrokenEntry[];
+/** The style used when nothing is known about the model. */
+export declare const CONSERVATIVE_DEFAULT: MapStyle;

package/dist/policy.generated.js

@@ -0,0 +1,27 @@
+/**
+ * Tier-3 sweep, 432 runs, 36 per arm per provider, 12 real MCP scenarios × 3 reps.
+ * `explicit` was 144/144 tasks; turns and lookups fell on all four providers.
+ * Only rows clearing the 5% floor appear.
+ */
+export const POLICY = [
+    { model: "claude-opus-5", objective: "cost", mapStyle: "explicit", effectPct: -9.0, n: 36, sweep: "2026-07-26T03-07-25", measured: "2026-07-26" },
+    { model: "gemini-3.1-pro-preview", objective: "cost", mapStyle: "explicit", effectPct: -15.4, n: 36, sweep: "2026-07-26T03-07-25", measured: "2026-07-26" },
+    { model: "gpt-5.6-sol", objective: "cost", mapStyle: "explicit", effectPct: -20.7, n: 36, sweep: "2026-07-26T03-07-25", measured: "2026-07-26" },
+    // grok-4.5 is deliberately absent for `cost`: explicit measured +13.2% there, so it
+    // falls through to the conservative default. An absent row means "no measured
+    // improvement", never "untested".
+];
+/**
+ * Empty, and that is the current truth rather than an oversight.
+ *
+ * The one pair ever measured unsafe was `nocode` on grok-4.5 — a 19% silent
+ * failure rate, the model answering with no tool call at all. In 0.2.0 that style
+ * was removed from the library outright, so the pair can no longer be requested.
+ * Deleting a footgun beats documenting it.
+ *
+ * The mechanism stays because it is the safety valve: if a future sweep finds a
+ * (model, style) pair that fails, it goes here and is refused at the boundary.
+ */
+export const BROKEN = [];
+/** The style used when nothing is known about the model. */
+export const CONSERVATIVE_DEFAULT = "name+required";

package/dist/types.d.ts

@@ -34,27 +34,41 @@ export type NormalizedTool = {
  *  3  minified           — single dispatcher + opaque codes
  */
 export type Level = 0 | 1 | 2 | 3;
+import type { Objective } from "./policy.generated.js";
+export type { Objective };
 /**
  * How each line of the level-3 `<toolmap>` is rendered.
  *
- *   name           `a0 github_create_issue`
- *   name+required  `a0 github_create_issue owner,repo,title`
+ *   name+required  `a0 github_create_issue owner,repo,title`   <- default
+ *   explicit       `a0 github_create_issue owner,repo,title`
+ *                  `a0 scorecard_lf_daily ()`  <- () = takes no required args
  *   signature      `a0 github_create_issue(owner,repo,title,body?,labels?)`
- *   terse          `a0 create new issue in repository`
  *
- * `name` is the default and the smallest. `name+required` costs a few tokens
- * per tool and exists to cut malformed arguments on models that fill the
- * generic argument bag badly — the dispatcher levels give up provider-side
- * constrained decoding, and this buys some of it back cheaply. `signature` also
- * names the optional parameters, which removes most remaining `q()` lookups —
- * a bigger cached map traded for fewer turns, which matters on models where
- * every turn pays for a fresh round of reasoning. `terse` drops the real name
- * entirely; it is the most aggressive and the least legible.
+ * Three styles, each with a measured reason to exist. Six others were tried and
+ * removed in 0.2.0 because they made things worse — see docs/RESULTS.md Round 6.
+ *
+ * `name+required` is the default: the only style perfect on all four providers with
+ * zero malformed arguments. It names required parameters because the dispatcher
+ * levels give up provider-side constrained decoding, and a few tokens per tool buys
+ * some of it back.
+ *
+ * `explicit` adds one thing: a `()` marker on tools that declare no required
+ * parameters. On a real catalogue 44% of tools are like that, and their line would
+ * otherwise be a bare name — indistinguishable from a tool whose parameters were
+ * omitted, so the model spends a lookup finding out it could just call it. Measured
+ * over 432 runs it cut lookups on all four providers and cost 9–21% less on three,
+ * but **13% more on grok-4.5**. Reach for it via `objective: "cost"` rather than
+ * setting it by hand.
+ *
+ * `signature` names optional parameters too, which removes most remaining lookups.
+ * A bigger cached map traded for fewer turns — worth it where every turn pays for a
+ * fresh round of reasoning, and measurably worse on xAI for reasons we cannot yet
+ * explain.
  */
-export type MapStyle = "name" | "name+required" | "signature" | "terse";
+export type MapStyle = "name+required" | "explicit" | "signature";
 export type CompressOptions = {
     level?: Level;
-    /** Level 3 only. Ignored at levels 0–2, which emit no map. Default "name". */
+    /** Level 3 only. Ignored at levels 0–2, which emit no map. Default "name+required". */
     mapStyle?: MapStyle;
     /**
      * Group tools into namespaces. Default splits on the first `_` or `.`,
@@ -70,6 +84,25 @@ export type CompressOptions = {
     searchLimit?: number;
     /** Validate arguments against the original schema before dispatch. Default true. */
     validate?: boolean;
+    /**
+     * Exact model id, e.g. "gpt-5.6-sol". When given, level 3 selects the best
+     * *measured* map style for that model from `src/policy.generated.ts`.
+     *
+     * Never a family: `gpt-5.6-sol` failing says nothing certain about `gpt-5.7`. An
+     * unknown model gets the conservative default — an absence of evidence, not a
+     * prediction. Behaviour is unchanged if you omit this.
+     */
+    model?: string;
+    /**
+     * What the selection optimises. Default "occupancy", because reclaiming context
+     * window is the point — prompt caching already makes tool tokens cheap but does not
+     * reclaim the room they take.
+     *
+     * As measured, "occupancy" currently selects the default on every known model: no
+     * style beat it by more than 3.1% on that axis, under the 5% floor. The table has
+     * real entries only for "cost".
+     */
+    objective?: Objective;
 };
 export type Resolution = {
     kind: "call";
@@ -86,6 +119,12 @@ export type Resolution = {
 };
 export type CompressStats = {
     level: Level;
+    /** The map style actually used. */
+    mapStyle?: MapStyle;
+    /** What the caller asked for, when it differed from what was used. */
+    requestedMapStyle?: MapStyle;
+    /** Why a requested style was not used. Absent when nothing was substituted. */
+    fallbackReason?: string;
     toolCount: number;
     wireToolCount: number;
     originalChars: number;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toolgz",
-  "version": "0.1.2",
+  "version": "0.2.1",
   "description": "Compress LLM tool definitions to reclaim context window. Measured across Anthropic, OpenAI, Google and xAI.",
   "main": "dist/index.js",
   "scripts": {
@@ -9,7 +9,10 @@
     "bench": "tsx bench/harness/run.ts",
     "build": "tsc",
     "prepublishOnly": "npm run build && npm test",
-    "test:live": "vitest run tests/integration"
+    "test:live": "vitest run tests/integration",
+    "bench:tier1": "tsx -r dotenv/config bench/harness/run-multi.ts --provider=all --suite=real --variants --reps=1 --scenario=real-daily-vs-weekly,real-overwrite-vs-append,real-status-vs-result,real-rollup-vs-detail",
+    "bench:tier2": "tsx -r dotenv/config bench/harness/run-multi.ts --provider=all --suite=real --variants --reps=2 --scenario=real-daily-vs-weekly,real-overwrite-vs-append,real-status-vs-result,real-rollup-vs-detail,real-two-step-scheduling,real-summary-vs-details",
+    "bench:tier3": "tsx -r dotenv/config bench/harness/run-multi.ts --provider=all --suite=real --variants --reps=3"
   },
   "keywords": [
     "llm",