toolgz 0.2.0 → 0.2.2

package/README.md

@@ -1,6 +1,8 @@
 <h1>toolgz</h1>
 
-<p><strong>Your agent spends 30–70k tokens of context on tool definitions before the user types a word. toolgz gets ~80% of it back.</strong></p>
+<p><strong>Your agent spends 30–70k tokens of context on tool definitions before the user types a word. toolgz gets up to ~85% of it back.</strong></p>
+
+<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–32% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
 
 <p>
 <a href="#measured-results">420-run cross-provider sweep</a> ·
@@ -29,12 +31,18 @@ Reclaiming the room is what this does.
 ## The fix, in three lines
 
 ```ts
-import { compress, forAnthropic } from "toolgz";
+import { compress, recommendLevel, forAnthropic } from "toolgz";
 
-const c = compress(myTools);                  // your existing MCP/SDK tool array
+const { level } = recommendLevel(myTools);    // 1 for small sets, 3 for large ones
+const c = compress(myTools, { level });       // your existing MCP/SDK tool array
 const { tools, system } = forAnthropic(c);    // send these instead
 ```
 
+`compress(myTools)` with no `level` gives you **level 1** — safe, native tool calling,
+provider schema enforcement intact, and 13–32% smaller. The 71–85% figures below are
+**level 3**, which is what `recommendLevel` returns once a tool set is big enough to
+amortise the dispatcher. Ask for it explicitly with `{ level: 3 }` if you prefer.
+
 Then translate the model's call back before you dispatch:
 
 ```ts
@@ -75,6 +83,9 @@ claim about real deployments.
   <img src="docs/img/savings-light.svg" alt="Prompt tokens saved versus uncompressed tool definitions, by compression level, for each of four providers">
 </picture>
 
+**All figures below are level 3** (`minified-plus`, the shipped default map style).
+Level 1 on the same sweep saves 13–32%; level 2 is dominated by level 3.
+
 | Provider | Model | Tool block | Prompt tokens | Latency | Tasks |
 |---|---|---:|---:|---:|:-:|
 | Anthropic | `claude-opus-5` | 9,242 → **1,284** | 30,817 → **4,628** (**−85%**) | 15.0s → **12.1s** | 15/15 |
@@ -273,15 +284,534 @@ style in 0.2.0.
 
 ---
 
-## Documentation
+## Using it: the full guide
+
+Everything below was a separate `docs/GUIDE.md`. It is inline now, deliberately: the
+same claims lived in both files and drifted apart — the README advertised level 3's
+savings while its own quick-start example used level 1. One document cannot contradict
+itself.
+
+Reference material that is generated or historical stays separate, because it is not
+hand-maintained prose:
+
+- **[docs/RESULTS.md](docs/RESULTS.md)** — every benchmark round, the raw evidence log
+- **[docs/BEFORE-AFTER.md](docs/BEFORE-AFTER.md)** — generated by executing the library
+- **[docs/RELEASING.md](docs/RELEASING.md)** — release process
+
+---
+
+## 1. The agent loop, in full
+
+A complete, working Anthropic loop. The only additions to a normal loop are the
+`compress()` call at the top and the `resolve()` call before dispatch.
+
+```ts
+import Anthropic from "@anthropic-ai/sdk";
+import { compress, forAnthropic } from "toolgz";
+
+const client = new Anthropic();
+const SYSTEM = "You are a helpful operations agent. Use the tools available.";
+
+// 1. Compress once, outside the loop. It is pure and deterministic.
+const c = compress(myTools, { level: 3 });
+const { tools, system } = forAnthropic(c);
+
+export async function run(userMessage: string) {
+  const messages: any[] = [{ role: "user", content: userMessage }];
+
+  for (let turn = 0; turn < 12; turn++) {
+    const res = await client.messages.create({
+      model: "claude-opus-5",
+      max_tokens: 8000,
+      system: [{ type: "text", text: SYSTEM }, ...(system ?? [])],
+      tools: tools as any,
+      messages,
+    });
+
+    const calls = res.content.filter((b: any) => b.type === "tool_use");
+    if (!calls.length) return res;          // model is done
+
+    // 2. Echo the assistant turn back verbatim. On Anthropic this matters:
+    //    thinking blocks must be returned unchanged or the model re-reasons.
+    messages.push({ role: "assistant", content: res.content });
+
+    const results: any[] = [];
+    for (const call of calls) {
+      const r = c.resolve(call.name, (call as any).input);
+
+      if (r.kind === "call") {
+        const output = await myDispatch(r.name, r.args);   // your real handler
+        results.push({
+          type: "tool_result",
+          tool_use_id: call.id,
+          content: JSON.stringify(output),
+        });
+      } else if (r.kind === "meta") {
+        // The model asked toolgz a lookup question. Answer it; don't dispatch.
+        results.push({
+          type: "tool_result",
+          tool_use_id: call.id,
+          content: r.result,
+        });
+      } else {
+        // Bad arguments or an unknown code. The message is written for the
+        // model to read — hand it straight back and let it retry.
+        results.push({
+          type: "tool_result",
+          tool_use_id: call.id,
+          content: `Error: ${r.message}`,
+          is_error: true,
+        });
+      }
+    }
+
+    messages.push({ role: "user", content: results });
+  }
+}
+```
+
+Three rules that matter:
+
+1. **Call `compress()` once**, outside the loop. Calling it per turn is wasteful
+   and, if you vary the options, breaks prompt caching.
+2. **Never dispatch on `meta` or `error`.** Only `kind === "call"` is a real
+   tool invocation.
+3. **Feed errors back rather than throwing.** Recovery is the design; the error
+   strings are written for the model.
+
+---
+
+## 2. Handling the three outcomes
+
+`resolve()` returns a discriminated union. Handle all three.
+
+```ts
+const r = c.resolve(rawName, rawArgs);
+
+switch (r.kind) {
+  case "call":
+    // r.name  → the original tool name, e.g. "github_create_issue"
+    // r.args  → validated against your original schema
+    await myDispatch(r.name, r.args);
+    break;
+
+  case "meta":
+    // The model used a toolgz lookup tool (q / describe_op).
+    // r.result is text to return as the tool result. Do not dispatch.
+    reply(r.result);
+    break;
+
+  case "error":
+    // r.message    → model-readable: names the tool, the parameter, the fix
+    // r.recoverable → true when retrying can plausibly succeed
+    reply(r.message, { isError: true });
+    break;
+}
+```
+
+What produces each:
+
+| Outcome | Cause | You should |
+|---|---|---|
+| `call` | valid invocation | dispatch it |
+| `meta` | model asked for a definition or searched the map | return `r.result` as the tool result |
+| `error` | missing/unknown/wrong-typed argument, or an unknown code | return `r.message` with an error flag |
+
+Argument validation runs against your **original** schema, not the compressed
+one, so an `error` here means the call genuinely would not have worked.
+
+---
+
+## 3. Provider setup
+
+The core is provider-neutral. Adapters handle wire shape and cache placement.
+They are pure functions and never mutate what you pass them.
+
+### Anthropic
+
+```ts
+import { compress, forAnthropic } from "toolgz";
+
+const c = compress(myTools, { level: 3 });
+const { tools, system } = forAnthropic(c);          // or { ttl: "1h" }
+
+await client.messages.create({
+  model: "claude-opus-5",
+  max_tokens: 8000,
+  system: [{ type: "text", text: SYSTEM }, ...(system ?? [])],
+  tools,
+  messages,
+});
+```
+
+`forAnthropic` places exactly one `cache_control` breakpoint on the last
+eligible tool, and skips any tool carrying `defer_loading` because the API
+rejects that combination.
 
-| | |
+### OpenAI
+
+OpenAI has **two endpoints with two different tool shapes**, and toolgz has an
+adapter for each. Pick by whether you want reasoning.
+
+**If you want tools *and* reasoning — use `/v1/responses`.** On the GPT-5.x
+line, `/v1/chat/completions` refuses the combination outright:
+
+```
+Function tools with reasoning_effort are not supported for gpt-5.6-sol in
+/v1/chat/completions. To use function tools, use /v1/responses or set
+reasoning_effort to 'none'.
+```
+
+```ts
+import { compress, forOpenAIResponses } from "toolgz";
+
+const c = compress(myTools, { level: 3 });
+const { tools, systemPreamble } = forOpenAIResponses(c);   // flat tool shape
+
+const res = await client.responses.create({
+  model: "gpt-5.6-sol",
+  reasoning: { effort: "high" },
+  max_output_tokens: 8000,
+  tools,
+  input: [
+    { type: "message", role: "developer",
+      content: SYSTEM + (systemPreamble ? "\n\n" + systemPreamble : "") },
+    { type: "message", role: "user", content: userMessage },
+  ],
+});
+```
+
+Two things about `/v1/responses` that will bite you otherwise:
+
+- History is a flat `input[]` list, not `messages`. A tool call round-trip is
+  `{type:"function_call", call_id, name, arguments}` followed by
+  `{type:"function_call_output", call_id, output}`.
+- **Reasoning items must be echoed back** alongside tool outputs on the next
+  turn, or the model re-reasons from scratch every turn. Push the whole
+  `response.output` array back verbatim.
+
+**If you don't need reasoning**, chat completions is fine and the tool shape is
+the nested one:
+
+```ts
+import { compress, forOpenAI } from "toolgz";
+
+const { tools, systemPreamble } = forOpenAI(compress(myTools, { level: 3 }));
+
+await client.chat.completions.create({
+  model: "gpt-5.6-sol",
+  messages: [
+    { role: "system", content: SYSTEM + (systemPreamble ? "\n\n" + systemPreamble : "") },
+    ...messages,
+  ],
+  tools,
+});
+```
+
+Either way, OpenAI's prefix caching is automatic with a ~1024-token floor, so
+there is no breakpoint to place — just keep your prefix stable. Cached input
+bills at roughly a tenth of the normal rate.
+
+`forOpenAI` emits `{type:"function", function:{…}}`; `forOpenAIResponses` emits
+the flat `{type:"function", name, …}`. Sending one shape to the other endpoint
+is a validation error, so match them.
+
+### Google Gemini
+
+```ts
+import { compress, forGemini } from "toolgz";
+
+const c = compress(myTools, { level: 3 });
+const { tools, systemPreamble } = forGemini(c);
+
+await ai.models.generateContent({
+  model: "gemini-3.1-pro-preview",
+  contents,
+  config: {
+    systemInstruction: SYSTEM + (systemPreamble ? "\n\n" + systemPreamble : ""),
+    tools,
+  },
+});
+```
+
+`forGemini` strips the JSON Schema keywords Gemini rejects
+(`additionalProperties`, `$schema`, `default`, `examples`) recursively.
+
+### xAI
+
+xAI's API is OpenAI-compatible, so use `forOpenAI`:
+
+```ts
+import OpenAI from "openai";
+import { compress, forOpenAI } from "toolgz";
+
+const client = new OpenAI({
+  apiKey: process.env.XAI_API_KEY,
+  baseURL: "https://api.x.ai/v1",
+});
+const { tools, systemPreamble } = forOpenAI(compress(myTools, { level: 3 }));
+```
+
+> **Note:** xAI excludes reasoning tokens from `completion_tokens`. If you're
+> tracking spend, add `usage.completion_tokens_details.reasoning_tokens`.
+
+---
+
+## 4. Prompt caching
+
+**Read this even if you skip everything else.** Prompt caching is what makes the
+per-turn cost of your tool block collapse, and it is easy to break by accident.
+
+Caching is a **prefix match**. Any byte that changes invalidates everything after
+it. Tool definitions render first, so a single reordered tool re-bills your whole
+prompt.
+
+`compress()` is built for this: it sorts tools by name and emits byte-identical
+output for identical input. There is a test asserting it, and that test does not
+get deleted.
+
+What breaks caching — check your own code for these:
+
+| Mistake | Why it breaks |
 |---|---|
-| **[Complete guide](docs/GUIDE.md)** | Install → working agent loop. Per-provider setup for all four, prompt caching, MCP aggregation, troubleshooting. Start here. |
-| **[Before / after](docs/BEFORE-AFTER.md)** | Generated, not illustrated. Both artifacts toolgz modifies, at every level. |
-| **[Full results](docs/RESULTS.md)** | Every number, the methodology, and what it does **not** establish. |
-| **[Announcement drafts](docs/ANNOUNCEMENT.md)** | Ready-to-post write-ups for HN and LinkedIn, plus claims *not* to make. |
-| **[Releasing](docs/RELEASING.md)** | Publishing to npm over OIDC, with no long-lived token. |
+| `new Date().toISOString()` in the system prompt | prefix differs every request |
+| a request id or UUID near the top | same |
+| building tools from an unsorted `Set`/`Map` | serialization order drifts |
+| calling `compress()` with different options per turn | different payload |
+| adding or removing a tool mid-conversation | tool block is position 0 |
+
+Verify it's working — don't assume:
+
+```ts
+const res = await client.messages.create({ /* … */ });
+console.log(res.usage.cache_read_input_tokens);   // should be > 0 after turn 1
+```
+
+If that stays `0` across turns with an identical prefix, something upstream is
+varying. Diff two rendered request bodies to find it.
+
+---
+
+## 5. Composing with Anthropic's native tool search
+
+Anthropic ships server-side tool search (`tool_search_tool_regex_20251119` plus
+`defer_loading: true`). It defers *whole schemas*; toolgz shrinks *each schema*.
+They're orthogonal and compose:
+
+```ts
+const c = compress(myTools, { level: 1 });
+const tools = [
+  { type: "tool_search_tool_regex_20251119", name: "tool_search_tool_regex" },
+  ...(c.tools as any[]).map((t, i) => (i < 5 ? t : { ...t, defer_loading: true })),
+];
+```
+
+Two API constraints: at least one tool must stay non-deferred, and
+`cache_control` cannot go on a deferred tool (`forAnthropic` handles the second).
+
+**One measured caveat.** `defer_loading` hides tools until the model *elects to
+search*. On Claude Opus 5 it elects reliably — 20/20 tasks. On Haiku 4.5 it
+often does not: **6 of 30 tasks**, with four of five scenarios answered in a
+single turn and zero tool calls. No error is raised; the request succeeds and the
+answer is simply unaided.
+
+A dispatcher doesn't share that failure mode, for a structural reason: `t` and
+`q` are ordinary always-visible tools, so the model cannot forget to search —
+searching is the only thing on offer. Deferred loading makes discovery
+*optional*; a dispatcher makes it *the entry point*.
+
+Rule of thumb: compose with native search on frontier models; prefer level 3
+alone below that tier.
+
+---
+
+## 6. MCP servers
+
+MCP tool definitions already match the input shape, so there's nothing to
+convert:
+
+```ts
+const { tools: mcpTools } = await mcpClient.listTools();
+const c = compress(mcpTools, { level: 3 });
+```
+
+Aggregating several servers is where the payoff is largest. Namespace the names
+so the grouping is meaningful:
+
+```ts
+const all = [];
+for (const [serverName, client] of servers) {
+  const { tools } = await client.listTools();
+  all.push(...tools.map((t) => ({ ...t, name: `${serverName}_${t.name}` })));
+}
+
+const c = compress(all, { level: 3 });
+```
+
+Default namespacing splits on the first `_` or `.`. Override it if your names
+don't follow that convention:
+
+```ts
+compress(all, {
+  level: 3,
+  namespaceOf: (name) => {
+    const [ns, ...rest] = name.split("::");
+    return { ns, op: rest.join("::") };
+  },
+});
+```
+
+---
+
+## 7. Troubleshooting
+
+**The model calls `t` or `q` and my dispatcher explodes.**
+You're dispatching on every tool call. Only `kind === "call"` is real; `meta`
+and `error` are toolgz's own tools and must be answered, not dispatched. See
+[§2](#2-handling-the-three-outcomes).
+
+**`resolve()` returns `error: unknown parameter "x"`.**
+The model invented a parameter. Feed `r.message` back as an error tool result —
+it names the accepted parameters and the model will retry. This is expected on
+levels 2 and 3 and is exactly what validation is for.
+
+**Lots of malformed arguments.**
+First check you have not overridden `mapStyle` away from the default
+`"name+required"` — on the current sweep that default produced **zero**
+malformed arguments on all four providers, while the bare-name map still
+produced them. If you are on the default and still seeing errors, try
+`mapStyle: "signature"` (the model then has the optional parameters too), or
+drop to level 1, which keeps provider-side schema enforcement.
+
+The error messages are written to be actionable: a near-miss parameter name is
+named explicitly ("You passed \"query\" — did you mean \"q\"? Rename it."), and a
+wrong-case enum value shows the exact accepted spelling. Feed them straight back
+and the retry usually lands first time.
+
+**`cache_read_input_tokens` is always 0.**
+Something in your prefix varies per request. See [§4](#4-prompt-caching).
+
+**Gemini rejects my schema.**
+Use `forGemini`, which strips the keywords Gemini won't accept. If a new one
+appears, it's a one-line addition to the adapter.
+
+**Savings look small.**
+Check tool count and shape with `recommendLevel(myTools)`. Under ~15 tools
+there's little to reclaim. Also check `c.stats`:
+
+```ts
+console.log(c.stats);
+// { level: 3, toolCount: 100, wireToolCount: 2,
+//   originalChars: 61461, compressedChars: 2211, savedPct: 96.4 }
+```
+
+`savedPct` is a character-count proxy, not a token count — useful for a quick
+sanity check, not for billing.
+
+**I need the real token numbers.**
+Count them with the provider's own endpoint. Never use `tiktoken` for Claude —
+it's OpenAI's tokenizer and is wrong for Claude by 15–20%+.
+
+```ts
+const { input_tokens } = await client.messages.countTokens({
+  model: "claude-opus-5",
+  tools: tools as any,
+  system: SYSTEM,
+  messages: [{ role: "user", content: "x" }],
+});
+```
+
+---
+
+## 8. Benchmarking your own tools
+
+Don't take our numbers for your workload — especially at level 3. The repo ships
+the harness:
+
+```bash
+git clone https://github.com/dperussina/toolgz
+cd toolgz && npm install
+cp .env.example .env          # add your keys
+
+npm test                      # 107 offline tests, no network, no cost
+
+# one provider, one scenario, cheap
+npx tsx bench/harness/run-multi.ts --provider=openai \
+  --scenario=acc-search-vs-list --reps=1
+
+# the full comparison (costs money)
+npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants
+
+npx tsx bench/analyze-multi.ts   # per-arm table + ranking stability
+```
+
+To use your own tools and tasks, edit `bench/fixtures/tools.ts` and
+`bench/scenarios-accuracy.ts`. A scenario declares its tools, the prompt, and
+the expected call, so grading is mechanical rather than model-judged.
+
+Raw per-run JSONL lands in `bench/results/` and is committed, so every published
+number can be recomputed rather than trusted.
+
+---
+
+## 9. API reference
+
+### `compress(tools, options?) → CompressResult`
+
+| Option | Type | Default | Notes |
+|---|---|---|---|
+| `level` | `0 \| 1 \| 2 \| 3` | `1` | see [Which level to use](#which-level-to-use) |
+| `mapStyle` | `"name+required" \| "explicit" \| "signature"` | `"name+required"` | level 3 only |
+| `namespaceOf` | `(name) => {ns, op}` | split on first `_`/`.` | levels 2–3 grouping |
+| `aliasOf` | `(ns) => string` | identity | level 2 tool naming |
+| `searchLimit` | `number` | `8` | max results from a `q` search |
+| `validate` | `boolean` | `true` | **leave this on** |
+| `model` | `string` | — | exact model id; picks the measured style. Omit and nothing changes |
+| `objective` | `"occupancy" \| "cost"` | `"occupancy"` | what the pick optimises. Only `cost` has entries |
+
+Returns:
+
+| Field | Type | Notes |
+|---|---|---|
+| `tools` | `unknown[]` | send these; Anthropic shape |
+| `systemPreamble` | `string` | append to your system prompt; `""` at levels 0–1 |
+| `cachePreamble` | `boolean` | whether the preamble should sit behind a breakpoint |
+| `resolve(name, args)` | `→ Resolution` | translate a model call back |
+| `codeFor(name)` | `→ string` | real name → level-3 code; throws below level 3 |
+| `stats` | `CompressStats` | `level`, `mapStyle`, `requestedMapStyle`, `fallbackReason`, `toolCount`, `wireToolCount`, `originalChars`, `compressedChars`, `savedPct` |
+| `encodeCallForTest(name, args)` | `→ {name, args}` | build the raw call a model would emit; test aid |
+
+### `recommendLevel(tools, namespaceOf?) → Recommendation`
+
+`{ level, reason, toolCount, namespaceCount, opsPerNamespace }`. Returns 1 or 3.
+
+### Provider adapters
+
+| Adapter | Endpoint | Tool shape |
+|---|---|---|
+| `forAnthropic(c, { cache?, ttl? })` → `{ tools, system }` | Messages API | Anthropic native; places one `cache_control` breakpoint |
+| `forOpenAI(c)` → `{ tools, systemPreamble }` | `/v1/chat/completions` | nested `{type, function:{…}}` |
+| `forOpenAIResponses(c)` → `{ tools, systemPreamble }` | `/v1/responses` | **flat** `{type, name, …}` — required for tools + reasoning |
+| `forGemini(c)` → `{ tools, systemPreamble }` | `generateContent` | one `functionDeclarations` array |
+
+Also exported: `recommendLevel(tools)` → `{ level, reason }`; `selectMapStyle(options)`
+→ `{ mapStyle, requestedMapStyle?, fallbackReason? }` (pure, so you can see a pick
+without compressing); and `POLICY`, `BROKEN`, `CONSERVATIVE_DEFAULT` — the measured
+table, so a style choice is never a black box.
+
+### Renderers (exported for tooling)
+
+- `signatureLine(tool, nameOverride?)` → `"name(a,b?:x|y)"`
+- `flattenSchema(schema)` → schema with prose and boilerplate removed
+- `countSchemaTokensApprox(value)` → character-count proxy, **not** a token count
+
+---
+
+Measured results and their limits: [RESULTS.md](RESULTS.md).
+Methodology and repo conventions: [../AGENTS.md](../AGENTS.md).
+
+
+---
 
 ## Providers
 
@@ -329,7 +859,7 @@ and it does not get deleted.
 ## Development
 
 ```bash
-npm test        # 131 tests, offline, no cost
+npm test        # 251 tests, offline, no cost
 npm run build   # tsc → dist/ with .d.ts
 
 npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants   # costs money

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toolgz",
-  "version": "0.2.0",
+  "version": "0.2.2",
   "description": "Compress LLM tool definitions to reclaim context window. Measured across Anthropic, OpenAI, Google and xAI.",
   "main": "dist/index.js",
   "scripts": {