toolgz 0.2.2 → 0.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -101,6 +101,55 @@ Recompute any figure with
|
|
|
101
101
|
`npx tsx bench/analyze-multi.ts --sweep=2026-07-25T19-19` against the raw per-run
|
|
102
102
|
records in [`bench/results/`](bench/results).
|
|
103
103
|
|
|
104
|
+
### How it scales, measured on real tools
|
|
105
|
+
|
|
106
|
+
The table above is a synthetic 100-tool fixture. This is the real one: **149 tools
|
|
107
|
+
harvested from 14 live MCP servers**, scaled by replicating the corpus, measured with
|
|
108
|
+
Anthropic's `count_tokens` and confirmed against live API calls.
|
|
109
|
+
|
|
110
|
+
| Tools | Uncompressed | Level 3 | Reclaimed |
|
|
111
|
+
|---:|---:|---:|---:|
|
|
112
|
+
| 149 (the real corpus) | 68,536 | **3,022** | **95.6%** |
|
|
113
|
+
| 435 | 199,822 | **8,690** | **95.7%** |
|
|
114
|
+
| 800 | 368,826 | **16,006** | **95.7%** |
|
|
115
|
+
| 1,200 | 552,795 | **23,880** | **95.7%** |
|
|
116
|
+
|
|
117
|
+
**The ratio does not decay with scale.** Both the uncompressed block and the map grow
|
|
118
|
+
linearly, so 95.7% holds from 149 tools to 1,200. Real MCP tools measure ~460 tokens
|
|
119
|
+
each — 17% heavier than the ~393 in the published academic benchmark, so a real
|
|
120
|
+
catalogue hits limits sooner than synthetic ones suggest.
|
|
121
|
+
|
|
122
|
+
Which gives a practical ceiling per context window:
|
|
123
|
+
|
|
124
|
+
| Context window | Tools that fit uncompressed | With level 3 |
|
|
125
|
+
|---|---:|---:|
|
|
126
|
+
| 8K (small/local models) | **17** | ~409 |
|
|
127
|
+
| 32K | 71 | ~1,638 |
|
|
128
|
+
| 200K (typical frontier cap) | **434** | ~10,000 |
|
|
129
|
+
|
|
130
|
+
That 200K row is the one to note: **most deployments cap at 200k, making ~434 real
|
|
131
|
+
tools a hard ceiling.** An independent study ([Sakizli 2026](https://github.com/SKZL-AI/tscg))
|
|
132
|
+
measured the same threshold at ~494 tools using tools 17% lighter than ours — two
|
|
133
|
+
separate measurements agreeing within about 15%.
|
|
134
|
+
|
|
135
|
+
### What we could not test
|
|
136
|
+
|
|
137
|
+
**We have not demonstrated that compression improves accuracy, and we do not claim it.**
|
|
138
|
+
|
|
139
|
+
The published study above finds a *binary enablement* effect: at 8K with 28 tools,
|
|
140
|
+
uncompressed schemas overflow the window and exact-match accuracy collapses to 2.6%,
|
|
141
|
+
while compression restores it (+20.5pp average). At 32K, where both fit, four of five
|
|
142
|
+
models show ≤1pp difference — the effect is **budget-driven, not intrinsic**.
|
|
143
|
+
|
|
144
|
+
We tried to reproduce it and could not, for an honest reason: every provider we test
|
|
145
|
+
against has a window far larger than our corpus needs. Uncompressed requests ran
|
|
146
|
+
successfully at 149, 435 and **800 tools (368,826 tokens)** on `claude-opus-5`, picking
|
|
147
|
+
the correct tool each time. Reaching overflow on a 1M window would take ~2,173 tools.
|
|
148
|
+
|
|
149
|
+
So the enablement regime — where this stops being an optimisation and becomes a
|
|
150
|
+
prerequisite — lives on **small-context models we do not currently test**. If you run
|
|
151
|
+
local models at 8K–32K, that study is more relevant to you than our benchmarks are.
|
|
152
|
+
|
|
104
153
|
### What about cost?
|
|
105
154
|
|
|
106
155
|
**Cost is not the claim, and we deliberately do not lead with it.** Prompt caching
|
|
@@ -859,7 +908,7 @@ and it does not get deleted.
|
|
|
859
908
|
## Development
|
|
860
909
|
|
|
861
910
|
```bash
|
|
862
|
-
npm test #
|
|
911
|
+
npm test # 252 tests, offline, no cost
|
|
863
912
|
npm run build # tsc → dist/ with .d.ts
|
|
864
913
|
|
|
865
914
|
npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants # costs money
|