mcp-context-cost 0.9.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +99 -2
- package/dist/audit/audit.d.ts +46 -0
- package/dist/audit/audit.js +96 -1
- package/dist/audit/run.d.ts +24 -0
- package/dist/audit/run.js +52 -0
- package/dist/cli.js +74 -3
- package/dist/core/capture-index.d.ts +76 -0
- package/dist/core/capture-index.js +92 -0
- package/dist/core/regression.d.ts +181 -0
- package/dist/core/regression.js +199 -0
- package/dist/core/server-diff.d.ts +73 -0
- package/dist/core/server-diff.js +197 -0
- package/dist/core/tool-shape.d.ts +94 -0
- package/dist/core/tool-shape.js +106 -0
- package/dist/sweep/published-stats.d.ts +5 -0
- package/dist/sweep/published-stats.js +9 -0
- package/dist/sweep/regen.js +18 -2
- package/dist/sweep/regressions.d.ts +26 -0
- package/dist/sweep/regressions.js +251 -0
- package/dist/sweep/report.d.ts +2 -1
- package/dist/sweep/report.js +14 -1
- package/dist/sweep/sweep-all.js +6 -1
- package/dist/sweep/tool-shape.d.ts +3 -0
- package/dist/sweep/tool-shape.js +26 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -193,9 +193,43 @@ still match today and silence for the rest. Most installs will show a mix:
|
|
|
193
193
|
memory 9 2,378 4.2% —
|
|
194
194
|
```
|
|
195
195
|
|
|
196
|
+
Add `--suggest` to place each of your tools in the measured set's tool-shape distribution
|
|
197
|
+
([method](docs/METHODOLOGY.md#tool-shape)) and get advice only where the data can point at
|
|
198
|
+
something. Only descriptions draw advice — schemas are functional surface; descriptions are
|
|
199
|
+
prose every request carries — and only descriptions at or above the 90th percentile of the
|
|
200
|
+
1,150 measured tools:
|
|
201
|
+
|
|
202
|
+
```
|
|
203
|
+
suggest — descriptions at or above the 90th percentile of measured tools
|
|
204
|
+
(baseline 2026-09-03: 1,150 tools across 81 measured servers):
|
|
205
|
+
stub · wordy — 345 tokens: description 321 (p92), schema 14
|
|
206
|
+
rewriting the description toward the measured median (27) would recover ≈294 tokens on every request
|
|
207
|
+
1 of 2 tools sit inside the distribution — no advice where nothing is measurably unusual.
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
A config where nothing is out of distribution is told that in those words, and a baseline
|
|
211
|
+
that cannot be fetched is a named problem, never a silently skipped check.
|
|
212
|
+
|
|
213
|
+
Add `--changed` to ask the other question — *did the servers I already have get heavier?*
|
|
214
|
+
Each installed server is identified against the published capture history by its canonical
|
|
215
|
+
hash, never by its name, because the name in your config is a label you chose and the bytes
|
|
216
|
+
are not:
|
|
217
|
+
|
|
218
|
+
```
|
|
219
|
+
changed — published versions of your servers that have moved since
|
|
220
|
+
(index 2026-09-04, 2 published captures; matched by canonical hash, never by name):
|
|
221
|
+
notes (published as obsidian) — you have the capture published 2026-08-19 at 1,132 tokens;
|
|
222
|
+
the current one is 2,062 (+930, 2026-08-26)
|
|
223
|
+
updating all 1 would add 930 tokens to every request in this client.
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
A server whose bytes match no published capture — a version never measured here, a fork, a
|
|
227
|
+
pin — is reported as unidentified with nothing claimed about it.
|
|
228
|
+
Method: [capture index](docs/METHODOLOGY.md#capture-index).
|
|
229
|
+
|
|
196
230
|
Flags: `--json` (full report on stdout, progress on stderr), `--budget N`,
|
|
197
231
|
`--baseline <report.json>`, `--max-increase N`, `--context N` (default 200,000),
|
|
198
|
-
`--timeout ms`, `--concurrency N`, `--docker`, `--claude`.
|
|
232
|
+
`--timeout ms`, `--concurrency N`, `--docker`, `--claude`, `--suggest`, `--changed`.
|
|
199
233
|
|
|
200
234
|
## Where the numbers come from
|
|
201
235
|
|
|
@@ -221,6 +255,14 @@ sample of that range; the full range is in
|
|
|
221
255
|
Each measured server also has a [detail page](https://athakur3.github.io/mcp-context-cost/servers/)
|
|
222
256
|
showing which tools its tokens are in.)*
|
|
223
257
|
|
|
258
|
+
Because the set is re-measured on a rotation and most entries launch unpinned, the same data
|
|
259
|
+
answers a question no client asks: **what did this server cost last month?**
|
|
260
|
+
[results/regressions.md](results/regressions.md) reports each server's most recent movement —
|
|
261
|
+
dated to when it happened, separated into *shipped more tools* versus *same tools, rewritten*,
|
|
262
|
+
and compared only within one isolation. The ecosystem ratchets upward: of the servers whose
|
|
263
|
+
cost has moved at all, 9 moved up against 1 that moved down. Method:
|
|
264
|
+
[cost movement](docs/METHODOLOGY.md#cost-movement).
|
|
265
|
+
|
|
224
266
|
If you publish a server, the same measurement is available as a badge, so your users can see
|
|
225
267
|
the cost before they install rather than after:
|
|
226
268
|
|
|
@@ -304,6 +346,58 @@ Then in your README:
|
|
|
304
346
|
[](<link target>)
|
|
305
347
|
```
|
|
306
348
|
|
|
349
|
+
### Defend the number, don't just display it
|
|
350
|
+
|
|
351
|
+
A badge says what your server costs today; it does nothing about the release
|
|
352
|
+
that adds 1,200 tokens to every user's context next month. Across the servers
|
|
353
|
+
measured here, that release is the norm rather than the exception — the
|
|
354
|
+
[movement report](results/regressions.md) has nine servers ratcheting upward
|
|
355
|
+
against one that got cheaper, and none of those maintainers had a check that
|
|
356
|
+
would have said so first. `measure` takes the same gate flags `audit` does, so
|
|
357
|
+
your own CI can be that check:
|
|
358
|
+
|
|
359
|
+
```bash
|
|
360
|
+
# on your default branch, once — commit the result
|
|
361
|
+
npx -y mcp-context-cost measure --name my-server --command "node dist/index.js"
|
|
362
|
+
cp results/my-server/measurement.json .context-cost/baseline.json
|
|
363
|
+
|
|
364
|
+
# on every pull request
|
|
365
|
+
npx -y mcp-context-cost measure --name my-server --command "node dist/index.js" \
|
|
366
|
+
--baseline .context-cost/baseline.json --max-increase 500
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
```
|
|
370
|
+
diff vs baseline .context-cost/baseline.json
|
|
371
|
+
my-server: 61 → 182 +121 tokens (2 → 3 tools)
|
|
372
|
+
added: bulk_export (43)
|
|
373
|
+
grew: search 30 → 108 (+78)
|
|
374
|
+
|
|
375
|
+
INCREASE FAIL: +121 tokens, over the 100 allowed — this change adds that to every request of every install.
|
|
376
|
+
```
|
|
377
|
+
|
|
378
|
+
Both sides are single measurements carrying per-tool counts, so an established
|
|
379
|
+
change is attributed exactly: which tools arrived, which grew, and by how much.
|
|
380
|
+
And `--max-increase` fails on more than the number — a server that stops
|
|
381
|
+
starting on the branch makes the total go *down*, and reporting that as an
|
|
382
|
+
improvement is the one mistake a gate like this must not make, so a change that
|
|
383
|
+
could not be established fails too.
|
|
384
|
+
|
|
385
|
+
As a GitHub Action, that whole workflow is five lines
|
|
386
|
+
([full example](examples/server-author-ci.yml)):
|
|
387
|
+
|
|
388
|
+
```yaml
|
|
389
|
+
- uses: athakur3/mcp-context-cost@v1
|
|
390
|
+
with:
|
|
391
|
+
name: my-server
|
|
392
|
+
command: node dist/index.js
|
|
393
|
+
baseline: .context-cost/baseline.json
|
|
394
|
+
max-increase: 500
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
It exposes `tokens`, `tools`, `status`, `measurement` and `badge` as outputs —
|
|
398
|
+
available whether the gate passed or not — so a later step can comment the
|
|
399
|
+
number on the pull request or publish the badge.
|
|
400
|
+
|
|
307
401
|
Point the link at the measurement behind the number — for servers in this sweep that is
|
|
308
402
|
`https://athakur3.github.io/mcp-context-cost/servers/<name>.html`; otherwise the
|
|
309
403
|
methodology page. A badge nobody can audit is decoration.
|
|
@@ -329,7 +423,10 @@ color bands are frozen against the observed distribution of the first full sweep
|
|
|
329
423
|
|
|
330
424
|
## Status
|
|
331
425
|
|
|
332
|
-
Active. Every row carries the date of its own most recent measurement
|
|
426
|
+
Active. Every row carries the date of its own most recent measurement, and what the data
|
|
427
|
+
says as a whole is written up, dated, in
|
|
428
|
+
[The State of MCP Context Cost](https://athakur3.github.io/mcp-context-cost/state-of-mcp-context-cost)
|
|
429
|
+
(September 2026). Two
|
|
333
430
|
weekly jobs re-measure the set — the `memory` reference server on Mondays, and a rotating
|
|
334
431
|
sixth of the list on Wednesdays, so every row comes round within six weeks. Read each row's
|
|
335
432
|
date as the date it means, and don't take the cadence on trust — the build history is
|
package/dist/audit/audit.d.ts
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
1
|
import { type DivergenceRun } from '../core/divergence.js';
|
|
2
|
+
import { type ToolShapeBaseline, type ToolSuggestion } from '../core/tool-shape.js';
|
|
3
|
+
import { type CaptureIndex, type CaptureVerdict } from '../core/capture-index.js';
|
|
2
4
|
import type { Measurement, MeasurementStatus, ToolMeasurement } from '../core/types.js';
|
|
3
5
|
import type { ConfiguredServer, LoadedConfig } from './config.js';
|
|
4
6
|
import { type DeferralVerdict, type ToolSearchEnv, type ToolSearchSource } from './deferral.js';
|
|
@@ -43,6 +45,31 @@ export interface TrimAdvice {
|
|
|
43
45
|
recoverableTokens: number;
|
|
44
46
|
recoverableShare: number;
|
|
45
47
|
}
|
|
48
|
+
/**
|
|
49
|
+
* `--suggest`: this config's tools placed in the measured set's tool-shape
|
|
50
|
+
* distribution. Only descriptions draw advice (schemas are functional surface;
|
|
51
|
+
* descriptions are prose every request carries), and only descriptions the
|
|
52
|
+
* baseline puts at or above the threshold percentile — a config where nothing
|
|
53
|
+
* is measurably unusual gets that said in those words, not advice invented to
|
|
54
|
+
* have some.
|
|
55
|
+
*/
|
|
56
|
+
export interface ConfigSuggestions {
|
|
57
|
+
/** Heaviest-recovery first. */
|
|
58
|
+
outOfDistribution: ToolSuggestion[];
|
|
59
|
+
checkedTools: number;
|
|
60
|
+
}
|
|
61
|
+
/**
|
|
62
|
+
* `--changed`: one locally measured server placed against the published capture
|
|
63
|
+
* history, joined by canonical hash rather than by name. The local label is
|
|
64
|
+
* carried alongside the published server the bytes identify, because they need
|
|
65
|
+
* not agree — a config may call the official GitHub server anything at all, and
|
|
66
|
+
* a name that disagrees with the bytes is information rather than noise.
|
|
67
|
+
*/
|
|
68
|
+
export interface ServerCaptureVerdict {
|
|
69
|
+
/** The name this config gave the server. */
|
|
70
|
+
name: string;
|
|
71
|
+
verdict: CaptureVerdict;
|
|
72
|
+
}
|
|
46
73
|
export interface AuditConfigResult {
|
|
47
74
|
client: string;
|
|
48
75
|
source: string;
|
|
@@ -54,6 +81,10 @@ export interface AuditConfigResult {
|
|
|
54
81
|
skipped: AuditServerResult[];
|
|
55
82
|
heaviestTools: HeaviestTool[];
|
|
56
83
|
trimAdvice: TrimAdvice | null;
|
|
84
|
+
/** Present only when `--suggest` ran with a usable baseline. */
|
|
85
|
+
suggestions?: ConfigSuggestions;
|
|
86
|
+
/** Present only when `--changed` ran with a usable capture index. */
|
|
87
|
+
captureVerdicts?: ServerCaptureVerdict[];
|
|
57
88
|
/**
|
|
58
89
|
* Whether this client loads the total up front or defers it, and — when the
|
|
59
90
|
* client decides that by a threshold — which side of it this stack is on.
|
|
@@ -128,6 +159,17 @@ export interface AuditReport {
|
|
|
128
159
|
model: string;
|
|
129
160
|
measuredAt: string;
|
|
130
161
|
};
|
|
162
|
+
/** Which published tool-shape baseline `--suggest` read its percentiles from. */
|
|
163
|
+
toolShape?: {
|
|
164
|
+
generatedAt: string;
|
|
165
|
+
toolCount: number;
|
|
166
|
+
serverCount: number;
|
|
167
|
+
};
|
|
168
|
+
/** Which published capture index `--changed` joined against. */
|
|
169
|
+
captureIndex?: {
|
|
170
|
+
generatedAt: string;
|
|
171
|
+
captureCount: number;
|
|
172
|
+
};
|
|
131
173
|
/** Present only when a baseline report was supplied (`--baseline`). */
|
|
132
174
|
diff?: AuditDiff;
|
|
133
175
|
/** Present only when `--max-increase` was supplied alongside a baseline. */
|
|
@@ -160,6 +202,10 @@ export declare function buildReport(configs: LoadedConfig[], measured: Map<strin
|
|
|
160
202
|
generatedAt?: string;
|
|
161
203
|
/** Published `tools-delta/v1` run to join against (`--claude`); omit to skip the join. */
|
|
162
204
|
divergence?: DivergenceRun | null;
|
|
205
|
+
/** Published `tool-shape/v1` baseline (`--suggest`); omit to skip suggestions. */
|
|
206
|
+
toolShape?: ToolShapeBaseline | null;
|
|
207
|
+
/** Published `capture-index/v1` (`--changed`); omit to skip the version join. */
|
|
208
|
+
captureIndex?: CaptureIndex | null;
|
|
163
209
|
/**
|
|
164
210
|
* The audited machine's SHELL tool-search variables. Passed in rather than
|
|
165
211
|
* read here so this stays pure and a report is reproducible from its
|
package/dist/audit/audit.js
CHANGED
|
@@ -14,10 +14,22 @@
|
|
|
14
14
|
*/
|
|
15
15
|
import { METHODOLOGY_VERSION } from '../core/canonical.js';
|
|
16
16
|
import { isCurrent } from '../core/divergence.js';
|
|
17
|
+
import { SUGGEST_DESCRIPTION_PERCENTILE, suggestFor, } from '../core/tool-shape.js';
|
|
18
|
+
import { identify } from '../core/capture-index.js';
|
|
17
19
|
import { evaluateDeferral, PUBLISHED_WIRE_TO_CLIENT_RATIO, SHELL_SOURCE, } from './deferral.js';
|
|
18
20
|
import { formatDiff, formatGate } from './diff.js';
|
|
19
21
|
export const DEFAULT_CONTEXT_WINDOW = 200_000;
|
|
20
22
|
const TRIM_TOOL_COUNT = 3;
|
|
23
|
+
function buildSuggestions(pool, baseline) {
|
|
24
|
+
const outOfDistribution = [];
|
|
25
|
+
for (const { server, t } of pool) {
|
|
26
|
+
const s = suggestFor(server, t, baseline);
|
|
27
|
+
if (s)
|
|
28
|
+
outOfDistribution.push(s);
|
|
29
|
+
}
|
|
30
|
+
outOfDistribution.sort((a, b) => b.approxRecoverableTokens - a.approxRecoverableTokens);
|
|
31
|
+
return { outOfDistribution, checkedTools: pool.length };
|
|
32
|
+
}
|
|
21
33
|
function buildTrimAdvice(sortedTools, totalTokens) {
|
|
22
34
|
if (totalTokens <= 0 || sortedTools.length < 2)
|
|
23
35
|
return null;
|
|
@@ -189,6 +201,7 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
189
201
|
const ok = [];
|
|
190
202
|
const skipped = [];
|
|
191
203
|
const tools = [];
|
|
204
|
+
const shapePool = [];
|
|
192
205
|
// Counted only for servers that put a number into the total: a twin that
|
|
193
206
|
// failed to launch is already a floor, and adds nothing to a sum.
|
|
194
207
|
let sharedHere = 0;
|
|
@@ -240,8 +253,11 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
240
253
|
claudeTokens: opts.divergence ? (isCurrent(divRow, m.canonicalSha256 ?? null) ? divRow.claudeDelta : null) : undefined,
|
|
241
254
|
notes: m.status === 'dynamic' ? m.notes : undefined,
|
|
242
255
|
});
|
|
243
|
-
for (const t of m.tools)
|
|
256
|
+
for (const t of m.tools) {
|
|
244
257
|
tools.push({ server: s.name, tool: t.name, tokens: t.tokens });
|
|
258
|
+
if (opts.toolShape)
|
|
259
|
+
shapePool.push({ server: s.name, t });
|
|
260
|
+
}
|
|
245
261
|
}
|
|
246
262
|
const totalTokens = ok.reduce((a, s) => a + (s.tokens ?? 0), 0);
|
|
247
263
|
const toolCount = ok.reduce((a, s) => a + (s.toolCount ?? 0), 0);
|
|
@@ -264,6 +280,10 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
264
280
|
skipped,
|
|
265
281
|
heaviestTools: tools.slice(0, 5),
|
|
266
282
|
trimAdvice: buildTrimAdvice(tools, totalTokens),
|
|
283
|
+
suggestions: opts.toolShape ? buildSuggestions(shapePool, opts.toolShape) : undefined,
|
|
284
|
+
captureVerdicts: opts.captureIndex
|
|
285
|
+
? ok.map((s) => ({ name: s.name, verdict: identify(s.canonicalSha256, opts.captureIndex) }))
|
|
286
|
+
: undefined,
|
|
267
287
|
};
|
|
268
288
|
built.push(result);
|
|
269
289
|
shared.set(result, sharedHere);
|
|
@@ -282,6 +302,19 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
282
302
|
if (opts.divergence) {
|
|
283
303
|
report.claudeDivergence = { model: opts.divergence.model, measuredAt: opts.divergence.measuredAt };
|
|
284
304
|
}
|
|
305
|
+
if (opts.toolShape) {
|
|
306
|
+
report.toolShape = {
|
|
307
|
+
generatedAt: opts.toolShape.generatedAt,
|
|
308
|
+
toolCount: opts.toolShape.toolCount,
|
|
309
|
+
serverCount: opts.toolShape.serverCount,
|
|
310
|
+
};
|
|
311
|
+
}
|
|
312
|
+
if (opts.captureIndex) {
|
|
313
|
+
report.captureIndex = {
|
|
314
|
+
generatedAt: opts.captureIndex.generatedAt,
|
|
315
|
+
captureCount: Object.keys(opts.captureIndex.captures).length,
|
|
316
|
+
};
|
|
317
|
+
}
|
|
285
318
|
if (typeof opts.budget === 'number') {
|
|
286
319
|
// The worst config is the gate: passing because your *lightest* client fits
|
|
287
320
|
// would be a green check on a session you don't run.
|
|
@@ -617,6 +650,68 @@ export function formatReport(report) {
|
|
|
617
650
|
`(${names}) would recover ${n(cfg.trimAdvice.recoverableTokens)} tokens ` +
|
|
618
651
|
`(${pct(cfg.trimAdvice.recoverableShare)} of this config) — if your client supports per-tool filtering.`);
|
|
619
652
|
}
|
|
653
|
+
if (cfg.captureVerdicts) {
|
|
654
|
+
const behind = cfg.captureVerdicts.filter((v) => v.verdict.kind === 'behind');
|
|
655
|
+
const current = cfg.captureVerdicts.filter((v) => v.verdict.kind === 'current');
|
|
656
|
+
const unknown = cfg.captureVerdicts.length - behind.length - current.length;
|
|
657
|
+
const idx = report.captureIndex
|
|
658
|
+
? `index ${report.captureIndex.generatedAt}, ${n(report.captureIndex.captureCount)} published captures`
|
|
659
|
+
: 'the published capture index';
|
|
660
|
+
lines.push('');
|
|
661
|
+
if (behind.length === 0) {
|
|
662
|
+
lines.push(` changed: no server here is running a published capture that has since moved ` +
|
|
663
|
+
`(${idx}; matched by canonical hash, never by name).`);
|
|
664
|
+
}
|
|
665
|
+
else {
|
|
666
|
+
const total = behind.reduce((a, v) => a + v.verdict.deltaTokens, 0);
|
|
667
|
+
lines.push(` changed — published versions of your servers that have moved since ` +
|
|
668
|
+
`(${idx}; matched by canonical hash, never by name):`);
|
|
669
|
+
for (const { name, verdict: v } of behind) {
|
|
670
|
+
// The local label and the published server are printed together: the
|
|
671
|
+
// bytes decide which server this is, and a name that disagrees is a
|
|
672
|
+
// fact worth seeing rather than one to smooth over.
|
|
673
|
+
const alias = name === v.server ? name : `${name} (published as ${v.server})`;
|
|
674
|
+
lines.push(` ${alias} — you have the capture published ${v.yourDate} at ${n(v.yourTokens)} tokens; ` +
|
|
675
|
+
`the current one is ${n(v.currentTokens)} (${v.deltaTokens >= 0 ? '+' : '−'}${n(Math.abs(v.deltaTokens))}, ${v.currentDate})`);
|
|
676
|
+
}
|
|
677
|
+
lines.push(` updating all ${behind.length} would ${total >= 0 ? 'add' : 'remove'} ${n(Math.abs(total))} tokens ` +
|
|
678
|
+
`${total >= 0 ? 'to' : 'from'} every request in this client.`);
|
|
679
|
+
}
|
|
680
|
+
if (unknown > 0) {
|
|
681
|
+
lines.push(` ${unknown} server${unknown === 1 ? '' : 's'} could not be identified: the installed bytes match no ` +
|
|
682
|
+
`published capture — a version never measured here, or one published before the index began. ` +
|
|
683
|
+
`Nothing is claimed about ${unknown === 1 ? 'it' : 'them'}.`);
|
|
684
|
+
}
|
|
685
|
+
}
|
|
686
|
+
if (cfg.suggestions) {
|
|
687
|
+
const sg = cfg.suggestions;
|
|
688
|
+
const base = report.toolShape
|
|
689
|
+
? `baseline ${report.toolShape.generatedAt}: ${n(report.toolShape.toolCount)} tools across ` +
|
|
690
|
+
`${report.toolShape.serverCount} measured servers`
|
|
691
|
+
: 'published baseline';
|
|
692
|
+
lines.push('');
|
|
693
|
+
if (sg.outOfDistribution.length === 0) {
|
|
694
|
+
lines.push(` suggest: every description in this config sits inside the measured distribution — ` +
|
|
695
|
+
`nothing the data can point at (${sg.checkedTools} tools against ${base}).`);
|
|
696
|
+
}
|
|
697
|
+
else {
|
|
698
|
+
lines.push(` suggest — descriptions at or above the ${SUGGEST_DESCRIPTION_PERCENTILE}th percentile of ` +
|
|
699
|
+
`measured tools (${base}):`);
|
|
700
|
+
const shown = sg.outOfDistribution.slice(0, 8);
|
|
701
|
+
for (const s of shown) {
|
|
702
|
+
lines.push(` ${s.server} · ${s.tool} — ${n(s.tokens)} tokens: description ${n(s.descriptionTokens)} ` +
|
|
703
|
+
`(p${s.descriptionPercentile}), schema ${n(s.inputSchemaTokens)}`);
|
|
704
|
+
lines.push(` rewriting the description toward the measured median (${n(s.medianDescriptionTokens)}) ` +
|
|
705
|
+
`would recover ≈${n(s.approxRecoverableTokens)} tokens on every request`);
|
|
706
|
+
}
|
|
707
|
+
if (sg.outOfDistribution.length > shown.length) {
|
|
708
|
+
lines.push(` …and ${sg.outOfDistribution.length - shown.length} more above the threshold.`);
|
|
709
|
+
}
|
|
710
|
+
const within = sg.checkedTools - sg.outOfDistribution.length;
|
|
711
|
+
lines.push(` ${within} of ${sg.checkedTools} tools sit inside the distribution — ` +
|
|
712
|
+
`no advice where nothing is measurably unusual.`);
|
|
713
|
+
}
|
|
714
|
+
}
|
|
620
715
|
if (cfg.skipped.length) {
|
|
621
716
|
lines.push('');
|
|
622
717
|
lines.push(' not measured');
|
package/dist/audit/run.d.ts
CHANGED
|
@@ -1,10 +1,16 @@
|
|
|
1
1
|
import type { Measurement } from '../core/types.js';
|
|
2
2
|
import { type DivergenceRun } from '../core/divergence.js';
|
|
3
|
+
import { type ToolShapeBaseline } from '../core/tool-shape.js';
|
|
4
|
+
import { type CaptureIndex } from '../core/capture-index.js';
|
|
3
5
|
import { type AuditReport } from './audit.js';
|
|
4
6
|
import { type ToolSearchEnv, type ToolSearchSource } from './deferral.js';
|
|
5
7
|
import { type LoadedConfig } from './config.js';
|
|
6
8
|
/** Where the published `tools-delta/v1` run lives when `--claude` doesn't override it. */
|
|
7
9
|
export declare const DEFAULT_DIVERGENCE_URL = "https://raw.githubusercontent.com/athakur3/mcp-context-cost/main/results/divergence.json";
|
|
10
|
+
/** Where the published `tool-shape/v1` baseline lives when `--suggest` doesn't override it. */
|
|
11
|
+
export declare const DEFAULT_TOOL_SHAPE_URL = "https://raw.githubusercontent.com/athakur3/mcp-context-cost/main/results/tool-shape.json";
|
|
12
|
+
/** Where the published `capture-index/v1` lives when `--changed` doesn't override it. */
|
|
13
|
+
export declare const DEFAULT_CAPTURE_INDEX_URL = "https://raw.githubusercontent.com/athakur3/mcp-context-cost/main/results/capture-index.json";
|
|
8
14
|
export interface AuditOptions {
|
|
9
15
|
/** Explicit config path(s); when empty, every known client location is tried. */
|
|
10
16
|
configPaths?: string[];
|
|
@@ -19,6 +25,14 @@ export interface AuditOptions {
|
|
|
19
25
|
claude?: boolean;
|
|
20
26
|
/** Override the divergence.json source — mainly for tests and self-hosted mirrors. */
|
|
21
27
|
divergenceUrl?: string;
|
|
28
|
+
/** Place this config's tools in the published tool-shape distribution and advise where the data can. */
|
|
29
|
+
suggest?: boolean;
|
|
30
|
+
/** Override the tool-shape.json source — mainly for tests and self-hosted mirrors. */
|
|
31
|
+
toolShapeUrl?: string;
|
|
32
|
+
/** Identify each server against the published capture history, by hash, and report what has moved. */
|
|
33
|
+
changed?: boolean;
|
|
34
|
+
/** Override the capture-index.json source — mainly for tests and self-hosted mirrors. */
|
|
35
|
+
captureIndexUrl?: string;
|
|
22
36
|
/**
|
|
23
37
|
* The tool-search variables as this process's SHELL has them. Defaults to
|
|
24
38
|
* this process's environment. Overridable so a test can state a machine
|
|
@@ -33,6 +47,16 @@ export interface AuditOptions {
|
|
|
33
47
|
settings?: ToolSearchSource[];
|
|
34
48
|
onProgress?: (name: string, done: number, total: number) => void;
|
|
35
49
|
}
|
|
50
|
+
/** Fetch and parse the published capture index. Never throws: a failure is a report problem, not a crash. */
|
|
51
|
+
export declare function fetchCaptureIndex(url: string): Promise<{
|
|
52
|
+
index: CaptureIndex | null;
|
|
53
|
+
problem?: string;
|
|
54
|
+
}>;
|
|
55
|
+
/** Fetch and parse the published tool-shape baseline. Never throws: a failure is a report problem, not a crash. */
|
|
56
|
+
export declare function fetchToolShape(url: string): Promise<{
|
|
57
|
+
baseline: ToolShapeBaseline | null;
|
|
58
|
+
problem?: string;
|
|
59
|
+
}>;
|
|
36
60
|
/** Fetch and parse the published divergence run. Never throws: a failure is a report problem, not a crash. */
|
|
37
61
|
export declare function fetchDivergence(url: string): Promise<{
|
|
38
62
|
run: DivergenceRun | null;
|
package/dist/audit/run.js
CHANGED
|
@@ -7,11 +7,43 @@
|
|
|
7
7
|
import { homedir } from 'node:os';
|
|
8
8
|
import { measureServer } from '../sweep/run.js';
|
|
9
9
|
import { parseDivergence } from '../core/divergence.js';
|
|
10
|
+
import { parseToolShapeBaseline } from '../core/tool-shape.js';
|
|
11
|
+
import { parseCaptureIndex } from '../core/capture-index.js';
|
|
10
12
|
import { buildReport, serverKey } from './audit.js';
|
|
11
13
|
import { toolSearchEnv } from './deferral.js';
|
|
12
14
|
import { configCandidates, loadConfigs, loadSettingsSources, settingsCandidates, } from './config.js';
|
|
13
15
|
/** Where the published `tools-delta/v1` run lives when `--claude` doesn't override it. */
|
|
14
16
|
export const DEFAULT_DIVERGENCE_URL = 'https://raw.githubusercontent.com/athakur3/mcp-context-cost/main/results/divergence.json';
|
|
17
|
+
/** Where the published `tool-shape/v1` baseline lives when `--suggest` doesn't override it. */
|
|
18
|
+
export const DEFAULT_TOOL_SHAPE_URL = 'https://raw.githubusercontent.com/athakur3/mcp-context-cost/main/results/tool-shape.json';
|
|
19
|
+
/** Where the published `capture-index/v1` lives when `--changed` doesn't override it. */
|
|
20
|
+
export const DEFAULT_CAPTURE_INDEX_URL = 'https://raw.githubusercontent.com/athakur3/mcp-context-cost/main/results/capture-index.json';
|
|
21
|
+
/** Fetch and parse the published capture index. Never throws: a failure is a report problem, not a crash. */
|
|
22
|
+
export async function fetchCaptureIndex(url) {
|
|
23
|
+
try {
|
|
24
|
+
const res = await fetch(url, { signal: AbortSignal.timeout(15_000) });
|
|
25
|
+
if (!res.ok)
|
|
26
|
+
return { index: null, problem: `capture index: HTTP ${res.status} fetching ${url}` };
|
|
27
|
+
const index = parseCaptureIndex(await res.text());
|
|
28
|
+
return index ? { index } : { index: null, problem: `capture index: malformed data at ${url}` };
|
|
29
|
+
}
|
|
30
|
+
catch (e) {
|
|
31
|
+
return { index: null, problem: `capture index: failed to fetch ${url}: ${e.message}` };
|
|
32
|
+
}
|
|
33
|
+
}
|
|
34
|
+
/** Fetch and parse the published tool-shape baseline. Never throws: a failure is a report problem, not a crash. */
|
|
35
|
+
export async function fetchToolShape(url) {
|
|
36
|
+
try {
|
|
37
|
+
const res = await fetch(url, { signal: AbortSignal.timeout(15_000) });
|
|
38
|
+
if (!res.ok)
|
|
39
|
+
return { baseline: null, problem: `tool shape: HTTP ${res.status} fetching ${url}` };
|
|
40
|
+
const baseline = parseToolShapeBaseline(await res.text());
|
|
41
|
+
return baseline ? { baseline } : { baseline: null, problem: `tool shape: malformed data at ${url}` };
|
|
42
|
+
}
|
|
43
|
+
catch (e) {
|
|
44
|
+
return { baseline: null, problem: `tool shape: failed to fetch ${url}: ${e.message}` };
|
|
45
|
+
}
|
|
46
|
+
}
|
|
15
47
|
/** Fetch and parse the published divergence run. Never throws: a failure is a report problem, not a crash. */
|
|
16
48
|
export async function fetchDivergence(url) {
|
|
17
49
|
try {
|
|
@@ -89,14 +121,34 @@ export async function runAudit(opts = {}) {
|
|
|
89
121
|
divergence = fetched.run;
|
|
90
122
|
divergenceProblem = fetched.problem;
|
|
91
123
|
}
|
|
124
|
+
let toolShape = null;
|
|
125
|
+
let toolShapeProblem;
|
|
126
|
+
if (opts.suggest) {
|
|
127
|
+
const fetched = await fetchToolShape(opts.toolShapeUrl ?? DEFAULT_TOOL_SHAPE_URL);
|
|
128
|
+
toolShape = fetched.baseline;
|
|
129
|
+
toolShapeProblem = fetched.problem;
|
|
130
|
+
}
|
|
131
|
+
let captureIndex = null;
|
|
132
|
+
let captureIndexProblem;
|
|
133
|
+
if (opts.changed) {
|
|
134
|
+
const fetched = await fetchCaptureIndex(opts.captureIndexUrl ?? DEFAULT_CAPTURE_INDEX_URL);
|
|
135
|
+
captureIndex = fetched.index;
|
|
136
|
+
captureIndexProblem = fetched.problem;
|
|
137
|
+
}
|
|
92
138
|
const report = buildReport(configs, measured, {
|
|
93
139
|
contextWindow: opts.contextWindow,
|
|
94
140
|
budget: opts.budget,
|
|
95
141
|
divergence,
|
|
142
|
+
toolShape,
|
|
143
|
+
captureIndex,
|
|
96
144
|
env: opts.env ?? toolSearchEnv(process.env),
|
|
97
145
|
settings: opts.settings ?? discoverSettings(opts),
|
|
98
146
|
});
|
|
99
147
|
if (divergenceProblem)
|
|
100
148
|
report.problems.push(divergenceProblem);
|
|
149
|
+
if (toolShapeProblem)
|
|
150
|
+
report.problems.push(toolShapeProblem);
|
|
151
|
+
if (captureIndexProblem)
|
|
152
|
+
report.problems.push(captureIndexProblem);
|
|
101
153
|
return report;
|
|
102
154
|
}
|
package/dist/cli.js
CHANGED
|
@@ -99,8 +99,19 @@ function rejectUnknownFlags(cmd, argv, spec) {
|
|
|
99
99
|
const [, , cmd, ...rest] = process.argv;
|
|
100
100
|
if (cmd === 'audit') {
|
|
101
101
|
rejectUnknownFlags('audit', rest, {
|
|
102
|
-
value: [
|
|
103
|
-
|
|
102
|
+
value: [
|
|
103
|
+
'config',
|
|
104
|
+
'budget',
|
|
105
|
+
'baseline',
|
|
106
|
+
'max-increase',
|
|
107
|
+
'context',
|
|
108
|
+
'timeout',
|
|
109
|
+
'concurrency',
|
|
110
|
+
'divergence-url',
|
|
111
|
+
'tool-shape-url',
|
|
112
|
+
'capture-index-url',
|
|
113
|
+
],
|
|
114
|
+
boolean: ['json', 'docker', 'claude', 'suggest', 'changed'],
|
|
104
115
|
});
|
|
105
116
|
const argOf = (name) => {
|
|
106
117
|
const i = rest.indexOf(`--${name}`);
|
|
@@ -171,6 +182,10 @@ if (cmd === 'audit') {
|
|
|
171
182
|
docker: rest.includes('--docker'),
|
|
172
183
|
claude: rest.includes('--claude'),
|
|
173
184
|
divergenceUrl: argOf('divergence-url'),
|
|
185
|
+
suggest: rest.includes('--suggest'),
|
|
186
|
+
toolShapeUrl: argOf('tool-shape-url'),
|
|
187
|
+
changed: rest.includes('--changed'),
|
|
188
|
+
captureIndexUrl: argOf('capture-index-url'),
|
|
174
189
|
// Progress goes to stderr so `--json` stdout stays a single parseable object.
|
|
175
190
|
onProgress: json ? undefined : (name, done, total) => process.stderr.write(` [${done}/${total}] ${name}\n`),
|
|
176
191
|
});
|
|
@@ -280,7 +295,7 @@ else if (cmd === 'verify') {
|
|
|
280
295
|
}
|
|
281
296
|
else if (cmd === 'measure') {
|
|
282
297
|
rejectUnknownFlags('measure', rest, {
|
|
283
|
-
value: ['name', 'command', 'remote', 'timeout', 'docker-image'],
|
|
298
|
+
value: ['name', 'command', 'remote', 'timeout', 'docker-image', 'baseline', 'max-increase', 'budget'],
|
|
284
299
|
boolean: ['docker'],
|
|
285
300
|
});
|
|
286
301
|
const argOf = (name) => {
|
|
@@ -303,6 +318,45 @@ else if (cmd === 'measure') {
|
|
|
303
318
|
console.error('usage: mcp-context-cost measure --name <slug> --command "npx -y <server>" [--timeout ms] [--docker]');
|
|
304
319
|
process.exit(2);
|
|
305
320
|
}
|
|
321
|
+
const { diffServer, evaluateServerGate, formatServerDiff, parseBaselineMeasurement, } = await import('./core/server-diff.js');
|
|
322
|
+
// Gate limits are read before anything is measured: an unusable number is a
|
|
323
|
+
// usage error, and finding that out after a two-minute container launch is
|
|
324
|
+
// the wrong time to find it out.
|
|
325
|
+
const numericFlag = (flag) => {
|
|
326
|
+
const raw = argOf(flag);
|
|
327
|
+
if (raw === undefined)
|
|
328
|
+
return undefined;
|
|
329
|
+
const v = Number(raw);
|
|
330
|
+
if (!Number.isFinite(v) || v < 0) {
|
|
331
|
+
console.error(`--${flag} must be a non-negative number, got '${raw}'`);
|
|
332
|
+
process.exit(2);
|
|
333
|
+
}
|
|
334
|
+
return v;
|
|
335
|
+
};
|
|
336
|
+
const maxIncrease = numericFlag('max-increase');
|
|
337
|
+
const budget = numericFlag('budget');
|
|
338
|
+
const baselinePath = argOf('baseline');
|
|
339
|
+
if (maxIncrease !== undefined && !baselinePath) {
|
|
340
|
+
console.error('--max-increase needs --baseline <measurement.json> to compare against');
|
|
341
|
+
process.exit(2);
|
|
342
|
+
}
|
|
343
|
+
let baseline = null;
|
|
344
|
+
if (baselinePath) {
|
|
345
|
+
let raw;
|
|
346
|
+
try {
|
|
347
|
+
raw = readFileSync(baselinePath, 'utf8');
|
|
348
|
+
}
|
|
349
|
+
catch (e) {
|
|
350
|
+
console.error(`cannot read baseline ${baselinePath}: ${e.message}`);
|
|
351
|
+
process.exit(2);
|
|
352
|
+
}
|
|
353
|
+
const parsed = parseBaselineMeasurement(raw);
|
|
354
|
+
if (!parsed.measurement) {
|
|
355
|
+
console.error(`${baselinePath}: ${parsed.problem}`);
|
|
356
|
+
process.exit(2);
|
|
357
|
+
}
|
|
358
|
+
baseline = parsed.measurement;
|
|
359
|
+
}
|
|
306
360
|
const { measureServer } = await import('./sweep/run.js');
|
|
307
361
|
const m = await measureServer(name, remoteUrl ? `npx -y mcp-remote ${remoteUrl}` : command, {
|
|
308
362
|
timeoutMs: Number(argOf('timeout') ?? 60_000),
|
|
@@ -314,6 +368,20 @@ else if (cmd === 'measure') {
|
|
|
314
368
|
console.log(ok
|
|
315
369
|
? `${name}: ${m.totalTokens} tokens across ${m.toolCount} tools (${m.status}) — results/${name}/measurement.json, badges/${name}.json`
|
|
316
370
|
: `${name}: ${m.status} — ${m.notes ?? ''}`);
|
|
371
|
+
if (baseline || budget !== undefined || maxIncrease !== undefined) {
|
|
372
|
+
const diff = diffServer(name, baseline, m);
|
|
373
|
+
if (baseline) {
|
|
374
|
+
console.log('');
|
|
375
|
+
console.log(`diff vs baseline ${baselinePath}`);
|
|
376
|
+
console.log(formatServerDiff(diff));
|
|
377
|
+
}
|
|
378
|
+
const gate = evaluateServerGate(diff, { budget, maxIncrease });
|
|
379
|
+
if (!gate.pass) {
|
|
380
|
+
console.log('');
|
|
381
|
+
console.error(gate.failure);
|
|
382
|
+
process.exit(1);
|
|
383
|
+
}
|
|
384
|
+
}
|
|
317
385
|
process.exit(ok ? 0 : 1);
|
|
318
386
|
}
|
|
319
387
|
else if (cmd !== undefined && cmd !== '--help' && cmd !== '-h') {
|
|
@@ -331,5 +399,8 @@ else {
|
|
|
331
399
|
console.log(' verify --remote <url> [--json] same, fetched from a measurement URL');
|
|
332
400
|
console.log(' measure --name x --command "npx -y <server>" run a one-off measurement');
|
|
333
401
|
console.log(' measure --remote <url> [--name x] measure a remote server via mcp-remote');
|
|
402
|
+
console.log(' [--baseline <measurement.json>] [--max-increase N] [--budget N]');
|
|
403
|
+
console.log(' gate your own server in CI: fail the');
|
|
404
|
+
console.log(' build when a change adds too much');
|
|
334
405
|
console.log('exit codes: 0 ok, 1 verification/measurement/budget failed, 2 usage error');
|
|
335
406
|
}
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Which published version is this, by its bytes?
|
|
3
|
+
*
|
|
4
|
+
* `audit --changed` answers "did the servers in my config get heavier?" — and
|
|
5
|
+
* the whole question turns on joining a machine's installed server to the
|
|
6
|
+
* published history. Matching by *name* would be a lie waiting to happen: a
|
|
7
|
+
* config's keys are arbitrary local labels, so a server a user calls `github`
|
|
8
|
+
* may be a fork, a pin, or something else entirely, and reporting the official
|
|
9
|
+
* server's movement against it would be a confident false statement.
|
|
10
|
+
*
|
|
11
|
+
* So the join is byte identity — the same discipline the Claude column already
|
|
12
|
+
* uses to decide whether it may print. `results/capture-index.json` maps the
|
|
13
|
+
* `canonicalSha256` of every capture the project has ever published to the
|
|
14
|
+
* server and date it belongs to. A local measurement either *is* one of those
|
|
15
|
+
* captures, exactly, or it is not in the published history at all, and there is
|
|
16
|
+
* no third state to be fuzzy about.
|
|
17
|
+
*
|
|
18
|
+
* The index is derivable from the per-server tool vectors, which is why it can
|
|
19
|
+
* only see as far back as those vectors go: a version published before the
|
|
20
|
+
* vectors existed is not in the index and reads as unknown, not as absent from
|
|
21
|
+
* history. Versioned independently, like every published artifact.
|
|
22
|
+
*/
|
|
23
|
+
/** Method identifier, versioned independently of METHODOLOGY_VERSION. */
|
|
24
|
+
export declare const CAPTURE_INDEX_METHOD = "capture-index/v1";
|
|
25
|
+
export interface IndexedCapture {
|
|
26
|
+
server: string;
|
|
27
|
+
/** The day this capture was first measured. */
|
|
28
|
+
date: string;
|
|
29
|
+
totalTokens: number;
|
|
30
|
+
toolCount: number;
|
|
31
|
+
}
|
|
32
|
+
export interface CaptureIndex {
|
|
33
|
+
method: string;
|
|
34
|
+
/** UTC day the index was derived (YYYY-MM-DD). */
|
|
35
|
+
generatedAt: string;
|
|
36
|
+
/** canonicalSha256 → the capture it identifies. */
|
|
37
|
+
captures: Record<string, IndexedCapture>;
|
|
38
|
+
/** server → the newest published capture's hash. */
|
|
39
|
+
current: Record<string, string>;
|
|
40
|
+
}
|
|
41
|
+
export declare function parseCaptureIndex(text: string): CaptureIndex | null;
|
|
42
|
+
/**
|
|
43
|
+
* What the index can say about one locally measured server.
|
|
44
|
+
*
|
|
45
|
+
* `unknown` is the honest and common outcome: users install versions this
|
|
46
|
+
* project has never measured, and versions published before the capture index
|
|
47
|
+
* existed are not in it either. It is reported, not hidden — an absence of a
|
|
48
|
+
* record about that version, not a statement that nothing changed.
|
|
49
|
+
*/
|
|
50
|
+
export type CaptureVerdict = {
|
|
51
|
+
kind: 'behind';
|
|
52
|
+
/** The published server this capture belongs to — established by bytes, not by name. */
|
|
53
|
+
server: string;
|
|
54
|
+
yourDate: string;
|
|
55
|
+
yourTokens: number;
|
|
56
|
+
currentDate: string;
|
|
57
|
+
currentTokens: number;
|
|
58
|
+
/** What moving to the current published version would add to every request. */
|
|
59
|
+
deltaTokens: number;
|
|
60
|
+
} | {
|
|
61
|
+
kind: 'current';
|
|
62
|
+
server: string;
|
|
63
|
+
date: string;
|
|
64
|
+
tokens: number;
|
|
65
|
+
} | {
|
|
66
|
+
kind: 'unknown';
|
|
67
|
+
};
|
|
68
|
+
/**
|
|
69
|
+
* Identify a local capture against the published index.
|
|
70
|
+
*
|
|
71
|
+
* A hash that matches the server's newest published capture is `current`; one
|
|
72
|
+
* that matches an older capture is `behind`, carrying the exact delta to
|
|
73
|
+
* current. Anything else — including a version newer than anything published —
|
|
74
|
+
* is `unknown`, because the index cannot describe what it has never measured.
|
|
75
|
+
*/
|
|
76
|
+
export declare function identify(canonicalSha256: string | null | undefined, index: CaptureIndex): CaptureVerdict;
|