mcp-context-cost 0.11.3 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -13
- package/dist/audit/deferral.js +14 -1
- package/dist/core/regression.d.ts +87 -5
- package/dist/core/regression.js +63 -22
- package/dist/core/types.d.ts +23 -1
- package/dist/sweep/client.d.ts +22 -0
- package/dist/sweep/client.js +93 -2
- package/dist/sweep/cross-check.js +3 -0
- package/dist/sweep/dashboard.js +17 -2
- package/dist/sweep/docker.d.ts +46 -0
- package/dist/sweep/docker.js +82 -4
- package/dist/sweep/history.d.ts +16 -1
- package/dist/sweep/history.js +18 -5
- package/dist/sweep/published-stats.js +48 -3
- package/dist/sweep/regressions.js +131 -18
- package/dist/sweep/report.d.ts +60 -0
- package/dist/sweep/report.js +44 -1
- package/dist/sweep/run.d.ts +83 -0
- package/dist/sweep/run.js +107 -8
- package/dist/sweep/server-pages.js +15 -4
- package/dist/sweep/servers-schema.d.ts +41 -0
- package/dist/sweep/servers-schema.js +245 -0
- package/dist/sweep/session-start.js +2 -0
- package/dist/sweep/sweep-all.js +15 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,6 +2,27 @@
|
|
|
2
2
|
|
|
3
3
|
[](https://www.npmjs.com/package/mcp-context-cost)
|
|
4
4
|
[](https://github.com/athakur3/mcp-context-cost/actions/workflows/ci.yml)
|
|
5
|
+
[](https://github.com/marketplace/actions/mcp-context-cost)
|
|
6
|
+
|
|
7
|
+
Two entry points: `audit`, for the config you run (below), and a GitHub Action, for the
|
|
8
|
+
server you publish — five lines in a workflow that fail a pull request adding more context
|
|
9
|
+
cost than you meant to ship:
|
|
10
|
+
|
|
11
|
+
```yaml
|
|
12
|
+
- uses: athakur3/mcp-context-cost@v1
|
|
13
|
+
with:
|
|
14
|
+
name: my-server
|
|
15
|
+
command: node dist/index.js
|
|
16
|
+
baseline: .context-cost/baseline.json
|
|
17
|
+
max-increase: 500
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Inputs: `name`, `command` or `remote`, `baseline`, `max-increase`, `budget`, `timeout`,
|
|
21
|
+
`version`. Outputs: `tokens`, `tools`, `status`, `measurement`, `badge` — written whether the
|
|
22
|
+
gate passed or not, so a later step can comment the number on the PR or publish the badge.
|
|
23
|
+
[Full workflow](examples/server-author-ci.yml) ·
|
|
24
|
+
[how the gate decides](#defend-the-number-dont-just-display-it) ·
|
|
25
|
+
[what the number is](docs/METHODOLOGY.md).
|
|
5
26
|
|
|
6
27
|
**What do the MCP servers in your config cost you before you type anything — and what did
|
|
7
28
|
that last config change add to every session you will ever run?**
|
|
@@ -105,7 +126,7 @@ costs **more** than loading the definitions would.
|
|
|
105
126
|
|
|
106
127
|
Three things the report will not do: it will not convert between units silently (in
|
|
107
128
|
threshold mode the stack is compared as a range, because the audit counts wire bytes and the
|
|
108
|
-
threshold is counted in what the client sends to the API — measured at 0.20×–1.92× across
|
|
129
|
+
threshold is counted in what the client sends to the API — measured at 0.20×–1.92× across 23
|
|
109
130
|
servers); it will not claim a posture the machine did not state readably, which is four
|
|
110
131
|
refusals and not one — when two places set the same variable to different values, when a
|
|
111
132
|
settings file exists and cannot be read, when the place that would decide sets the variable
|
|
@@ -183,8 +204,8 @@ INCREASE FAIL:
|
|
|
183
204
|
Add `--claude` to annotate each server with its Anthropic-request cost from the published
|
|
184
205
|
[Claude divergence](docs/METHODOLOGY.md#claude-divergence) run — an exact number when the
|
|
185
206
|
published capture hash matches what you have installed, `—` (silence, not a stale guess)
|
|
186
|
-
when it doesn't. The run holds
|
|
187
|
-
and [results/leaderboard.md](results/leaderboard.md) prints a claude number for the
|
|
207
|
+
when it doesn't. The run holds 23 rows — the measured servers it covered when it last ran —
|
|
208
|
+
and [results/leaderboard.md](results/leaderboard.md) prints a claude number for the 15 that
|
|
188
209
|
still match today and silence for the rest. Most installs will show a mix:
|
|
189
210
|
|
|
190
211
|
```
|
|
@@ -235,7 +256,7 @@ Flags: `--json` (full report on stdout, progress on stderr), `--budget N`,
|
|
|
235
256
|
|
|
236
257
|
The number `audit` gives you is the same measurement, run across a curated set of public
|
|
237
258
|
servers — which is how you can tell it is a measurement and not this tool's opinion. It also
|
|
238
|
-
shows what you are choosing between: across the
|
|
259
|
+
shows what you are choosing between: across the 86 servers measured, cost spans **1,700×**,
|
|
239
260
|
from `postgres` at 32 tokens to `github` at 54,622. The table below is a
|
|
240
261
|
sample of that range; the full range is in
|
|
241
262
|
[results/leaderboard.md](results/leaderboard.md).
|
|
@@ -244,13 +265,13 @@ sample of that range; the full range is in
|
|
|
244
265
|
|---|---:|---:|
|
|
245
266
|
| github (official) | **54,622 tokens** | 44 |
|
|
246
267
|
| xcodebuildmcp | 26,594 | 24 |
|
|
247
|
-
| brave-search | 25,
|
|
268
|
+
| brave-search | 25,487 | 8 |
|
|
248
269
|
| notion | 17,500 | 24 |
|
|
249
270
|
| playwright *(4.8M installs/week)* | 4,024 | 24 |
|
|
250
271
|
| filesystem (reference) | 2,823 | 14 |
|
|
251
272
|
| markitdown | 64 | 1 |
|
|
252
273
|
|
|
253
|
-
*(
|
|
274
|
+
*(86 of 106 popular servers measured, each row dated by its own most recent sweep — full table in
|
|
254
275
|
[results/leaderboard.md](results/leaderboard.md); every failure is listed with its reason.
|
|
255
276
|
Each measured server also has a [detail page](https://athakur3.github.io/mcp-context-cost/servers/)
|
|
256
277
|
showing which tools its tokens are in.)*
|
|
@@ -260,7 +281,7 @@ answers a question no client asks: **what did this server cost last month?**
|
|
|
260
281
|
[results/regressions.md](results/regressions.md) reports each server's most recent movement —
|
|
261
282
|
dated to when it happened, separated into *shipped more tools* versus *same tools, rewritten*,
|
|
262
283
|
and compared only within one isolation. The ecosystem ratchets upward: of the servers whose
|
|
263
|
-
cost has moved at all,
|
|
284
|
+
cost has moved at all, 11 moved up against 6 that moved down. Method:
|
|
264
285
|
[cost movement](docs/METHODOLOGY.md#cost-movement).
|
|
265
286
|
|
|
266
287
|
If you publish a server, the same measurement is available as a badge, so your users can see
|
|
@@ -328,7 +349,7 @@ number is *not*, config policy, failure taxonomy, frozen color bands, known dive
|
|
|
328
349
|
| `spec/fixtures/` | golden vectors shared by the TypeScript and bash implementations |
|
|
329
350
|
| `tools/` | the one script that calls a network API (Claude divergence); kept out of the package so the library stays offline |
|
|
330
351
|
| `upstream/` | `badge.sh` + composite-action patch + bash tests — the self-serve badge recipe, carried here |
|
|
331
|
-
| `servers.yaml` |
|
|
352
|
+
| `servers.yaml` | 106 curated candidates with live install metrics and provenance |
|
|
332
353
|
| `results/` · `badges/` | measurements, leaderboard, history series, shields endpoint JSONs |
|
|
333
354
|
| `docs/` | methodology, dashboard, and a generated page per measured server |
|
|
334
355
|
|
|
@@ -350,11 +371,11 @@ Then in your README:
|
|
|
350
371
|
|
|
351
372
|
A badge says what your server costs today; it does nothing about the release
|
|
352
373
|
that adds 1,200 tokens to every user's context next month. Across the servers
|
|
353
|
-
measured here
|
|
354
|
-
[movement report](results/regressions.md) has
|
|
355
|
-
against
|
|
356
|
-
would have said so first. `measure` takes the same
|
|
357
|
-
your own CI can be that check:
|
|
374
|
+
measured here most costs hold steady from sweep to sweep, but when a cost does
|
|
375
|
+
move it usually moves up: the [movement report](results/regressions.md) has 11
|
|
376
|
+
servers ratcheting upward against 6 that got cheaper, and none of those
|
|
377
|
+
maintainers had a check that would have said so first. `measure` takes the same
|
|
378
|
+
gate flags `audit` does, so your own CI can be that check:
|
|
358
379
|
|
|
359
380
|
```bash
|
|
360
381
|
# on your default branch, once — commit the result
|
package/dist/audit/deferral.js
CHANGED
|
@@ -229,7 +229,20 @@ export function resolveToolSearchSources(sources) {
|
|
|
229
229
|
export const PUBLISHED_WIRE_TO_CLIENT_RATIO = {
|
|
230
230
|
low: 0.2,
|
|
231
231
|
high: 1.92,
|
|
232
|
-
|
|
232
|
+
// A snapshot of the run this package was cut against, which is what `source`
|
|
233
|
+
// below says it is — the installed package has no `results/` to read, so when
|
|
234
|
+
// a live run is supplied `wireToClientRatio` uses that instead and this is
|
|
235
|
+
// never consulted. It may therefore lag the run on trunk, and a test holds it
|
|
236
|
+
// to the two things that matter: the band must still be accurate to the
|
|
237
|
+
// precision it is published at, because a wrong band gives a wrong
|
|
238
|
+
// above/below verdict, and the count must never *exceed* the run, because
|
|
239
|
+
// that would be a claim about servers nobody measured.
|
|
240
|
+
//
|
|
241
|
+
// It read 20 from the day the run covered the top 20 until the run widened to
|
|
242
|
+
// every measured server on 2026-09-05, with nothing comparing the two. The
|
|
243
|
+
// band had not moved — the servers added sat inside it — which is exactly how
|
|
244
|
+
// a number like this goes wrong quietly.
|
|
245
|
+
servers: 23,
|
|
233
246
|
source: 'the published claude-opus-5 divergence run',
|
|
234
247
|
};
|
|
235
248
|
/** Derive the band from a supplied divergence run, falling back to the published one. */
|
|
@@ -42,6 +42,8 @@ export interface DatedMeasurement {
|
|
|
42
42
|
tokens: number;
|
|
43
43
|
toolCount: number;
|
|
44
44
|
status: string;
|
|
45
|
+
/** What the server called itself at `initialize`, or `''` when not recorded. */
|
|
46
|
+
version?: string;
|
|
45
47
|
}
|
|
46
48
|
/** Method identifier, versioned independently of METHODOLOGY_VERSION. */
|
|
47
49
|
export declare const REGRESSION_METHOD = "cost-regression/v1";
|
|
@@ -67,6 +69,14 @@ export interface ToolVectorEntry {
|
|
|
67
69
|
canonicalSha256: string;
|
|
68
70
|
totalTokens: number;
|
|
69
71
|
tools: ToolVector[];
|
|
72
|
+
/**
|
|
73
|
+
* The version the server reported at `initialize`, absent when it reported
|
|
74
|
+
* none or when the entry was stored before this field existed. Entries are
|
|
75
|
+
* deduped by capture and never rewritten, so an existing one does not gain a
|
|
76
|
+
* version retroactively — which is correct: identical bytes are the same
|
|
77
|
+
* definitions, and what upstream called them that day is not on record.
|
|
78
|
+
*/
|
|
79
|
+
version?: string;
|
|
70
80
|
}
|
|
71
81
|
export interface ToolVectorFile {
|
|
72
82
|
method: string;
|
|
@@ -150,6 +160,17 @@ export interface CostChange {
|
|
|
150
160
|
fromToolCount: number;
|
|
151
161
|
toToolCount: number;
|
|
152
162
|
deltaTools: number;
|
|
163
|
+
/**
|
|
164
|
+
* What the server called itself on each side, or `''` where the row does not
|
|
165
|
+
* say. Never inferred: a movement whose earlier row predates the version
|
|
166
|
+
* column names no release, because nothing on disk records which one it was.
|
|
167
|
+
*
|
|
168
|
+
* The two being *equal* is a reading in its own right, and a common one —
|
|
169
|
+
* the cost moved while the version did not, which means the change came from
|
|
170
|
+
* a dependency the server does not pin rather than from its own release.
|
|
171
|
+
*/
|
|
172
|
+
fromVersion: string;
|
|
173
|
+
toVersion: string;
|
|
153
174
|
mechanism: Mechanism;
|
|
154
175
|
/** True when the movement clears both thresholds. */
|
|
155
176
|
significant: boolean;
|
|
@@ -162,11 +183,44 @@ export interface CostChange {
|
|
|
162
183
|
/** Per-tool breakdown, or null when the two captures are not both on record. */
|
|
163
184
|
attribution: ToolAttribution | null;
|
|
164
185
|
}
|
|
186
|
+
/**
|
|
187
|
+
* A cost that has been measured more than once, under the same isolation, and
|
|
188
|
+
* has not moved.
|
|
189
|
+
*
|
|
190
|
+
* Published because it is a measured fact and not the absence of one. The
|
|
191
|
+
* report used to swallow it: `latestChange` returned null both for this and
|
|
192
|
+
* for "there is nothing to compare", and the page counted every null the same
|
|
193
|
+
* way — so a number four sweeps had confirmed read exactly like a number
|
|
194
|
+
* measured once.
|
|
195
|
+
*/
|
|
196
|
+
export interface UnchangedSeries {
|
|
197
|
+
server: string;
|
|
198
|
+
tokens: number;
|
|
199
|
+
toolCount: number;
|
|
200
|
+
/** The first date this cost was recorded — when it became this, not when it was last looked at. */
|
|
201
|
+
since: string;
|
|
202
|
+
/** The newest measurement on record. */
|
|
203
|
+
measuredThrough: string;
|
|
204
|
+
/** How many comparable measurements carry this same cost, `since` included. */
|
|
205
|
+
sweeps: number;
|
|
206
|
+
}
|
|
207
|
+
/**
|
|
208
|
+
* What one server's series supports. Three readings, and they are three
|
|
209
|
+
* different claims: the cost moved, the cost was confirmed and held, or there
|
|
210
|
+
* is not yet a second comparable measurement to say either.
|
|
211
|
+
*/
|
|
212
|
+
export type SeriesReading = {
|
|
213
|
+
kind: 'changed';
|
|
214
|
+
change: CostChange;
|
|
215
|
+
} | {
|
|
216
|
+
kind: 'unchanged';
|
|
217
|
+
held: UnchangedSeries;
|
|
218
|
+
} | {
|
|
219
|
+
kind: 'incomparable';
|
|
220
|
+
};
|
|
165
221
|
export declare function isSignificant(deltaTokens: number, deltaPct: number): boolean;
|
|
166
222
|
/**
|
|
167
|
-
*
|
|
168
|
-
* isn't one: fewer than two comparable measurements, or a series that has never
|
|
169
|
-
* changed.
|
|
223
|
+
* What one server's history supports, as one of three readings.
|
|
170
224
|
*
|
|
171
225
|
* Deliberately not "the newest pair". A server that grew 82% one week and held
|
|
172
226
|
* that cost since has a newest pair of zero, and reporting only that would hide
|
|
@@ -176,10 +230,23 @@ export declare function isSignificant(deltaTokens: number, deltaPct: number): bo
|
|
|
176
230
|
* actually happened — `2026-08-19 → 2026-08-26`, not a span up to today.
|
|
177
231
|
* `measuredThrough` then carries how long the new cost has held.
|
|
178
232
|
*
|
|
233
|
+
* Where that walk ends decides the reading. Reaching the start of the series
|
|
234
|
+
* means every comparable measurement agrees, which is `unchanged` — a fact
|
|
235
|
+
* about the server. Having fewer than two comparable rows to walk means there
|
|
236
|
+
* is nothing to compare, which is `incomparable` — an absence. The two used
|
|
237
|
+
* to share one `null` return, and the report counted them together.
|
|
238
|
+
*
|
|
179
239
|
* `rows` is that server's history, already narrowed to the run a trend may be
|
|
180
240
|
* drawn across (`plottableSeries`), so the isolation rule is applied once, in
|
|
181
241
|
* the place that owns it.
|
|
182
242
|
*/
|
|
243
|
+
export declare function readSeries(server: string, rows: DatedMeasurement[], vectors?: ToolVectorFile | null): SeriesReading;
|
|
244
|
+
/**
|
|
245
|
+
* The most recent comparable movement, or null when there was none. Kept as
|
|
246
|
+
* the narrow question `readSeries` answers in full, for the callers that only
|
|
247
|
+
* ever wanted a movement — but a caller that publishes a count must ask
|
|
248
|
+
* `readSeries`, because null here is two different facts.
|
|
249
|
+
*/
|
|
183
250
|
export declare function latestChange(server: string, rows: DatedMeasurement[], vectors?: ToolVectorFile | null): CostChange | null;
|
|
184
251
|
export interface RegressionSummary {
|
|
185
252
|
changes: CostChange[];
|
|
@@ -188,7 +255,22 @@ export interface RegressionSummary {
|
|
|
188
255
|
significant: number;
|
|
189
256
|
/** Net tokens the measured set gained (or lost) across every listed movement. */
|
|
190
257
|
netTokens: number;
|
|
191
|
-
/**
|
|
258
|
+
/**
|
|
259
|
+
* Servers measured more than once under the same isolation whose cost has
|
|
260
|
+
* not moved. Counted apart from `withoutComparison`, which they were folded
|
|
261
|
+
* into while both readings shared one `null`.
|
|
262
|
+
*/
|
|
263
|
+
unchanged: UnchangedSeries[];
|
|
264
|
+
/**
|
|
265
|
+
* Servers with no second comparable measurement to diff against — a first
|
|
266
|
+
* measurement, or every earlier run taken under different isolation. Not
|
|
267
|
+
* "has not moved": that is `unchanged`.
|
|
268
|
+
*/
|
|
192
269
|
withoutComparison: number;
|
|
193
270
|
}
|
|
194
|
-
|
|
271
|
+
/**
|
|
272
|
+
* `unchanged` is required rather than defaulted: every reading a caller
|
|
273
|
+
* classifies belongs to exactly one of the three, and a default would let a
|
|
274
|
+
* caller that forgot them publish a total that quietly does not sum.
|
|
275
|
+
*/
|
|
276
|
+
export declare function summarize(changes: CostChange[], withoutComparison: number, unchanged: UnchangedSeries[]): RegressionSummary;
|
package/dist/core/regression.js
CHANGED
|
@@ -38,6 +38,7 @@ export function parseToolVectorFile(text) {
|
|
|
38
38
|
date: e.date,
|
|
39
39
|
canonicalSha256: e.canonicalSha256,
|
|
40
40
|
totalTokens: e.totalTokens,
|
|
41
|
+
...(typeof e.version === 'string' && e.version ? { version: e.version } : {}),
|
|
41
42
|
tools: e.tools
|
|
42
43
|
.filter((t) => !!t && typeof t.name === 'string' && typeof t.tokens === 'number')
|
|
43
44
|
.map((t) => ({ name: t.name, tokens: t.tokens })),
|
|
@@ -62,6 +63,7 @@ export function vectorEntryOf(m) {
|
|
|
62
63
|
date,
|
|
63
64
|
canonicalSha256: m.canonicalSha256,
|
|
64
65
|
totalTokens: m.totalTokens,
|
|
66
|
+
...(m.serverVersion ? { version: m.serverVersion } : {}),
|
|
65
67
|
tools: m.tools.map((t) => ({ name: t.name, tokens: t.tokens })),
|
|
66
68
|
};
|
|
67
69
|
}
|
|
@@ -139,9 +141,7 @@ export function isSignificant(deltaTokens, deltaPct) {
|
|
|
139
141
|
return Math.abs(deltaPct) >= SIGNIFICANT_PCT && Math.abs(deltaTokens) >= SIGNIFICANT_TOKENS;
|
|
140
142
|
}
|
|
141
143
|
/**
|
|
142
|
-
*
|
|
143
|
-
* isn't one: fewer than two comparable measurements, or a series that has never
|
|
144
|
-
* changed.
|
|
144
|
+
* What one server's history supports, as one of three readings.
|
|
145
145
|
*
|
|
146
146
|
* Deliberately not "the newest pair". A server that grew 82% one week and held
|
|
147
147
|
* that cost since has a newest pair of zero, and reporting only that would hide
|
|
@@ -151,14 +151,20 @@ export function isSignificant(deltaTokens, deltaPct) {
|
|
|
151
151
|
* actually happened — `2026-08-19 → 2026-08-26`, not a span up to today.
|
|
152
152
|
* `measuredThrough` then carries how long the new cost has held.
|
|
153
153
|
*
|
|
154
|
+
* Where that walk ends decides the reading. Reaching the start of the series
|
|
155
|
+
* means every comparable measurement agrees, which is `unchanged` — a fact
|
|
156
|
+
* about the server. Having fewer than two comparable rows to walk means there
|
|
157
|
+
* is nothing to compare, which is `incomparable` — an absence. The two used
|
|
158
|
+
* to share one `null` return, and the report counted them together.
|
|
159
|
+
*
|
|
154
160
|
* `rows` is that server's history, already narrowed to the run a trend may be
|
|
155
161
|
* drawn across (`plottableSeries`), so the isolation rule is applied once, in
|
|
156
162
|
* the place that owns it.
|
|
157
163
|
*/
|
|
158
|
-
export function
|
|
164
|
+
export function readSeries(server, rows, vectors) {
|
|
159
165
|
const usable = rows.filter((r) => r.status === 'measured' || r.status === 'dynamic');
|
|
160
166
|
if (usable.length < 2)
|
|
161
|
-
return
|
|
167
|
+
return { kind: 'incomparable' };
|
|
162
168
|
const newest = usable[usable.length - 1];
|
|
163
169
|
// Walk back over measurements identical to the newest: the first of that run
|
|
164
170
|
// is when the current cost arrived.
|
|
@@ -169,14 +175,26 @@ export function latestChange(server, rows, vectors) {
|
|
|
169
175
|
break;
|
|
170
176
|
toIdx--;
|
|
171
177
|
}
|
|
178
|
+
/** The trailing run reaches back this far, and every row in it agrees. */
|
|
179
|
+
const held = () => ({
|
|
180
|
+
kind: 'unchanged',
|
|
181
|
+
held: {
|
|
182
|
+
server,
|
|
183
|
+
tokens: newest.tokens,
|
|
184
|
+
toolCount: newest.toolCount,
|
|
185
|
+
since: usable[toIdx].date,
|
|
186
|
+
measuredThrough: newest.date,
|
|
187
|
+
sweeps: usable.length - toIdx,
|
|
188
|
+
},
|
|
189
|
+
});
|
|
172
190
|
if (toIdx === 0)
|
|
173
|
-
return
|
|
191
|
+
return held(); // every comparable measurement agrees
|
|
174
192
|
const to = usable[toIdx];
|
|
175
193
|
const from = usable[toIdx - 1];
|
|
176
194
|
const deltaTokens = to.tokens - from.tokens;
|
|
177
195
|
const deltaTools = to.toolCount - from.toolCount;
|
|
178
196
|
if (deltaTokens === 0 && deltaTools === 0)
|
|
179
|
-
return
|
|
197
|
+
return held();
|
|
180
198
|
const deltaPct = from.tokens > 0 ? (deltaTokens / from.tokens) * 100 : 0;
|
|
181
199
|
// Attribution needs both sides on record. Matched by *cost as of that date*,
|
|
182
200
|
// not by date equality: vectors are deduped by capture and keep the first
|
|
@@ -209,23 +227,43 @@ export function latestChange(server, rows, vectors) {
|
|
|
209
227
|
attribution = attribute(fromVec, toVec, deltaTokens);
|
|
210
228
|
}
|
|
211
229
|
return {
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
230
|
+
kind: 'changed',
|
|
231
|
+
change: {
|
|
232
|
+
server,
|
|
233
|
+
fromDate: from.date,
|
|
234
|
+
toDate: to.date,
|
|
235
|
+
fromTokens: from.tokens,
|
|
236
|
+
toTokens: to.tokens,
|
|
237
|
+
deltaTokens,
|
|
238
|
+
deltaPct,
|
|
239
|
+
fromToolCount: from.toolCount,
|
|
240
|
+
toToolCount: to.toolCount,
|
|
241
|
+
deltaTools,
|
|
242
|
+
fromVersion: from.version ?? '',
|
|
243
|
+
toVersion: to.version ?? '',
|
|
244
|
+
mechanism: mechanismOf(deltaTokens, deltaTools),
|
|
245
|
+
significant: isSignificant(deltaTokens, deltaPct),
|
|
246
|
+
measuredThrough: newest.date,
|
|
247
|
+
attribution,
|
|
248
|
+
},
|
|
226
249
|
};
|
|
227
250
|
}
|
|
228
|
-
|
|
251
|
+
/**
|
|
252
|
+
* The most recent comparable movement, or null when there was none. Kept as
|
|
253
|
+
* the narrow question `readSeries` answers in full, for the callers that only
|
|
254
|
+
* ever wanted a movement — but a caller that publishes a count must ask
|
|
255
|
+
* `readSeries`, because null here is two different facts.
|
|
256
|
+
*/
|
|
257
|
+
export function latestChange(server, rows, vectors) {
|
|
258
|
+
const reading = readSeries(server, rows, vectors);
|
|
259
|
+
return reading.kind === 'changed' ? reading.change : null;
|
|
260
|
+
}
|
|
261
|
+
/**
|
|
262
|
+
* `unchanged` is required rather than defaulted: every reading a caller
|
|
263
|
+
* classifies belongs to exactly one of the three, and a default would let a
|
|
264
|
+
* caller that forgot them publish a total that quietly does not sum.
|
|
265
|
+
*/
|
|
266
|
+
export function summarize(changes, withoutComparison, unchanged) {
|
|
229
267
|
const sorted = [...changes].sort((a, b) => Math.abs(b.deltaPct) - Math.abs(a.deltaPct));
|
|
230
268
|
return {
|
|
231
269
|
changes: sorted,
|
|
@@ -233,6 +271,9 @@ export function summarize(changes, withoutComparison) {
|
|
|
233
271
|
shrank: sorted.filter((c) => c.deltaTokens < 0).length,
|
|
234
272
|
significant: sorted.filter((c) => c.significant).length,
|
|
235
273
|
netTokens: sorted.reduce((a, c) => a + c.deltaTokens, 0),
|
|
274
|
+
// Longest-held first: the sentence this section exists to publish is "this
|
|
275
|
+
// number has been the same since <date>", so the oldest `since` leads.
|
|
276
|
+
unchanged: [...unchanged].sort((a, b) => a.since.localeCompare(b.since) || b.tokens - a.tokens),
|
|
236
277
|
withoutComparison,
|
|
237
278
|
};
|
|
238
279
|
}
|
package/dist/core/types.d.ts
CHANGED
|
@@ -1,5 +1,15 @@
|
|
|
1
1
|
/** Status taxonomy — every swept server gets exactly one; no silent drops. */
|
|
2
|
-
export type MeasurementStatus = 'measured' | 'auth-required' | 'startup-failure' | 'timeout'
|
|
2
|
+
export type MeasurementStatus = 'measured' | 'auth-required' | 'startup-failure' | 'timeout'
|
|
3
|
+
/**
|
|
4
|
+
* This harness cannot run the server, for a reason that is a property of the
|
|
5
|
+
* harness rather than of the software: an OS or architecture the package does
|
|
6
|
+
* not ship for, or a backing service the isolation deliberately does not
|
|
7
|
+
* provide. Distinct from `startup-failure`, which asserts the server did not
|
|
8
|
+
* come up — a claim about someone else's code that these entries do not
|
|
9
|
+
* support. Only ever set when the entry declares the reason AND the failure's
|
|
10
|
+
* own text corroborates it (see `notApplicable` in report.ts).
|
|
11
|
+
*/
|
|
12
|
+
| 'not-applicable' | 'dynamic' | 'remote-auth-wall';
|
|
3
13
|
export interface ToolMeasurement {
|
|
4
14
|
name: string;
|
|
5
15
|
/** Tokens of the whole tool object, canonically serialized. */
|
|
@@ -43,6 +53,18 @@ export interface Measurement {
|
|
|
43
53
|
image?: string;
|
|
44
54
|
network?: string;
|
|
45
55
|
note?: string;
|
|
56
|
+
/**
|
|
57
|
+
* The architecture the measurement ran on, as `<platform>/<arch>` (e.g.
|
|
58
|
+
* `linux/amd64`). Part of the isolation because some packages ship builds
|
|
59
|
+
* for only some of them: `local-mcp` was published as a startup failure on
|
|
60
|
+
* the strength of a run whose real finding was "this laptop is arm64 and
|
|
61
|
+
* the package has no arm64 runtime" — a fact about the machine that the
|
|
62
|
+
* record gave no way to see.
|
|
63
|
+
*
|
|
64
|
+
* Absent on records written before this was captured, which is why it is
|
|
65
|
+
* optional; absence means unknown, never "the same as yours".
|
|
66
|
+
*/
|
|
67
|
+
arch?: string;
|
|
46
68
|
};
|
|
47
69
|
/** Request timeout in force during this measurement. */
|
|
48
70
|
timeoutMs?: number;
|
package/dist/sweep/client.d.ts
CHANGED
|
@@ -13,6 +13,28 @@ export interface WireCapture {
|
|
|
13
13
|
instructions: string | null;
|
|
14
14
|
stderrTail: string;
|
|
15
15
|
}
|
|
16
|
+
/**
|
|
17
|
+
* The part of a dead server's stderr worth keeping as evidence.
|
|
18
|
+
*
|
|
19
|
+
* A failure record is only useful if it contains the failure, and a plain tail
|
|
20
|
+
* reliably keeps the least useful part. `npx` prints a deprecation warning per
|
|
21
|
+
* transitive dependency and a version notice at the end, and a crashing process
|
|
22
|
+
* prints its message *before* the stack — so the last N characters of stderr
|
|
23
|
+
* are npm warnings and stack frames on exactly the servers whose failure needs
|
|
24
|
+
* explaining. Several published records ended up saying nothing about why the
|
|
25
|
+
* server did not start.
|
|
26
|
+
*
|
|
27
|
+
* This is not cosmetic. `run.ts` classifies a failure by reading these words:
|
|
28
|
+
* a record whose message was cut off is filed as `startup-failure` — the server
|
|
29
|
+
* is broken — when the surviving text would have said `auth-required`. Spending
|
|
30
|
+
* the budget on the message rather than the frames is what keeps the published
|
|
31
|
+
* taxonomy describing the server.
|
|
32
|
+
*
|
|
33
|
+
* Noise is only dropped while something else survives. A package that fails
|
|
34
|
+
* *inside* npm (EBADPLATFORM, a failed postinstall) has npm's own lines as its
|
|
35
|
+
* only evidence, and a server whose whole output is a stack keeps the stack.
|
|
36
|
+
*/
|
|
37
|
+
export declare function evidenceTail(stderr: string, limit?: number): string;
|
|
16
38
|
export declare class McpStdioClient {
|
|
17
39
|
private child;
|
|
18
40
|
private buffer;
|
package/dist/sweep/client.js
CHANGED
|
@@ -6,6 +6,92 @@
|
|
|
6
6
|
*/
|
|
7
7
|
import { spawn } from 'node:child_process';
|
|
8
8
|
const PROTOCOL_VERSION = '2025-06-18';
|
|
9
|
+
/**
|
|
10
|
+
* The part of a dead server's stderr worth keeping as evidence.
|
|
11
|
+
*
|
|
12
|
+
* A failure record is only useful if it contains the failure, and a plain tail
|
|
13
|
+
* reliably keeps the least useful part. `npx` prints a deprecation warning per
|
|
14
|
+
* transitive dependency and a version notice at the end, and a crashing process
|
|
15
|
+
* prints its message *before* the stack — so the last N characters of stderr
|
|
16
|
+
* are npm warnings and stack frames on exactly the servers whose failure needs
|
|
17
|
+
* explaining. Several published records ended up saying nothing about why the
|
|
18
|
+
* server did not start.
|
|
19
|
+
*
|
|
20
|
+
* This is not cosmetic. `run.ts` classifies a failure by reading these words:
|
|
21
|
+
* a record whose message was cut off is filed as `startup-failure` — the server
|
|
22
|
+
* is broken — when the surviving text would have said `auth-required`. Spending
|
|
23
|
+
* the budget on the message rather than the frames is what keeps the published
|
|
24
|
+
* taxonomy describing the server.
|
|
25
|
+
*
|
|
26
|
+
* Noise is only dropped while something else survives. A package that fails
|
|
27
|
+
* *inside* npm (EBADPLATFORM, a failed postinstall) has npm's own lines as its
|
|
28
|
+
* only evidence, and a server whose whole output is a stack keeps the stack.
|
|
29
|
+
*/
|
|
30
|
+
export function evidenceTail(stderr, limit = 600) {
|
|
31
|
+
const withoutNoise = drop(stderr, (l) => /^npm (warn|notice)\b/.test(l));
|
|
32
|
+
const withoutFrames = drop(withoutNoise, (l) => /^at\s/.test(l));
|
|
33
|
+
return bothEnds(withoutFrames, limit);
|
|
34
|
+
}
|
|
35
|
+
/**
|
|
36
|
+
* Keep the start and the end, eliding the middle.
|
|
37
|
+
*
|
|
38
|
+
* Dropping npm noise and stack frames is not enough on its own: a CLI that
|
|
39
|
+
* rejects its environment often prints one line saying why and then its entire
|
|
40
|
+
* usage screen, which is neither. kubernetes-mcp-server does exactly that, and
|
|
41
|
+
* a tail-only budget kept forty lines of flag documentation while discarding
|
|
42
|
+
* "no current-context is set and no contexts are defined in kubeconfig" — the
|
|
43
|
+
* only sentence in the output that explained anything.
|
|
44
|
+
*
|
|
45
|
+
* Failures put their explanation at one end or the other — a crash message
|
|
46
|
+
* above its aftermath, or an error at the end of a log — so both ends are kept
|
|
47
|
+
* and the middle is what goes. The split leans towards the head because a
|
|
48
|
+
* message that precedes its own noise is the more common shape here.
|
|
49
|
+
*/
|
|
50
|
+
function bothEnds(text, limit) {
|
|
51
|
+
if (text.length <= limit)
|
|
52
|
+
return text;
|
|
53
|
+
const elision = '\n […] \n';
|
|
54
|
+
const budget = Math.max(0, limit - elision.length);
|
|
55
|
+
const lines = text.split('\n');
|
|
56
|
+
// Whole lines only: a boundary cut mid-word ("ool/prompt change") reads as
|
|
57
|
+
// corruption and loses the token an evidence string would match on.
|
|
58
|
+
const take = (from, to, cap, fromEnd) => {
|
|
59
|
+
const out = [];
|
|
60
|
+
let used = 0;
|
|
61
|
+
for (let i = fromEnd ? to : from; fromEnd ? i >= from : i <= to; i += fromEnd ? -1 : 1) {
|
|
62
|
+
const cost = lines[i].length + 1;
|
|
63
|
+
if (used + cost > cap)
|
|
64
|
+
break;
|
|
65
|
+
fromEnd ? out.unshift(lines[i]) : out.push(lines[i]);
|
|
66
|
+
used += cost;
|
|
67
|
+
}
|
|
68
|
+
return { out, used };
|
|
69
|
+
};
|
|
70
|
+
const headCap = Math.ceil(budget * 0.6);
|
|
71
|
+
const head = take(0, lines.length - 1, headCap, false);
|
|
72
|
+
// Whole lines, except when the first line alone overruns the budget. A server
|
|
73
|
+
// that logs structured JSON puts its entire message on one line, so that line
|
|
74
|
+
// is both the most informative thing in the output and the only one that can
|
|
75
|
+
// never fit — slack-mcp-server's `{"level":"fatal","message":"Authentication
|
|
76
|
+
// required: ..."}` was dropped in full, and the record it left behind said a
|
|
77
|
+
// child process exited. Truncated evidence beats none.
|
|
78
|
+
const headText = head.out.length > 0 ? head.out.join('\n') : lines[0].slice(0, headCap);
|
|
79
|
+
const headUsed = head.out.length > 0 ? head.used : headText.length;
|
|
80
|
+
const tailFrom = head.out.length > 0 ? head.out.length : 1;
|
|
81
|
+
const tail = take(tailFrom, lines.length - 1, budget - headUsed, true);
|
|
82
|
+
if (headText === '' && tail.out.length === 0)
|
|
83
|
+
return text.slice(0, budget) + elision;
|
|
84
|
+
return `${headText}${elision}${tail.out.join('\n')}`;
|
|
85
|
+
}
|
|
86
|
+
/** Drop matching lines, keeping the input whole if that would leave nothing. */
|
|
87
|
+
function drop(text, isNoise) {
|
|
88
|
+
const kept = text
|
|
89
|
+
.split('\n')
|
|
90
|
+
.filter((l) => !isNoise(l.trim()))
|
|
91
|
+
.join('\n')
|
|
92
|
+
.trim();
|
|
93
|
+
return kept || text.trim();
|
|
94
|
+
}
|
|
9
95
|
export class McpStdioClient {
|
|
10
96
|
child;
|
|
11
97
|
buffer = '';
|
|
@@ -37,7 +123,7 @@ export class McpStdioClient {
|
|
|
37
123
|
resolve();
|
|
38
124
|
});
|
|
39
125
|
this.child.on('exit', (code) => {
|
|
40
|
-
const tail = this.stderrTail
|
|
126
|
+
const tail = evidenceTail(this.stderrTail);
|
|
41
127
|
this.deadReason = `server exited (code ${code})${tail ? `; stderr tail: ${tail}` : ''}`;
|
|
42
128
|
for (const p of this.pending.values())
|
|
43
129
|
p.reject(new Error(this.deadReason));
|
|
@@ -95,7 +181,12 @@ export class McpStdioClient {
|
|
|
95
181
|
return new Promise((resolve, reject) => {
|
|
96
182
|
const timer = setTimeout(() => {
|
|
97
183
|
this.pending.delete(id);
|
|
98
|
-
|
|
184
|
+
// A process that is killed mid-hang never reaches the exit handler, so
|
|
185
|
+
// without this a timed-out record carries no evidence at all — it says
|
|
186
|
+
// only that we waited. What the server managed to print before it
|
|
187
|
+
// stopped answering is usually the whole explanation.
|
|
188
|
+
const tail = evidenceTail(this.stderrTail);
|
|
189
|
+
reject(new Error(`timeout after ${timeoutMs}ms waiting for ${method}${tail ? `; stderr tail: ${tail}` : ''}`));
|
|
99
190
|
}, timeoutMs);
|
|
100
191
|
this.pending.set(id, {
|
|
101
192
|
resolve: (v) => {
|
|
@@ -162,6 +162,7 @@ export function runCli(binPath, entry, opts) {
|
|
|
162
162
|
dummyEnv: entry.env ?? [],
|
|
163
163
|
dummyEnvValues: entry.envValues,
|
|
164
164
|
needsGit: entry.needsGit,
|
|
165
|
+
aptPackages: entry.aptPackages,
|
|
165
166
|
containerName,
|
|
166
167
|
binds: [`${binPath}:/opt/mcp-tokens:ro`],
|
|
167
168
|
});
|
|
@@ -287,6 +288,8 @@ if (isMain) {
|
|
|
287
288
|
dummyEnv: e.env ?? [],
|
|
288
289
|
dummyEnvValues: e.envValues,
|
|
289
290
|
needsGit: e.needsGit,
|
|
291
|
+
aptPackages: e.aptPackages,
|
|
292
|
+
notApplicable: e.notApplicable,
|
|
290
293
|
persist: false, // the measurements on disk are not this run's to rewrite
|
|
291
294
|
});
|
|
292
295
|
}
|
package/dist/sweep/dashboard.js
CHANGED
|
@@ -66,7 +66,22 @@ export function generateDashboard(root = process.cwd()) {
|
|
|
66
66
|
const median = totals.length ? totals.slice().sort((a, b) => a - b)[Math.floor(totals.length / 2)] : 0;
|
|
67
67
|
const max = totals.length ? Math.max(...totals) : 1;
|
|
68
68
|
const fmt = (n) => n.toLocaleString('en-US');
|
|
69
|
-
|
|
69
|
+
/**
|
|
70
|
+
* The stamp is the newest measurement on the page, not the moment the page
|
|
71
|
+
* was written.
|
|
72
|
+
*
|
|
73
|
+
* A wall-clock stamp made `docs/dashboard.html` differ on every regeneration,
|
|
74
|
+
* so the scheduled jobs' "no change to publish" guard could never fire: one
|
|
75
|
+
* file that always diffs is enough to publish a commit that says nothing
|
|
76
|
+
* happened. Dated to the data instead, the page changes when the data does —
|
|
77
|
+
* and the line now answers the question a reader actually has, which is how
|
|
78
|
+
* fresh these numbers are, not when the HTML was rendered.
|
|
79
|
+
*/
|
|
80
|
+
const dates = rows
|
|
81
|
+
.map((r) => String(r.m?.measuredAt ?? '').slice(0, 10))
|
|
82
|
+
.filter((d) => /^\d{4}-\d{2}-\d{2}$/.test(d))
|
|
83
|
+
.sort();
|
|
84
|
+
const newestMeasurement = dates[dates.length - 1] ?? null;
|
|
70
85
|
const barRows = measured
|
|
71
86
|
.map((r, i) => {
|
|
72
87
|
const m = r.m;
|
|
@@ -225,7 +240,7 @@ export function generateDashboard(root = process.cwd()) {
|
|
|
225
240
|
</style>
|
|
226
241
|
<div class="wrap">
|
|
227
242
|
<header>
|
|
228
|
-
<p class="eyebrow">methodology v1.0 · o200k_base ·
|
|
243
|
+
<p class="eyebrow">methodology v1.0 · o200k_base · newest measurement ${newestMeasurement ?? 'not yet taken'}</p>
|
|
229
244
|
<h1>mcp-context-cost</h1>
|
|
230
245
|
<p class="sub">What popular MCP servers cost in context tokens before the agent does any work — measured from raw <code>tools/list</code> captures, every number re-derivable from its published measurement file.</p>
|
|
231
246
|
</header>
|