mcp-context-cost 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +131 -38
- package/dist/audit/audit.d.ts +63 -0
- package/dist/audit/audit.js +120 -9
- package/dist/audit/diff.d.ts +124 -0
- package/dist/audit/diff.js +318 -0
- package/dist/audit/run.d.ts +12 -0
- package/dist/audit/run.js +28 -1
- package/dist/cli.d.ts +21 -0
- package/dist/cli.js +141 -7
- package/dist/sweep/dashboard.d.ts +8 -0
- package/dist/sweep/dashboard.js +53 -5
- package/dist/sweep/docker.d.ts +8 -0
- package/dist/sweep/docker.js +7 -2
- package/dist/sweep/report.d.ts +2 -0
- package/dist/sweep/run.d.ts +2 -0
- package/dist/sweep/run.js +6 -1
- package/dist/sweep/sweep-all.js +1 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -3,37 +3,12 @@
|
|
|
3
3
|
[](https://www.npmjs.com/package/mcp-context-cost)
|
|
4
4
|
[](https://github.com/athakur3/mcp-context-cost/actions/workflows/ci.yml)
|
|
5
5
|
|
|
6
|
-
**
|
|
7
|
-
|
|
6
|
+
**What do the MCP servers in your config cost you before you type anything — and what did
|
|
7
|
+
that last config change add to every session you will ever run?**
|
|
8
8
|
|
|
9
|
-
Every MCP server you wire into an agent injects its tool schemas into the model's context
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
| server | context cost | tools |
|
|
14
|
-
|---|---:|---:|
|
|
15
|
-
| github (official) | **54,422 tokens** | 44 |
|
|
16
|
-
| brave-search | 25,456 | 8 |
|
|
17
|
-
| notion | 17,500 | 24 |
|
|
18
|
-
| playwright *(4.8M installs/week)* | 4,024 | 24 |
|
|
19
|
-
| filesystem (reference) | 2,823 | 14 |
|
|
20
|
-
| markitdown | 64 | 1 |
|
|
21
|
-
|
|
22
|
-
*(57 of 82 popular servers measured, 2026-08-16 sweep — full table in
|
|
23
|
-
[results/leaderboard.md](results/leaderboard.md); every failure is listed with its reason.
|
|
24
|
-
Each measured server also has a [detail page](https://athakur3.github.io/mcp-context-cost/servers/)
|
|
25
|
-
showing which tools its tokens are in.)*
|
|
26
|
-
|
|
27
|
-
This project makes that cost **legible and disputable**:
|
|
28
|
-
|
|
29
|
-
```
|
|
30
|
-
[context cost | 12,430 tokens] ← shields.io badge, linked to the methodology
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
## What does *your* setup cost?
|
|
34
|
-
|
|
35
|
-
The leaderboard measures one server at a time. You don't run one server — you run a stack.
|
|
36
|
-
Point `audit` at your own MCP config and it measures every server you actually have installed:
|
|
9
|
+
Every MCP server you wire into an agent injects its tool schemas into the model's context on
|
|
10
|
+
every single request. You pay that whether or not the agent ends up using the tools, and no
|
|
11
|
+
client shows you the number. Point `audit` at your own MCP config:
|
|
37
12
|
|
|
38
13
|
```bash
|
|
39
14
|
npx -y mcp-context-cost audit
|
|
@@ -55,6 +30,10 @@ claude-desktop ~/Library/Application Support/Claude/claude_desktop_config.json
|
|
|
55
30
|
heaviest tools
|
|
56
31
|
sequential-thinking · sequentialthinking 990
|
|
57
32
|
memory · search_nodes 323
|
|
33
|
+
|
|
34
|
+
trim: disabling 3 tools (sequential-thinking·sequentialthinking, memory·search_nodes,
|
|
35
|
+
memory·open_nodes) would recover 1,635 tokens (20.7% of this config) — if your client
|
|
36
|
+
supports per-tool filtering.
|
|
58
37
|
```
|
|
59
38
|
|
|
60
39
|
It finds configs for Claude Desktop, Claude Code (`~/.claude.json`, `.mcp.json`), Cursor,
|
|
@@ -67,6 +46,12 @@ Totals are reported per config file, never merged: a context window belongs to o
|
|
|
67
46
|
session, so summing Cursor's servers into Claude Desktop's total would describe a session
|
|
68
47
|
nobody runs.
|
|
69
48
|
|
|
49
|
+
One nuance: Claude Code's tool search (default-on in recent versions) defers full MCP
|
|
50
|
+
schemas until used, loading only tool names at session start. Audit totals are the weight
|
|
51
|
+
of the schema surface itself — what loads upfront in clients without deferral (Claude
|
|
52
|
+
Desktop, Cursor, VS Code, Windsurf today), and what Claude Code's documented fallback
|
|
53
|
+
modes still load. Deferral-aware reporting is on the roadmap.
|
|
54
|
+
|
|
70
55
|
**In CI**, make it a gate — the bundlesize move for agents:
|
|
71
56
|
|
|
72
57
|
```bash
|
|
@@ -74,8 +59,107 @@ npx -y mcp-context-cost audit --config .mcp.json --budget 20000
|
|
|
74
59
|
# exits 1 when the stack exceeds the budget, so a PR adding a 25K-token server fails
|
|
75
60
|
```
|
|
76
61
|
|
|
77
|
-
|
|
78
|
-
|
|
62
|
+
The budget is an absolute ceiling. What a reviewer actually wants to know is what *this pull
|
|
63
|
+
request* did, so record a baseline and diff against it:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
npx -y mcp-context-cost audit --config .mcp.json --json > baseline.json # on main
|
|
67
|
+
npx -y mcp-context-cost audit --config .mcp.json --baseline baseline.json --max-increase 2000
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
diff vs baseline measured 2026-08-18T01:51:49.555Z (methodology 1.0)
|
|
72
|
+
|
|
73
|
+
.mcp.json
|
|
74
|
+
2,378 → 5,201 +2,823
|
|
75
|
+
|
|
76
|
+
added filesystem — → 2,823 +2,823
|
|
77
|
+
(1 server unchanged)
|
|
78
|
+
|
|
79
|
+
This change adds 2,823 tokens to every request in this client — 1.2% → 2.6% of a
|
|
80
|
+
200,000-token context window.
|
|
81
|
+
|
|
82
|
+
INCREASE FAIL:
|
|
83
|
+
.mcp.json: +2,823 tokens per request, over the 2,000 allowed
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
> **Version note.** `--baseline` and `--max-increase` are **not in the published 0.3.0** —
|
|
87
|
+
> they are on `main` and ship in the next release. This matters more than a normal
|
|
88
|
+
> unreleased-feature note: 0.3.0 ignores flags it does not recognise, so running the command
|
|
89
|
+
> above against it produces a plain audit and **exit 0** — a passing CI check on a gate that
|
|
90
|
+
> never ran. Builds after 0.3.0 reject unknown flags with exit 2 instead. Until the next
|
|
91
|
+
> release, pin the gate to a version that has it, or it is not gating anything.
|
|
92
|
+
|
|
93
|
+
A baseline is just a stored `audit --json` report, so any artifact store works. Without
|
|
94
|
+
`--max-increase` the diff is informational and the exit code is unchanged.
|
|
95
|
+
|
|
96
|
+
[`examples/github-actions.yml`](examples/github-actions.yml) is the whole thing as a workflow:
|
|
97
|
+
measure the base branch, measure the PR, fail on the difference.
|
|
98
|
+
|
|
99
|
+
`--max-increase` fails on more than the number — it also fails whenever the increase could
|
|
100
|
+
not be established. A server that measured yesterday and won't start today takes its tokens
|
|
101
|
+
out of the total in exactly the way uninstalling it would, and reporting that as a saving is
|
|
102
|
+
the one mistake this tool must not make. So a server that crossed the measured/unmeasured
|
|
103
|
+
line, a config with no baseline, or a baseline config this run never found each fail the
|
|
104
|
+
gate and name themselves:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
Not a clean comparison: a server changed measured-ness between the two runs.
|
|
108
|
+
The measured total moved −2,378, but that is not what your config did.
|
|
109
|
+
|
|
110
|
+
memory: measured 2,378 in the baseline and could not be measured now — its cost is
|
|
111
|
+
missing from the total, not gone from your config
|
|
112
|
+
→ true cost is at least 2,378 higher than the 0 measured now.
|
|
113
|
+
|
|
114
|
+
INCREASE FAIL:
|
|
115
|
+
.mcp.json: a server changed measured-ness, so the change could not be established exactly
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Add `--claude` to annotate each server with its Anthropic-request cost from the published
|
|
119
|
+
[Claude divergence](docs/METHODOLOGY.md#claude-divergence) run — an exact number when the
|
|
120
|
+
published capture hash matches what you have installed, `—` (silence, not a stale guess)
|
|
121
|
+
when it doesn't (today the run covers the top 15 measured servers, so most installs will
|
|
122
|
+
show a mix):
|
|
123
|
+
|
|
124
|
+
```
|
|
125
|
+
server tools tokens share claude
|
|
126
|
+
github 44 54,422 95.8% 18,406
|
|
127
|
+
memory 9 2,378 4.2% —
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Flags: `--json` (full report on stdout, progress on stderr), `--budget N`,
|
|
131
|
+
`--baseline <report.json>`, `--max-increase N`, `--context N` (default 200,000),
|
|
132
|
+
`--timeout ms`, `--concurrency N`, `--docker`, `--claude`.
|
|
133
|
+
|
|
134
|
+
## Where the numbers come from
|
|
135
|
+
|
|
136
|
+
The number `audit` gives you is the same measurement, run across a curated set of public
|
|
137
|
+
servers — which is how you can tell it is a measurement and not this tool's opinion. It also
|
|
138
|
+
shows what you are choosing between: across the 65 servers measured, cost spans **1,700×**,
|
|
139
|
+
from the 32-token `postgres` reference server to github's 54,422. The table below starts at
|
|
140
|
+
markitdown's 64 tokens, an 850× spread; the full range is in
|
|
141
|
+
[results/leaderboard.md](results/leaderboard.md).
|
|
142
|
+
|
|
143
|
+
| server | context cost | tools |
|
|
144
|
+
|---|---:|---:|
|
|
145
|
+
| github (official) | **54,422 tokens** | 44 |
|
|
146
|
+
| brave-search | 25,456 | 8 |
|
|
147
|
+
| notion | 17,500 | 24 |
|
|
148
|
+
| playwright *(4.8M installs/week)* | 4,024 | 24 |
|
|
149
|
+
| filesystem (reference) | 2,823 | 14 |
|
|
150
|
+
| markitdown | 64 | 1 |
|
|
151
|
+
|
|
152
|
+
*(65 of 82 popular servers measured, 2026-08-18 sweep — full table in
|
|
153
|
+
[results/leaderboard.md](results/leaderboard.md); every failure is listed with its reason.
|
|
154
|
+
Each measured server also has a [detail page](https://athakur3.github.io/mcp-context-cost/servers/)
|
|
155
|
+
showing which tools its tokens are in.)*
|
|
156
|
+
|
|
157
|
+
If you publish a server, the same measurement is available as a badge, so your users can see
|
|
158
|
+
the cost before they install rather than after:
|
|
159
|
+
|
|
160
|
+
```
|
|
161
|
+
[context cost | 12,430 tokens] ← shields.io badge, linked to the methodology
|
|
162
|
+
```
|
|
79
163
|
|
|
80
164
|
## What it costs on Claude
|
|
81
165
|
|
|
@@ -94,9 +178,9 @@ the method is [Claude divergence](docs/METHODOLOGY.md#claude-divergence).
|
|
|
94
178
|
|
|
95
179
|
## Why trust the number?
|
|
96
180
|
|
|
97
|
-
Every
|
|
98
|
-
SHA-256 of its canonical bytes, the pinned tokenizer (`o200k_base`), and the
|
|
99
|
-
command. Disputes reduce to a byte-level diff:
|
|
181
|
+
Every published number is backed by a `measurement.json` containing the raw `tools/list`
|
|
182
|
+
capture, the SHA-256 of its canonical bytes, the pinned tokenizer (`o200k_base`), and the
|
|
183
|
+
exact launch command. Disputes reduce to a byte-level diff:
|
|
100
184
|
|
|
101
185
|
```bash
|
|
102
186
|
npx -y mcp-context-cost verify results/github/measurement.json
|
|
@@ -110,6 +194,11 @@ Add `--json` for scripting (`{ ok, serverName, rederivedTokens, rederivedSha, pr
|
|
|
110
194
|
on stdout, `badge` omitted on failure). Exit codes: `0` ok, `1` verification/measurement
|
|
111
195
|
failed, `2` usage error.
|
|
112
196
|
|
|
197
|
+
`audit` runs that same code path on your own machine and reports each server's
|
|
198
|
+
`canonicalSha256` in `--json`, so you can check that the version you installed is byte-identical
|
|
199
|
+
to the one that was published — which is exactly what `--claude` uses to decide whether it is
|
|
200
|
+
allowed to show you a number.
|
|
201
|
+
|
|
113
202
|
Full definition: [docs/METHODOLOGY.md](docs/METHODOLOGY.md) — what is counted, what the
|
|
114
203
|
number is *not*, config policy, failure taxonomy, frozen color bands, known divergences.
|
|
115
204
|
|
|
@@ -119,7 +208,7 @@ number is *not*, config policy, failure taxonomy, frozen color bands, known dive
|
|
|
119
208
|
|---|---|
|
|
120
209
|
| `src/core/` | the measurement spec, executable — canonical form, tokenizer, bands, badge JSON |
|
|
121
210
|
| `src/sweep/` | raw-wire MCP stdio client + Dockerized batch sweep + leaderboard/dashboard generators |
|
|
122
|
-
| `src/audit/` | client-config discovery (5 clients, JSONC-tolerant)
|
|
211
|
+
| `src/audit/` | client-config discovery (5 clients, JSONC-tolerant), the per-stack report, and the baseline diff |
|
|
123
212
|
| `src/cli.ts` | `audit` (measure your own stack), `verify` (re-derive any published number), `measure` |
|
|
124
213
|
| `spec/fixtures/` | golden vectors shared by the TypeScript and bash implementations |
|
|
125
214
|
| `tools/` | the one script that calls a network API (Claude divergence); kept out of the package so the library stays offline |
|
|
@@ -152,7 +241,7 @@ Or self-serve from CI via the (staged) mcp-tokens-action badge inputs — see
|
|
|
152
241
|
## Development
|
|
153
242
|
|
|
154
243
|
```bash
|
|
155
|
-
npm test #
|
|
244
|
+
npm test # 158 TS tests incl. golden fixtures + dispute drills
|
|
156
245
|
npx tsc --noEmit # typecheck
|
|
157
246
|
./upstream/tests/badge-test.sh # 21 bash tests — byte-identical to the TS reference
|
|
158
247
|
npm run sweep:all -- --docker # full curated sweep (Docker isolation)
|
|
@@ -165,7 +254,11 @@ color bands are frozen against the observed distribution of the first full sweep
|
|
|
165
254
|
|
|
166
255
|
## Status
|
|
167
256
|
|
|
168
|
-
Active.
|
|
257
|
+
Active. 56 of the 65 numbers come from a single sweep on 2026-08-16, 3 from 2026-08-17, and
|
|
258
|
+
6 from 2026-08-18 (an upstream `mcp` package bump broke the old low-level-`Server` API these
|
|
259
|
+
six relied on; pinning `mcp<2` in their launch commands fixed startup, not this project's code);
|
|
260
|
+
the weekly job currently re-measures one server (`memory`), so treat the leaderboard as a
|
|
261
|
+
dated snapshot rather than a live feed. Badge PRs are open across the ecosystem and
|
|
169
262
|
[sd2k/mcp-tokens-action#5](https://github.com/sd2k/mcp-tokens-action/pull/5) proposes the
|
|
170
263
|
self-serve badge path upstream. See [ROADMAP.md](ROADMAP.md) for what's next —
|
|
171
264
|
contributions welcome, especially new `servers.yaml` entries.
|
package/dist/audit/audit.d.ts
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
|
+
import { type DivergenceRun } from '../core/divergence.js';
|
|
1
2
|
import type { Measurement, MeasurementStatus, ToolMeasurement } from '../core/types.js';
|
|
2
3
|
import type { ConfiguredServer, LoadedConfig } from './config.js';
|
|
4
|
+
import { type AuditDiff, type IncreaseGate } from './diff.js';
|
|
3
5
|
export declare const DEFAULT_CONTEXT_WINDOW = 200000;
|
|
4
6
|
export type AuditStatus = MeasurementStatus | 'remote-not-measurable';
|
|
5
7
|
export interface AuditServerResult {
|
|
@@ -15,6 +17,13 @@ export interface AuditServerResult {
|
|
|
15
17
|
/** Names only — a server's env values never enter a report. */
|
|
16
18
|
envVarNames: string[];
|
|
17
19
|
canonicalSha256?: string | null;
|
|
20
|
+
/**
|
|
21
|
+
* Anthropic-request cost from the published Claude divergence run, only when
|
|
22
|
+
* its captured hash matches this install (`--claude`). `null` means the
|
|
23
|
+
* install doesn't match what was published — silence, not a stale guess.
|
|
24
|
+
* `undefined` means `--claude` wasn't requested at all.
|
|
25
|
+
*/
|
|
26
|
+
claudeTokens?: number | null;
|
|
18
27
|
notes?: string;
|
|
19
28
|
}
|
|
20
29
|
export interface HeaviestTool {
|
|
@@ -22,6 +31,17 @@ export interface HeaviestTool {
|
|
|
22
31
|
tool: string;
|
|
23
32
|
tokens: number;
|
|
24
33
|
}
|
|
34
|
+
/**
|
|
35
|
+
* What turning off the heaviest few tools would recover, for clients that let
|
|
36
|
+
* you disable individual tools rather than whole servers (Claude Code's
|
|
37
|
+
* per-tool permission rules, Cursor's per-tool toggles). `null` when there's
|
|
38
|
+
* nothing worth trimming (one tool total, or no measured tokens).
|
|
39
|
+
*/
|
|
40
|
+
export interface TrimAdvice {
|
|
41
|
+
tools: HeaviestTool[];
|
|
42
|
+
recoverableTokens: number;
|
|
43
|
+
recoverableShare: number;
|
|
44
|
+
}
|
|
25
45
|
export interface AuditConfigResult {
|
|
26
46
|
client: string;
|
|
27
47
|
source: string;
|
|
@@ -32,7 +52,37 @@ export interface AuditConfigResult {
|
|
|
32
52
|
servers: AuditServerResult[];
|
|
33
53
|
skipped: AuditServerResult[];
|
|
34
54
|
heaviestTools: HeaviestTool[];
|
|
55
|
+
trimAdvice: TrimAdvice | null;
|
|
56
|
+
}
|
|
57
|
+
export interface BudgetFitStep {
|
|
58
|
+
name: string;
|
|
59
|
+
tokens: number;
|
|
60
|
+
/** What the config still costs after removing this one and everything above it. */
|
|
61
|
+
remaining: number;
|
|
62
|
+
}
|
|
63
|
+
export interface BudgetFit {
|
|
64
|
+
/** How far over the limit the worst config starts. */
|
|
65
|
+
overBy: number;
|
|
66
|
+
/** Heaviest-first removals until the remainder fits. Empty if nothing can be removed. */
|
|
67
|
+
drop: BudgetFitStep[];
|
|
68
|
+
keptCount: number;
|
|
69
|
+
keptTokens: number;
|
|
70
|
+
/** False when removing every measured server still would not fit — a limit set too low. */
|
|
71
|
+
feasible: boolean;
|
|
35
72
|
}
|
|
73
|
+
/**
|
|
74
|
+
* The smallest heaviest-first set of servers that gets a config under its budget.
|
|
75
|
+
*
|
|
76
|
+
* `--budget` used to print "BUDGET FAIL: 84,455 > 20,000" and stop, which tells a reader
|
|
77
|
+
* they have a problem and nothing about the shape of it. The whole point of the audit
|
|
78
|
+
* surface is that the person running it is the person paying the tokens, and "you are over"
|
|
79
|
+
* is a measurement where "these two are why" is a decision.
|
|
80
|
+
*
|
|
81
|
+
* Heaviest-first is ONE ordering, not a recommendation: this cannot know which servers you
|
|
82
|
+
* need, and dropping by weight will sometimes name the one you cannot live without. That
|
|
83
|
+
* caveat is printed with the result rather than left implied.
|
|
84
|
+
*/
|
|
85
|
+
export declare function planBudgetFit(config: AuditConfigResult, limit: number): BudgetFit;
|
|
36
86
|
export interface AuditReport {
|
|
37
87
|
methodologyVersion: string;
|
|
38
88
|
encoding: 'o200k_base';
|
|
@@ -44,7 +94,18 @@ export interface AuditReport {
|
|
|
44
94
|
worstTotal: number;
|
|
45
95
|
worstSource: string;
|
|
46
96
|
over: boolean;
|
|
97
|
+
/** Present only when over budget: the arithmetic of getting back under it. */
|
|
98
|
+
fit?: BudgetFit;
|
|
99
|
+
};
|
|
100
|
+
/** Present only when a divergence run was supplied (`--claude`). */
|
|
101
|
+
claudeDivergence?: {
|
|
102
|
+
model: string;
|
|
103
|
+
measuredAt: string;
|
|
47
104
|
};
|
|
105
|
+
/** Present only when a baseline report was supplied (`--baseline`). */
|
|
106
|
+
diff?: AuditDiff;
|
|
107
|
+
/** Present only when `--max-increase` was supplied alongside a baseline. */
|
|
108
|
+
increaseGate?: IncreaseGate;
|
|
48
109
|
problems: string[];
|
|
49
110
|
}
|
|
50
111
|
/** Cache key for measurement reuse: the exact argv two configs would spawn. */
|
|
@@ -58,6 +119,8 @@ export declare function buildReport(configs: LoadedConfig[], measured: Map<strin
|
|
|
58
119
|
contextWindow?: number;
|
|
59
120
|
budget?: number;
|
|
60
121
|
generatedAt?: string;
|
|
122
|
+
/** Published `tools-delta/v1` run to join against (`--claude`); omit to skip the join. */
|
|
123
|
+
divergence?: DivergenceRun | null;
|
|
61
124
|
}): AuditReport;
|
|
62
125
|
/** Human output. JSON output is the report object itself. */
|
|
63
126
|
export declare function formatReport(report: AuditReport): string;
|
package/dist/audit/audit.js
CHANGED
|
@@ -13,7 +13,52 @@
|
|
|
13
13
|
* commands shared by two configs are still only measured once.
|
|
14
14
|
*/
|
|
15
15
|
import { METHODOLOGY_VERSION } from '../core/canonical.js';
|
|
16
|
+
import { isCurrent } from '../core/divergence.js';
|
|
17
|
+
import { formatDiff, formatGate } from './diff.js';
|
|
16
18
|
export const DEFAULT_CONTEXT_WINDOW = 200_000;
|
|
19
|
+
const TRIM_TOOL_COUNT = 3;
|
|
20
|
+
function buildTrimAdvice(sortedTools, totalTokens) {
|
|
21
|
+
if (totalTokens <= 0 || sortedTools.length < 2)
|
|
22
|
+
return null;
|
|
23
|
+
const trimmed = sortedTools.slice(0, TRIM_TOOL_COUNT);
|
|
24
|
+
const recoverableTokens = trimmed.reduce((a, t) => a + t.tokens, 0);
|
|
25
|
+
return { tools: trimmed, recoverableTokens, recoverableShare: recoverableTokens / totalTokens };
|
|
26
|
+
}
|
|
27
|
+
/**
|
|
28
|
+
* The smallest heaviest-first set of servers that gets a config under its budget.
|
|
29
|
+
*
|
|
30
|
+
* `--budget` used to print "BUDGET FAIL: 84,455 > 20,000" and stop, which tells a reader
|
|
31
|
+
* they have a problem and nothing about the shape of it. The whole point of the audit
|
|
32
|
+
* surface is that the person running it is the person paying the tokens, and "you are over"
|
|
33
|
+
* is a measurement where "these two are why" is a decision.
|
|
34
|
+
*
|
|
35
|
+
* Heaviest-first is ONE ordering, not a recommendation: this cannot know which servers you
|
|
36
|
+
* need, and dropping by weight will sometimes name the one you cannot live without. That
|
|
37
|
+
* caveat is printed with the result rather than left implied.
|
|
38
|
+
*/
|
|
39
|
+
export function planBudgetFit(config, limit) {
|
|
40
|
+
const measured = config.servers
|
|
41
|
+
.filter((srv) => typeof srv.tokens === 'number' && srv.tokens > 0)
|
|
42
|
+
.sort((a, b) => b.tokens - a.tokens);
|
|
43
|
+
const overBy = config.totalTokens - limit;
|
|
44
|
+
const drop = [];
|
|
45
|
+
let remaining = config.totalTokens;
|
|
46
|
+
for (const srv of measured) {
|
|
47
|
+
if (remaining <= limit)
|
|
48
|
+
break;
|
|
49
|
+
remaining -= srv.tokens;
|
|
50
|
+
drop.push({ name: srv.name, tokens: srv.tokens, remaining });
|
|
51
|
+
}
|
|
52
|
+
return {
|
|
53
|
+
overBy,
|
|
54
|
+
drop,
|
|
55
|
+
keptCount: measured.length - drop.length,
|
|
56
|
+
keptTokens: remaining,
|
|
57
|
+
// Removing everything measured still leaves unmeasured/base cost behind, so the
|
|
58
|
+
// honest test is whether the remainder actually landed under the limit.
|
|
59
|
+
feasible: remaining <= limit,
|
|
60
|
+
};
|
|
61
|
+
}
|
|
17
62
|
/** Cache key for measurement reuse: the exact argv two configs would spawn. */
|
|
18
63
|
export function serverKey(s) {
|
|
19
64
|
return JSON.stringify(s.argv ?? [s.url ?? s.name]);
|
|
@@ -73,6 +118,7 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
73
118
|
});
|
|
74
119
|
continue;
|
|
75
120
|
}
|
|
121
|
+
const divRow = opts.divergence?.servers[s.name];
|
|
76
122
|
ok.push({
|
|
77
123
|
...base,
|
|
78
124
|
status: m.status,
|
|
@@ -80,6 +126,7 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
80
126
|
toolCount: m.toolCount,
|
|
81
127
|
share: null, // filled once the total is known
|
|
82
128
|
canonicalSha256: m.canonicalSha256,
|
|
129
|
+
claudeTokens: opts.divergence ? (isCurrent(divRow, m.canonicalSha256 ?? null) ? divRow.claudeDelta : null) : undefined,
|
|
83
130
|
notes: m.status === 'dynamic' ? m.notes : undefined,
|
|
84
131
|
});
|
|
85
132
|
for (const t of m.tools)
|
|
@@ -105,6 +152,7 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
105
152
|
servers: ok,
|
|
106
153
|
skipped,
|
|
107
154
|
heaviestTools: tools.slice(0, 5),
|
|
155
|
+
trimAdvice: buildTrimAdvice(tools, totalTokens),
|
|
108
156
|
});
|
|
109
157
|
}
|
|
110
158
|
results.sort((a, b) => b.totalTokens - a.totalTokens);
|
|
@@ -116,15 +164,20 @@ export function buildReport(configs, measured, opts = {}) {
|
|
|
116
164
|
configs: results,
|
|
117
165
|
problems,
|
|
118
166
|
};
|
|
167
|
+
if (opts.divergence) {
|
|
168
|
+
report.claudeDivergence = { model: opts.divergence.model, measuredAt: opts.divergence.measuredAt };
|
|
169
|
+
}
|
|
119
170
|
if (typeof opts.budget === 'number') {
|
|
120
171
|
// The worst config is the gate: passing because your *lightest* client fits
|
|
121
172
|
// would be a green check on a session you don't run.
|
|
122
173
|
const worst = results[0];
|
|
174
|
+
const over = (worst?.totalTokens ?? 0) > opts.budget;
|
|
123
175
|
report.budget = {
|
|
124
176
|
limit: opts.budget,
|
|
125
177
|
worstTotal: worst?.totalTokens ?? 0,
|
|
126
178
|
worstSource: worst?.source ?? '(none)',
|
|
127
|
-
over
|
|
179
|
+
over,
|
|
180
|
+
fit: over && worst ? planBudgetFit(worst, opts.budget) : undefined,
|
|
128
181
|
};
|
|
129
182
|
}
|
|
130
183
|
return report;
|
|
@@ -135,6 +188,7 @@ const pct = (x) => `${(x * 100).toFixed(1)}%`;
|
|
|
135
188
|
export function formatReport(report) {
|
|
136
189
|
const lines = [];
|
|
137
190
|
lines.push(`mcp-context-cost audit · methodology ${report.methodologyVersion} · ${report.encoding} · context window ${n(report.contextWindow)}`);
|
|
191
|
+
const showClaude = !!report.claudeDivergence;
|
|
138
192
|
for (const cfg of report.configs) {
|
|
139
193
|
lines.push('');
|
|
140
194
|
lines.push(`${cfg.client} ${cfg.source}`);
|
|
@@ -143,18 +197,21 @@ export function formatReport(report) {
|
|
|
143
197
|
tools: s.toolCount === null ? '—' : String(s.toolCount),
|
|
144
198
|
tokens: s.tokens === null ? '—' : n(s.tokens),
|
|
145
199
|
share: s.share === null ? '—' : pct(s.share),
|
|
200
|
+
claude: s.claudeTokens == null ? '—' : n(s.claudeTokens),
|
|
146
201
|
}));
|
|
147
202
|
const w = {
|
|
148
203
|
name: Math.max(6, ...rows.map((r) => r.name.length), 'total'.length),
|
|
149
204
|
tools: Math.max(5, ...rows.map((r) => r.tools.length)),
|
|
150
205
|
tokens: Math.max(6, ...rows.map((r) => r.tokens.length), n(cfg.totalTokens).length),
|
|
206
|
+
claude: Math.max(6, ...rows.map((r) => r.claude.length)),
|
|
151
207
|
};
|
|
152
|
-
const line = (name, tools, tokens, share) => ` ${name.padEnd(w.name)} ${tools.padStart(w.tools)} ${tokens.padStart(w.tokens)} ${share.padStart(6)}
|
|
153
|
-
|
|
208
|
+
const line = (name, tools, tokens, share, claude) => ` ${name.padEnd(w.name)} ${tools.padStart(w.tools)} ${tokens.padStart(w.tokens)} ${share.padStart(6)}` +
|
|
209
|
+
(showClaude ? ` ${claude.padStart(w.claude)}` : '');
|
|
210
|
+
lines.push(line('server', 'tools', 'tokens', 'share', 'claude'));
|
|
154
211
|
for (const r of rows)
|
|
155
|
-
lines.push(line(r.name, r.tools, r.tokens, r.share));
|
|
156
|
-
lines.push(` ${'─'.repeat(w.name + w.tools + w.tokens + 14)}`);
|
|
157
|
-
lines.push(line('total', String(cfg.toolCount), n(cfg.totalTokens), ''));
|
|
212
|
+
lines.push(line(r.name, r.tools, r.tokens, r.share, r.claude));
|
|
213
|
+
lines.push(` ${'─'.repeat(w.name + w.tools + w.tokens + 14 + (showClaude ? w.claude + 2 : 0))}`);
|
|
214
|
+
lines.push(line('total', String(cfg.toolCount), n(cfg.totalTokens), '', ''));
|
|
158
215
|
lines.push('');
|
|
159
216
|
lines.push(` Every request in this client carries ${n(cfg.totalTokens)} tokens of tool schemas — ` +
|
|
160
217
|
`${pct(cfg.contextShare)} of a ${n(report.contextWindow)}-token context window, before you type anything.`);
|
|
@@ -166,6 +223,13 @@ export function formatReport(report) {
|
|
|
166
223
|
lines.push(` ${`${t.server} · ${t.tool}`.padEnd(tw)} ${n(t.tokens).padStart(7)}`);
|
|
167
224
|
}
|
|
168
225
|
}
|
|
226
|
+
if (cfg.trimAdvice) {
|
|
227
|
+
const names = cfg.trimAdvice.tools.map((t) => `${t.server}·${t.tool}`).join(', ');
|
|
228
|
+
lines.push('');
|
|
229
|
+
lines.push(` trim: disabling ${cfg.trimAdvice.tools.length} tool${cfg.trimAdvice.tools.length === 1 ? '' : 's'} ` +
|
|
230
|
+
`(${names}) would recover ${n(cfg.trimAdvice.recoverableTokens)} tokens ` +
|
|
231
|
+
`(${pct(cfg.trimAdvice.recoverableShare)} of this config) — if your client supports per-tool filtering.`);
|
|
232
|
+
}
|
|
169
233
|
if (cfg.skipped.length) {
|
|
170
234
|
lines.push('');
|
|
171
235
|
lines.push(' not measured');
|
|
@@ -175,6 +239,11 @@ export function formatReport(report) {
|
|
|
175
239
|
}
|
|
176
240
|
}
|
|
177
241
|
}
|
|
242
|
+
if (showClaude) {
|
|
243
|
+
lines.push('');
|
|
244
|
+
lines.push(` claude = Anthropic-request cost from the ${report.claudeDivergence.measuredAt} ${report.claudeDivergence.model} ` +
|
|
245
|
+
`divergence run, shown only where the published capture hash matches this install; '—' means no current match.`);
|
|
246
|
+
}
|
|
178
247
|
if (report.problems.length) {
|
|
179
248
|
lines.push('');
|
|
180
249
|
lines.push('problems');
|
|
@@ -182,10 +251,52 @@ export function formatReport(report) {
|
|
|
182
251
|
lines.push(` ${p}`);
|
|
183
252
|
}
|
|
184
253
|
if (report.budget) {
|
|
254
|
+
const b = report.budget;
|
|
255
|
+
lines.push('');
|
|
256
|
+
if (!b.over) {
|
|
257
|
+
const headroom = b.limit - b.worstTotal;
|
|
258
|
+
lines.push(`budget ok: ${n(b.worstTotal)} ≤ ${n(b.limit)} — ${n(headroom)} to spare`);
|
|
259
|
+
}
|
|
260
|
+
else {
|
|
261
|
+
lines.push(`BUDGET FAIL: ${n(b.worstTotal)} > ${n(b.limit)} (${b.worstSource})`);
|
|
262
|
+
const fit = b.fit;
|
|
263
|
+
if (fit && fit.drop.length) {
|
|
264
|
+
lines.push('');
|
|
265
|
+
lines.push(` over by ${n(fit.overBy)}. Heaviest-first, this is what gets you under:`);
|
|
266
|
+
for (const step of fit.drop) {
|
|
267
|
+
const verdict = step.remaining <= b.limit ? 'fits' : 'still over';
|
|
268
|
+
lines.push(` drop ${step.name.padEnd(22)} ${n(step.tokens).padStart(9)} → ${n(step.remaining).padStart(9)} ${verdict}`);
|
|
269
|
+
}
|
|
270
|
+
lines.push('');
|
|
271
|
+
if (fit.feasible && fit.keptCount === 0) {
|
|
272
|
+
// Arithmetically it fits, and the answer is useless: the only way under this
|
|
273
|
+
// limit is to run no servers at all. Saying "fits" here would be true and
|
|
274
|
+
// misleading, which is the pair this whole tool exists to keep apart.
|
|
275
|
+
lines.push(` no subset fits: every measured server would have to go. The limit is below`);
|
|
276
|
+
lines.push(` what any one of these servers costs.`);
|
|
277
|
+
}
|
|
278
|
+
else if (fit.feasible) {
|
|
279
|
+
const share = b.limit > 0 ? ` (${pct(fit.keptTokens / b.limit)} of budget)` : '';
|
|
280
|
+
lines.push(` keeps ${fit.keptCount} server(s) at ${n(fit.keptTokens)} tokens${share}`);
|
|
281
|
+
}
|
|
282
|
+
else {
|
|
283
|
+
lines.push(` even removing every measured server leaves ${n(fit.keptTokens)} — the limit is below this config's floor`);
|
|
284
|
+
}
|
|
285
|
+
lines.push('');
|
|
286
|
+
lines.push(' This is arithmetic, not advice: it cannot know which servers you need, and by');
|
|
287
|
+
lines.push(' weight alone it will sometimes name the one you cannot work without. Use --json');
|
|
288
|
+
lines.push(' for the full per-server list and pick your own order.');
|
|
289
|
+
}
|
|
290
|
+
else if (fit) {
|
|
291
|
+
lines.push(' nothing measured could be removed to get under the limit.');
|
|
292
|
+
}
|
|
293
|
+
}
|
|
294
|
+
}
|
|
295
|
+
if (report.diff)
|
|
296
|
+
lines.push(formatDiff(report.diff, report.contextWindow));
|
|
297
|
+
if (report.increaseGate) {
|
|
185
298
|
lines.push('');
|
|
186
|
-
lines.push(report.
|
|
187
|
-
? `BUDGET FAIL: ${n(report.budget.worstTotal)} > ${n(report.budget.limit)} (${report.budget.worstSource})`
|
|
188
|
-
: `budget ok: ${n(report.budget.worstTotal)} ≤ ${n(report.budget.limit)}`);
|
|
299
|
+
lines.push(formatGate(report.increaseGate));
|
|
189
300
|
}
|
|
190
301
|
lines.push('');
|
|
191
302
|
lines.push('These are wire tokens — what the server puts on the wire, counted with o200k_base. What your model is billed');
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* `audit --baseline <report.json>` — what a config change costs every future session.
|
|
3
|
+
*
|
|
4
|
+
* `audit` answers "what do my servers cost right now". That is a number a reader
|
|
5
|
+
* has to have an opinion about. A diff against a stored earlier report answers
|
|
6
|
+
* the question that needs no opinion at all: *this change adds 17,000 tokens to
|
|
7
|
+
* every request you will ever send from this client.* Same measurement path,
|
|
8
|
+
* same per-config discipline — a baseline is just an earlier `audit --json`.
|
|
9
|
+
*
|
|
10
|
+
* The trap this file exists to avoid: a server that measured fine before and
|
|
11
|
+
* fails to start now makes the total go DOWN. Subtracting two totals would
|
|
12
|
+
* report that as an improvement, which is the flattering reading and the true
|
|
13
|
+
* one having the same shape. So a server that changed measured-ness is never
|
|
14
|
+
* given a delta — it is named, its known side is printed, and the direction of
|
|
15
|
+
* the resulting error is stated ("understates by at least 9,246").
|
|
16
|
+
*/
|
|
17
|
+
import type { AuditConfigResult, AuditReport } from './audit.js';
|
|
18
|
+
export type ServerDeltaKind = 'added' | 'removed' | 'changed' | 'unchanged'
|
|
19
|
+
/** Measured in the baseline, not measurable now — the total understates. */
|
|
20
|
+
| 'unmeasured-now'
|
|
21
|
+
/** Not measurable in the baseline, measured now — the increase overstates. */
|
|
22
|
+
| 'unmeasured-before'
|
|
23
|
+
/** Present and unmeasured in both runs — contributes 0 to both totals, but hides cost. */
|
|
24
|
+
| 'unmeasured-both';
|
|
25
|
+
export interface ServerDelta {
|
|
26
|
+
name: string;
|
|
27
|
+
kind: ServerDeltaKind;
|
|
28
|
+
/** Baseline tokens; `null` when absent from the baseline or unmeasured in it. */
|
|
29
|
+
before: number | null;
|
|
30
|
+
/** Current tokens; `null` when gone from the config or unmeasured now. */
|
|
31
|
+
after: number | null;
|
|
32
|
+
/** Signed change. `null` whenever the two sides are not the same kind of number. */
|
|
33
|
+
delta: number | null;
|
|
34
|
+
/** Why a delta is missing, in a sentence a reader can act on. */
|
|
35
|
+
note?: string;
|
|
36
|
+
}
|
|
37
|
+
export interface ConfigDiff {
|
|
38
|
+
client: string;
|
|
39
|
+
source: string;
|
|
40
|
+
/** How this config was paired with a baseline config. */
|
|
41
|
+
matchedBy: 'source' | 'sole-config' | 'unmatched';
|
|
42
|
+
beforeTotal: number | null;
|
|
43
|
+
afterTotal: number;
|
|
44
|
+
/** afterTotal - beforeTotal, or `null` when there is no baseline to subtract. */
|
|
45
|
+
delta: number | null;
|
|
46
|
+
beforeShare: number | null;
|
|
47
|
+
afterShare: number;
|
|
48
|
+
/**
|
|
49
|
+
* True when `delta` is the exact change in measured cost. False when a server
|
|
50
|
+
* crossed the measured/unmeasured line, which moves the total for a reason
|
|
51
|
+
* that is not a config change.
|
|
52
|
+
*/
|
|
53
|
+
exact: boolean;
|
|
54
|
+
/** Tokens the diff is known to be missing, and which way it leans. */
|
|
55
|
+
understatedBy: number;
|
|
56
|
+
overstatedBy: number;
|
|
57
|
+
servers: ServerDelta[];
|
|
58
|
+
}
|
|
59
|
+
export interface AuditDiff {
|
|
60
|
+
baselineGeneratedAt: string;
|
|
61
|
+
baselineMethodologyVersion: string;
|
|
62
|
+
/** False when something makes the two reports incommensurable at all (methodology bump). */
|
|
63
|
+
comparable: boolean;
|
|
64
|
+
/** Baseline configs that no current config matched — never silently dropped. */
|
|
65
|
+
droppedConfigs: {
|
|
66
|
+
client: string;
|
|
67
|
+
source: string;
|
|
68
|
+
totalTokens: number;
|
|
69
|
+
}[];
|
|
70
|
+
warnings: string[];
|
|
71
|
+
configs: ConfigDiff[];
|
|
72
|
+
/**
|
|
73
|
+
* The largest per-config increase. Per config, never merged: a context window
|
|
74
|
+
* belongs to one client session, so a portfolio-wide "total delta" would
|
|
75
|
+
* describe a session nobody runs. `null` when nothing could be compared.
|
|
76
|
+
*/
|
|
77
|
+
worstIncrease: {
|
|
78
|
+
source: string;
|
|
79
|
+
delta: number;
|
|
80
|
+
} | null;
|
|
81
|
+
}
|
|
82
|
+
/** Parse and shape-check a stored report. A baseline that cannot be read is never "no change". */
|
|
83
|
+
export declare function parseBaselineReport(text: string): {
|
|
84
|
+
report: AuditReport | null;
|
|
85
|
+
problem?: string;
|
|
86
|
+
};
|
|
87
|
+
export declare function diffConfig(before: AuditConfigResult | null, after: AuditConfigResult, matchedBy: ConfigDiff['matchedBy']): ConfigDiff;
|
|
88
|
+
/**
|
|
89
|
+
* Pair current configs with baseline configs.
|
|
90
|
+
*
|
|
91
|
+
* Exact source path first. Then one deliberate fallback: if each side has
|
|
92
|
+
* exactly one config, they are the same config seen from two machines — the CI
|
|
93
|
+
* case, where a baseline recorded at /Users/… meets a checkout at /home/runner/….
|
|
94
|
+
* Anything looser would pair two unrelated clients and call the difference a
|
|
95
|
+
* change, so everything else stays unmatched and says so.
|
|
96
|
+
*/
|
|
97
|
+
export declare function pairConfigs(before: AuditConfigResult[], after: AuditConfigResult[]): {
|
|
98
|
+
pairs: {
|
|
99
|
+
before: AuditConfigResult | null;
|
|
100
|
+
after: AuditConfigResult;
|
|
101
|
+
matchedBy: ConfigDiff['matchedBy'];
|
|
102
|
+
}[];
|
|
103
|
+
dropped: AuditConfigResult[];
|
|
104
|
+
};
|
|
105
|
+
export declare function buildDiff(baseline: AuditReport, current: AuditReport): AuditDiff;
|
|
106
|
+
export declare function formatDiff(diff: AuditDiff, contextWindow: number): string;
|
|
107
|
+
export interface IncreaseGate {
|
|
108
|
+
limit: number;
|
|
109
|
+
pass: boolean;
|
|
110
|
+
/** The increase the gate measured, when it got far enough to measure one. */
|
|
111
|
+
increase: number | null;
|
|
112
|
+
reasons: string[];
|
|
113
|
+
}
|
|
114
|
+
/**
|
|
115
|
+
* `--max-increase N` — the CI gate. Fails on an increase over the limit, and
|
|
116
|
+
* equally on any reason the increase could not be established.
|
|
117
|
+
*
|
|
118
|
+
* That second half is the point. A gate that passes when a server failed to
|
|
119
|
+
* start, or when the baseline covered a config this run never found, is a green
|
|
120
|
+
* check on a question nobody asked. Everything this portfolio has learned says
|
|
121
|
+
* unchecked must not read as clean, so an inexact diff fails and names why.
|
|
122
|
+
*/
|
|
123
|
+
export declare function evaluateIncreaseGate(diff: AuditDiff, limit: number): IncreaseGate;
|
|
124
|
+
export declare function formatGate(gate: IncreaseGate): string;
|