mcp-context-cost 0.3.0 → 0.5.0

package/README.md

@@ -3,37 +3,12 @@
 [![npm](https://img.shields.io/npm/v/mcp-context-cost)](https://www.npmjs.com/package/mcp-context-cost)
 [![CI](https://github.com/athakur3/mcp-context-cost/actions/workflows/ci.yml/badge.svg)](https://github.com/athakur3/mcp-context-cost/actions/workflows/ci.yml)
 
-**Reproducible context-cost measurement for MCP servers — for the one you publish, and for
-the stack you actually run.**
+**What do the MCP servers in your config cost you before you type anything — and what did
+that last config change add to every session you will ever run?**
 
-Every MCP server you wire into an agent injects its tool schemas into the model's context
-before any work happens. That cost is invisible — and it varies by **1,700×** across
-popular servers:
-
-| server | context cost | tools |
-|---|---:|---:|
-| github (official) | **54,422 tokens** | 44 |
-| brave-search | 25,456 | 8 |
-| notion | 17,500 | 24 |
-| playwright *(4.8M installs/week)* | 4,024 | 24 |
-| filesystem (reference) | 2,823 | 14 |
-| markitdown | 64 | 1 |
-
-*(57 of 82 popular servers measured, 2026-08-16 sweep — full table in
-[results/leaderboard.md](results/leaderboard.md); every failure is listed with its reason.
-Each measured server also has a [detail page](https://athakur3.github.io/mcp-context-cost/servers/)
-showing which tools its tokens are in.)*
-
-This project makes that cost **legible and disputable**:
-
-```
-[context cost | 12,430 tokens]   ← shields.io badge, linked to the methodology
-```
-
-## What does *your* setup cost?
-
-The leaderboard measures one server at a time. You don't run one server — you run a stack.
-Point `audit` at your own MCP config and it measures every server you actually have installed:
+Every MCP server you wire into an agent injects its tool schemas into the model's context on
+every single request. You pay that whether or not the agent ends up using the tools, and no
+client shows you the number. Point `audit` at your own MCP config:
 
 ```bash
 npx -y mcp-context-cost audit
@@ -49,12 +24,18 @@ claude-desktop  ~/Library/Application Support/Claude/claude_desktop_config.json
   ────────────────────────────────────────────
   total                   37   7,901
 
-  Every request in this client carries 7,901 tokens of tool schemas — 4.0% of a
-  200,000-token context window, before you type anything.
+  7,901 tokens of tool schemas — 4.0% of a 200,000-token context window.
+  No default deferral is on record for claude-desktop, so every request
+  carries these tokens before you type anything — an absence of a record
+  about the client, not a measurement of it.
 
   heaviest tools
     sequential-thinking · sequentialthinking      990
     memory · search_nodes                         323
+
+  trim: disabling 3 tools (sequential-thinking·sequentialthinking, memory·search_nodes,
+  memory·open_nodes) would recover 1,635 tokens (20.7% of this config) — if your client
+  supports per-tool filtering.
 ```
 
 It finds configs for Claude Desktop, Claude Code (`~/.claude.json`, `.mcp.json`), Cursor,
@@ -67,6 +48,72 @@ Totals are reported per config file, never merged: a context window belongs to o
 session, so summing Cursor's servers into Claude Desktop's total would describe a session
 nobody runs.
 
+### Where this cost is paid in full, and where it is deferred away
+
+Not every client puts every tool definition in context on every request, so the total above
+is not automatically your bill. Which client reads the config, and how that client is
+configured **on this machine**, decides it — and `audit` reads that rather than assuming it.
+
+**Clients with no default deferral on record** — Claude Desktop, Cursor, VS Code, Windsurf.
+The total is what every request carries, as in the example above. That sentence is an
+absence of a record about those clients, not a measurement of them, and the report says so
+in those words.
+
+**Claude Code defers MCP tool definitions by default** (its **tool search**): they are not
+in context at session start, and load when the model reaches for one. Three variables move
+that, and `audit` reads all three — from the shell it runs in *and* from the `env` block of
+Claude Code's own settings files (managed, `.claude/settings.local.json`,
+`.claude/settings.json`, `~/.claude/settings.json`), because a machine that switched
+deferral off in a settings file is not a machine running the default:
+
+| setting | what the audit reports |
+|---|---|
+| nothing set (the default) | every definition deferred, at any size — no threshold applies |
+| `ENABLE_TOOL_SEARCH=true` | same: every definition deferred |
+| `ENABLE_TOOL_SEARCH=false` | deferral off — every request carries the full total |
+| `ENABLE_TOOL_SEARCH=auto` / `auto:N` | deferred only once definitions reach 10% / N% of the context window |
+| `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS` set | tool search off — read first, because `ENABLE_TOOL_SEARCH` cannot override it |
+| `ANTHROPIC_BASE_URL` off `api.anthropic.com` | falls back to loading up front — consulted only while `ENABLE_TOOL_SEARCH` is unset |
+| anything else in `ENABLE_TOOL_SEARCH` | not a documented value, so nothing is claimed from it |
+
+On a machine where none of them is set, the same stack reads:
+
+```
+  7,901 tokens of tool schemas — 4.0% of a 200,000-token context window.
+  claude-code defers every MCP tool definition (tool search), with no threshold —
+  ENABLE_TOOL_SEARCH is unset here, which is the documented default. These tokens are NOT loaded
+  up front at any size; they load when the model reaches for a tool. Size
+  decides nothing here, so none of the arithmetic above changes the answer.
+  Where this was read — Claude Code takes these variables from the shell it
+  starts in and from the env block of its own settings files:
+    this shell — sets none of them
+    4 other settings file(s) it reads are not on this machine
+  The full number is paid where deferral does not apply:
+    a Microsoft Foundry deployment hosted on Azure, which rejects tool search server-side
+    Google Cloud's Agent Platform on a model earlier than the Claude 4.5 generation
+    a model without support for tool_reference blocks (before Sonnet 4.5 / Haiku 4.5 / Opus 4.5)
+    a server pinned with "alwaysLoad": true, whose tools load at session start regardless
+```
+
+Set `ENABLE_TOOL_SEARCH=false` in that shell and the same config reports the opposite —
+`loads every tool definition up front here`, naming the variable and the place it was read
+from. Deferring is also not free: what a deferring client *does* load at session start —
+tool names plus the server's `instructions` — is measured per server and published in the
+leaderboard's `session start` column, and for at least one server in the published set it
+costs **more** than loading the definitions would.
+
+Three things the report will not do: it will not convert between units silently (in
+threshold mode the stack is compared as a range, because the audit counts wire bytes and the
+threshold is counted in what the client sends to the API — measured at 0.20×–1.92× across 20
+servers); it will not pick a winner when two places on the machine set the same variable to
+different values, or when a settings file exists and cannot be read; and it will not pass an
+absence of a record off as a measurement. The first two print as unanswered questions. The
+third prints as an answer that names itself: for the four discovered clients with no default
+on record — `claude-desktop`, `cursor`, `vscode`, `windsurf` — the tokens are counted as
+loaded up front, and the report says so in those words, "an absence of a record about the
+client, not a measurement of it".
+Full model, sources and dates: [METHODOLOGY §who pays the number](docs/METHODOLOGY.md#who-pays).
+
 **In CI**, make it a gate — the bundlesize move for agents:
 
 ```bash
@@ -74,8 +121,107 @@ npx -y mcp-context-cost audit --config .mcp.json --budget 20000
 # exits 1 when the stack exceeds the budget, so a PR adding a 25K-token server fails
 ```
 
-Flags: `--json` (full report on stdout, progress on stderr), `--budget N`, `--context N`
-(default 200,000), `--timeout ms`, `--concurrency N`, `--docker`.
+The budget is an absolute ceiling. What a reviewer actually wants to know is what *this pull
+request* did, so record a baseline and diff against it:
+
+```bash
+npx -y mcp-context-cost audit --config .mcp.json --json > baseline.json          # on main
+npx -y mcp-context-cost audit --config .mcp.json --baseline baseline.json --max-increase 2000
+```
+
+```
+diff vs baseline measured 2026-08-18T01:51:49.555Z (methodology 1.0)
+
+  .mcp.json
+    2,378  →  5,201   +2,823
+
+    added             filesystem          — →     2,823  +2,823
+    (1 server unchanged)
+
+    This change adds 2,823 tokens to every request in this client — 1.2% → 2.6% of a
+    200,000-token context window.
+
+INCREASE FAIL:
+  .mcp.json: +2,823 tokens per request, over the 2,000 allowed
+```
+
+> **Version note.** `--baseline` and `--max-increase` shipped in **0.4.0** (published
+> 2026-08-18, and the current `latest`), so the command above gates on
+> `npx -y mcp-context-cost@latest`. Pinning to **0.3.0 or earlier** does not gate, and fails
+> quietly: those builds ignore flags they do not recognise, so the same command produces a
+> plain audit and **exit 0** — a passing CI check on a gate that never ran. 0.4.0 rejects
+> unknown flags with exit 2 instead. Pin at or above 0.4.0, or do not pin.
+
+A baseline is just a stored `audit --json` report, so any artifact store works. Without
+`--max-increase` the diff is informational and the exit code is unchanged.
+
+[`examples/github-actions.yml`](examples/github-actions.yml) is the whole thing as a workflow:
+measure the base branch, measure the PR, fail on the difference.
+
+`--max-increase` fails on more than the number — it also fails whenever the increase could
+not be established. A server that measured yesterday and won't start today takes its tokens
+out of the total in exactly the way uninstalling it would, and reporting that as a saving is
+the one mistake this tool must not make. So a server that crossed the measured/unmeasured
+line, a config with no baseline, or a baseline config this run never found each fail the
+gate and name themselves:
+
+```
+    Not a clean comparison: a server changed measured-ness between the two runs.
+    The measured total moved −2,378, but that is not what your config did.
+
+      memory: measured 2,378 in the baseline and could not be measured now — its cost is
+              missing from the total, not gone from your config
+      → true cost is at least 2,378 higher than the 0 measured now.
+
+INCREASE FAIL:
+  .mcp.json: a server changed measured-ness, so the change could not be established exactly
+```
+
+Add `--claude` to annotate each server with its Anthropic-request cost from the published
+[Claude divergence](docs/METHODOLOGY.md#claude-divergence) run — an exact number when the
+published capture hash matches what you have installed, `—` (silence, not a stale guess)
+when it doesn't (today the run covers the top 20 measured servers, so most installs will
+show a mix):
+
+```
+  server               tools  tokens   share   claude
+  github                  44  54,422   95.8%   18,406
+  memory                   9   2,378    4.2%       —
+```
+
+Flags: `--json` (full report on stdout, progress on stderr), `--budget N`,
+`--baseline <report.json>`, `--max-increase N`, `--context N` (default 200,000),
+`--timeout ms`, `--concurrency N`, `--docker`, `--claude`.
+
+## Where the numbers come from
+
+The number `audit` gives you is the same measurement, run across a curated set of public
+servers — which is how you can tell it is a measurement and not this tool's opinion. It also
+shows what you are choosing between: across the 69 servers measured, cost spans **1,700×**,
+from the 32-token `postgres` reference server to github's 54,422. The table below starts at
+markitdown's 64 tokens, an 850× spread; the full range is in
+[results/leaderboard.md](results/leaderboard.md).
+
+| server | context cost | tools |
+|---|---:|---:|
+| github (official) | **54,422 tokens** | 44 |
+| brave-search | 25,456 | 8 |
+| notion | 17,500 | 24 |
+| playwright *(4.8M installs/week)* | 4,024 | 24 |
+| filesystem (reference) | 2,823 | 14 |
+| markitdown | 64 | 1 |
+
+*(69 of 82 popular servers measured, sweeps of 2026-08-18 and 2026-08-19 — full table in
+[results/leaderboard.md](results/leaderboard.md); every failure is listed with its reason.
+Each measured server also has a [detail page](https://athakur3.github.io/mcp-context-cost/servers/)
+showing which tools its tokens are in.)*
+
+If you publish a server, the same measurement is available as a badge, so your users can see
+the cost before they install rather than after:
+
+```
+[context cost | 12,430 tokens]   ← shields.io badge, linked to the methodology
+```
 
 ## What it costs on Claude
 
@@ -94,9 +240,9 @@ the method is [Claude divergence](docs/METHODOLOGY.md#claude-divergence).
 
 ## Why trust the number?
 
-Every badge is backed by a `measurement.json` containing the raw `tools/list` capture, the
-SHA-256 of its canonical bytes, the pinned tokenizer (`o200k_base`), and the exact launch
-command. Disputes reduce to a byte-level diff:
+Every published number is backed by a `measurement.json` containing the raw `tools/list`
+capture, the SHA-256 of its canonical bytes, the pinned tokenizer (`o200k_base`), and the
+exact launch command. Disputes reduce to a byte-level diff:
 
 ```bash
 npx -y mcp-context-cost verify results/github/measurement.json
@@ -110,6 +256,11 @@ Add `--json` for scripting (`{ ok, serverName, rederivedTokens, rederivedSha, pr
 on stdout, `badge` omitted on failure). Exit codes: `0` ok, `1` verification/measurement
 failed, `2` usage error.
 
+`audit` runs that same code path on your own machine and reports each server's
+`canonicalSha256` in `--json`, so you can check that the version you installed is byte-identical
+to the one that was published — which is exactly what `--claude` uses to decide whether it is
+allowed to show you a number.
+
 Full definition: [docs/METHODOLOGY.md](docs/METHODOLOGY.md) — what is counted, what the
 number is *not*, config policy, failure taxonomy, frozen color bands, known divergences.
 
@@ -119,7 +270,7 @@ number is *not*, config policy, failure taxonomy, frozen color bands, known dive
 |---|---|
 | `src/core/` | the measurement spec, executable — canonical form, tokenizer, bands, badge JSON |
 | `src/sweep/` | raw-wire MCP stdio client + Dockerized batch sweep + leaderboard/dashboard generators |
-| `src/audit/` | client-config discovery (5 clients, JSONC-tolerant) + the per-stack report |
+| `src/audit/` | client-config discovery (5 clients, JSONC-tolerant), the per-stack report, and the baseline diff |
 | `src/cli.ts` | `audit` (measure your own stack), `verify` (re-derive any published number), `measure` |
 | `spec/fixtures/` | golden vectors shared by the TypeScript and bash implementations |
 | `tools/` | the one script that calls a network API (Claude divergence); kept out of the package so the library stays offline |
@@ -146,28 +297,36 @@ Point the link at the measurement behind the number — for servers in this swee
 `https://athakur3.github.io/mcp-context-cost/servers/<name>.html`; otherwise the
 methodology page. A badge nobody can audit is decoration.
 
+How many projects outside this repository actually display it is a dated reading rather
+than a guess — [docs/adoption.md](docs/adoption.md), regenerated by `npm run adoption`,
+which publishes the queries it ran and every file it examined. A zero there means the
+search ran and found none; if it could not run, it says that instead of publishing a zero.
+
 Or self-serve from CI via the (staged) mcp-tokens-action badge inputs — see
 [upstream/action-patch.md](upstream/action-patch.md).
 
 ## Development
 
 ```bash
-npm test                        # 105 TS tests incl. golden fixtures + dispute drills
+npm test                        # 403 TS tests incl. golden fixtures + dispute drills
 npx tsc --noEmit                # typecheck
 ./upstream/tests/badge-test.sh  # 21 bash tests — byte-identical to the TS reference
 npm run sweep:all -- --docker   # full curated sweep (Docker isolation)
 ```
 
-Notable engineering choices: the MCP client is a deliberate ~150-line raw-wire
+Notable engineering choices: the MCP client is a deliberate ~220-line raw-wire
 implementation (SDK schema-parsing can reorder keys, which would corrupt canonical bytes);
 sweep servers run in credential-free Docker containers with recorded isolation; the badge
 color bands are frozen against the observed distribution of the first full sweep.
 
 ## Status
 
-Active. The leaderboard refreshes weekly; badge PRs are open across the ecosystem and
-[sd2k/mcp-tokens-action#5](https://github.com/sd2k/mcp-tokens-action/pull/5) proposes the
-self-serve badge path upstream. See [ROADMAP.md](ROADMAP.md) for what's next —
+Active. 46 of the 69 numbers come from the sweep of 2026-08-19 and 23 from 2026-08-18. Two
+weekly jobs re-measure the set — one reference server (`memory`) every Monday, and a rotating
+sixth of the list every Wednesday, so the full set turns over in six weeks — but treat the
+leaderboard as a dated snapshot rather than a live feed. Badge PRs are open across the
+ecosystem and [sd2k/mcp-tokens-action#5](https://github.com/sd2k/mcp-tokens-action/pull/5)
+proposes the self-serve badge path upstream. See [ROADMAP.md](ROADMAP.md) for what's next —
 contributions welcome, especially new `servers.yaml` entries.
 
 MIT © 2026

package/dist/audit/audit.d.ts

@@ -1,5 +1,8 @@
+import { type DivergenceRun } from '../core/divergence.js';
 import type { Measurement, MeasurementStatus, ToolMeasurement } from '../core/types.js';
 import type { ConfiguredServer, LoadedConfig } from './config.js';
+import { type DeferralVerdict, type ToolSearchEnv, type ToolSearchSource } from './deferral.js';
+import { type AuditDiff, type IncreaseGate } from './diff.js';
 export declare const DEFAULT_CONTEXT_WINDOW = 200000;
 export type AuditStatus = MeasurementStatus | 'remote-not-measurable';
 export interface AuditServerResult {
@@ -15,6 +18,13 @@ export interface AuditServerResult {
     /** Names only — a server's env values never enter a report. */
     envVarNames: string[];
     canonicalSha256?: string | null;
+    /**
+     * Anthropic-request cost from the published Claude divergence run, only when
+     * its captured hash matches this install (`--claude`). `null` means the
+     * install doesn't match what was published — silence, not a stale guess.
+     * `undefined` means `--claude` wasn't requested at all.
+     */
+    claudeTokens?: number | null;
     notes?: string;
 }
 export interface HeaviestTool {
@@ -22,6 +32,17 @@ export interface HeaviestTool {
     tool: string;
     tokens: number;
 }
+/**
+ * What turning off the heaviest few tools would recover, for clients that let
+ * you disable individual tools rather than whole servers (Claude Code's
+ * per-tool permission rules, Cursor's per-tool toggles). `null` when there's
+ * nothing worth trimming (one tool total, or no measured tokens).
+ */
+export interface TrimAdvice {
+    tools: HeaviestTool[];
+    recoverableTokens: number;
+    recoverableShare: number;
+}
 export interface AuditConfigResult {
     client: string;
     source: string;
@@ -32,7 +53,47 @@ export interface AuditConfigResult {
     servers: AuditServerResult[];
     skipped: AuditServerResult[];
     heaviestTools: HeaviestTool[];
+    trimAdvice: TrimAdvice | null;
+    /**
+     * Whether this client loads the total up front or defers it, and — when the
+     * client decides that by a threshold — which side of it this stack is on.
+     * Every config carries one: the answer "no deferral is on record for this
+     * client" is a reading, not a gap.
+     *
+     * Configs that one session loads together share a single verdict object, so
+     * `deferral.sources` can name more files than this config's own `source`.
+     */
+    deferral: DeferralVerdict;
+}
+export interface BudgetFitStep {
+    name: string;
+    tokens: number;
+    /** What the config still costs after removing this one and everything above it. */
+    remaining: number;
 }
+export interface BudgetFit {
+    /** How far over the limit the worst config starts. */
+    overBy: number;
+    /** Heaviest-first removals until the remainder fits. Empty if nothing can be removed. */
+    drop: BudgetFitStep[];
+    keptCount: number;
+    keptTokens: number;
+    /** False when removing every measured server still would not fit — a limit set too low. */
+    feasible: boolean;
+}
+/**
+ * The smallest heaviest-first set of servers that gets a config under its budget.
+ *
+ * `--budget` used to print "BUDGET FAIL: 84,455 > 20,000" and stop, which tells a reader
+ * they have a problem and nothing about the shape of it. The whole point of the audit
+ * surface is that the person running it is the person paying the tokens, and "you are over"
+ * is a measurement where "these two are why" is a decision.
+ *
+ * Heaviest-first is ONE ordering, not a recommendation: this cannot know which servers you
+ * need, and dropping by weight will sometimes name the one you cannot live without. That
+ * caveat is printed with the result rather than left implied.
+ */
+export declare function planBudgetFit(config: AuditConfigResult, limit: number): BudgetFit;
 export interface AuditReport {
     methodologyVersion: string;
     encoding: 'o200k_base';
@@ -44,11 +105,35 @@ export interface AuditReport {
         worstTotal: number;
         worstSource: string;
         over: boolean;
+        /** Present only when over budget: the arithmetic of getting back under it. */
+        fit?: BudgetFit;
     };
+    /** Present only when a divergence run was supplied (`--claude`). */
+    claudeDivergence?: {
+        model: string;
+        measuredAt: string;
+    };
+    /** Present only when a baseline report was supplied (`--baseline`). */
+    diff?: AuditDiff;
+    /** Present only when `--max-increase` was supplied alongside a baseline. */
+    increaseGate?: IncreaseGate;
     problems: string[];
 }
 /** Cache key for measurement reuse: the exact argv two configs would spawn. */
 export declare function serverKey(s: ConfiguredServer): string;
+/**
+ * The measurement keys that stand for more than one distinct server.
+ *
+ * `serverKey` is the argv alone, so two entries running the same command under
+ * different environments are measured once and both are given that one number.
+ * Environment decides what a server serves — `GITHUB_TOOLSETS` on
+ * `github-mcp-server` selects which toolsets it lists — so for entries under one
+ * of these keys, the number reported is one entry's, not each one's.
+ *
+ * Same argv AND same environment is not collapsed: two clients pointing at an
+ * identical server are one measurement, which is the reuse this key is for.
+ */
+export declare function collapsedKeys(configs: LoadedConfig[]): Set<string>;
 /**
  * Assemble the report from configs + measurements. Pure: `runAudit` does the
  * spawning, this does the arithmetic, so totals and shares are testable without
@@ -58,6 +143,22 @@ export declare function buildReport(configs: LoadedConfig[], measured: Map<strin
     contextWindow?: number;
     budget?: number;
     generatedAt?: string;
+    /** Published `tools-delta/v1` run to join against (`--claude`); omit to skip the join. */
+    divergence?: DivergenceRun | null;
+    /**
+     * The audited machine's SHELL tool-search variables. Passed in rather than
+     * read here so this stays pure and a report is reproducible from its
+     * inputs; `runAudit` supplies the real environment. Omitted means the shell
+     * set nothing.
+     */
+    env?: ToolSearchEnv;
+    /**
+     * The other place those variables come from: Claude Code's own settings
+     * files, highest precedence first, as `loadSettingsSources` read them.
+     * `runAudit` supplies these. Omitted means they were not read here — which
+     * the report says, rather than reporting a default it did not establish.
+     */
+    settings?: ToolSearchSource[];
 }): AuditReport;
 /** Human output. JSON output is the report object itself. */
 export declare function formatReport(report: AuditReport): string;