@booyaka/mcp-vet 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/BENCHMARK.md +112 -0
- package/CHANGELOG.md +42 -0
- package/README.md +63 -6
- package/dist/cli.js +19 -0
- package/dist/conformance.d.ts +24 -0
- package/dist/conformance.js +310 -0
- package/dist/index.d.ts +2 -0
- package/dist/index.js +4 -1
- package/dist/python/mcp_ast_scan.py +58 -3
- package/dist/reporters.js +1 -0
- package/dist/rules.js +10 -0
- package/dist/ts-analyzer.js +68 -3
- package/dist/types.d.ts +7 -0
- package/package.json +2 -1
package/BENCHMARK.md
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# Benchmark: corpus, methodology, and honest limits
|
|
2
|
+
|
|
3
|
+
The README claims high precision on real MCP code. This file is the evidence
|
|
4
|
+
behind that claim — corpus, pinned commits, counts, labels, and what the
|
|
5
|
+
scanner is *known to miss* — so the claim is checkable rather than vibes.
|
|
6
|
+
|
|
7
|
+
> Prompted by community feedback on the launch post: *"'0 false positives' is
|
|
8
|
+
> encouraging but incomplete without corpus size, commit SHAs, labeled
|
|
9
|
+
> negatives, and recall."* Correct. Here they are.
|
|
10
|
+
|
|
11
|
+
## Corpus (pinned)
|
|
12
|
+
|
|
13
|
+
Scanned with `mcp-vet` v0.4.0 (`node dist/cli.js <roots> --json`), all rules
|
|
14
|
+
enabled, default confidence (`low`), on 2026-07-23:
|
|
15
|
+
|
|
16
|
+
| Repo | Commit | Scanned root | Files | LOC |
|
|
17
|
+
| --- | --- | --- | --- | --- |
|
|
18
|
+
| [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) | `d31124c982401739917fd817c2a59db344529c16` | `src/` | 78 | 14,742 |
|
|
19
|
+
| [modelcontextprotocol/typescript-sdk](https://github.com/modelcontextprotocol/typescript-sdk) | `1e1392e3f91583884fe82a0b4b91335875c3fba6` | `examples/` | 144 | 17,224 |
|
|
20
|
+
| [modelcontextprotocol/python-sdk](https://github.com/modelcontextprotocol/python-sdk) | `3a6f2996cdd8358957479791e8b26198c07d6a75` | `examples/` | 225 | 12,013 |
|
|
21
|
+
| **Total** | | | **447** | **43,979** |
|
|
22
|
+
|
|
23
|
+
File counts are candidate files (`.ts/.tsx/.js/.mjs/.cjs/.py`) under the
|
|
24
|
+
scanned roots, excluding `node_modules`.
|
|
25
|
+
|
|
26
|
+
## Results
|
|
27
|
+
|
|
28
|
+
**105 findings across 41 files** (TypeScript/JavaScript: 93, Python: 12).
|
|
29
|
+
By confidence: 66 high, 38 medium, 1 low.
|
|
30
|
+
|
|
31
|
+
| Pattern | Findings |
|
|
32
|
+
| --- | --- |
|
|
33
|
+
| `MCP_SESSION_ID` | 49 |
|
|
34
|
+
| `LOGGING_CAP` | 17 |
|
|
35
|
+
| `SAMPLING_CAP` | 16 |
|
|
36
|
+
| `ROOTS_CAP` | 15 |
|
|
37
|
+
| `INITIALIZE_HANDLER` | 4 |
|
|
38
|
+
| `TASKS_LEGACY` | 2 |
|
|
39
|
+
| `TASKS_RESULT_REMOVED` | 2 |
|
|
40
|
+
|
|
41
|
+
### Labeling
|
|
42
|
+
|
|
43
|
+
Every finding was manually reviewed against its source line:
|
|
44
|
+
|
|
45
|
+
- **104 / 105 true positives** — real references to a removed or deprecated
|
|
46
|
+
protocol surface (session headers/ids, handshake registration, legacy task
|
|
47
|
+
methods, deprecated capability declarations and method strings).
|
|
48
|
+
- **1 / 105 false positive (0.95%)** —
|
|
49
|
+
`stories/json_response/client.py:62` in the typescript-sdk examples:
|
|
50
|
+
`assert "mcp-session-id" not in response.headers`. That line is
|
|
51
|
+
*already-migrated* test code asserting the header is **absent**; flagging it
|
|
52
|
+
as "will break" is wrong. It is exactly what inline suppression
|
|
53
|
+
(`# mcp-vet-disable-line MCP_SESSION_ID`) is for, but we count it as a false
|
|
54
|
+
positive rather than defining it away. So the honest headline is
|
|
55
|
+
**"1 false positive in 44k LOC"**, not zero.
|
|
56
|
+
|
|
57
|
+
Notes on reading the numbers:
|
|
58
|
+
|
|
59
|
+
- Two occurrences on one line (e.g. `transport.sessionId && sessions.delete(transport.sessionId)`)
|
|
60
|
+
are reported as two findings — column-level dedup, not line-level.
|
|
61
|
+
- Findings in test files (`__tests__/…`) are counted as true positives: a test
|
|
62
|
+
that registers `sampling/createMessage` breaks the same way production code
|
|
63
|
+
does.
|
|
64
|
+
|
|
65
|
+
### Labeled negatives
|
|
66
|
+
|
|
67
|
+
Files asserted to stay **clean** are part of the repo's test suite and run in CI:
|
|
68
|
+
|
|
69
|
+
- `test/fixtures/clean/` — a full server written in the 2026-07-28 style
|
|
70
|
+
(per-request `_meta`, `sessionIdGenerator: undefined`, `-32602`).
|
|
71
|
+
- `test/fixtures/negatives/` — "false friend" patterns: `sessionId` on plain
|
|
72
|
+
app-level objects, `-32002` inside strings/comments, capability-like words
|
|
73
|
+
with no capabilities context.
|
|
74
|
+
- `test/fixtures/adversarial/caught/` — obfuscations the scanner **must**
|
|
75
|
+
catch: aliased imports (TS + Python), namespace-qualified SDK constants,
|
|
76
|
+
client transports resuming a `sessionId`.
|
|
77
|
+
|
|
78
|
+
Additionally, in the corpus above, comment-only mentions (e.g. `Mcp-Session-Id`
|
|
79
|
+
in a comment, `initialize` in prose) produced zero findings — the AST layer
|
|
80
|
+
distinguishes executable tokens from comments by construction.
|
|
81
|
+
|
|
82
|
+
## Recall — what the scanner is known to miss
|
|
83
|
+
|
|
84
|
+
Static token analysis proves known patterns are **absent**; it cannot prove
|
|
85
|
+
your server **speaks the new wire contract**. Recall is bounded by
|
|
86
|
+
construction, and the misses are locked into the test suite
|
|
87
|
+
(`test/fixtures/adversarial/missed/`, asserted to produce zero findings so any
|
|
88
|
+
silent claim-inflation fails CI):
|
|
89
|
+
|
|
90
|
+
- split/computed method strings — `'tasks' + '/list'`, `` `tasks/${op}` ``, f-strings
|
|
91
|
+
- computed capability keys — `{ ['roo'+'ts']: {} }`
|
|
92
|
+
- generated/loop-driven registration from string fragments
|
|
93
|
+
- framework-adapter indirection (route tables built at runtime)
|
|
94
|
+
- cross-module renames — a wrapper re-exporting an SDK constant under a new
|
|
95
|
+
name is flagged in the wrapper file, but a consumer importing only the new
|
|
96
|
+
name scans clean on its own
|
|
97
|
+
|
|
98
|
+
There is no corpus-wide recall *percentage*: that would require a labeled set
|
|
99
|
+
of every legacy usage in the wild, which nobody has. What we can say is: for
|
|
100
|
+
the pattern shapes listed in the README, detection is exact; for the shapes
|
|
101
|
+
above, it is zero, and the tool says so — pair the scan with runtime checks
|
|
102
|
+
(`mcp-vet fixtures`) to cover the difference.
|
|
103
|
+
|
|
104
|
+
## Reproducing
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
git clone --depth 1 https://github.com/modelcontextprotocol/servers
|
|
108
|
+
git clone --depth 1 https://github.com/modelcontextprotocol/typescript-sdk
|
|
109
|
+
git clone --depth 1 https://github.com/modelcontextprotocol/python-sdk
|
|
110
|
+
# check out the pinned SHAs above, then:
|
|
111
|
+
npx @booyaka/mcp-vet servers/src typescript-sdk/examples python-sdk/examples --json --no-files
|
|
112
|
+
```
|
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,48 @@ All notable changes to `mcp-vet` are documented here. The format is based on
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres
|
|
5
5
|
to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## [0.4.0]
|
|
8
|
+
|
|
9
|
+
The community-feedback release — everything in it traces to reader comments on
|
|
10
|
+
the launch post (issues #1–#6). Static analysis got sharper, and the tool now
|
|
11
|
+
ships the runtime half it was honest about not covering.
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **Client-side session-ownership detection** (#1) — a client transport
|
|
16
|
+
constructed with a real `sessionId`/`session_id` and reads of
|
|
17
|
+
`transport.sessionId` are flagged (`MCP_SESSION_ID`, medium). The migrated
|
|
18
|
+
`sessionId: undefined` / `session_id=None` forms are recognized as benign.
|
|
19
|
+
Servers going stateless is only half the migration; clients that still behave
|
|
20
|
+
as if they own a session break too.
|
|
21
|
+
- **Aliased-import resolution** (#2) — `import { InitializeRequestSchema as Init }`
|
|
22
|
+
(TS) and `from mcp.types import RootsCapability as RC` (Python) now flag both
|
|
23
|
+
the import line and the aliased usage sites. Python import lines surface
|
|
24
|
+
imported names even when only the alias is used later.
|
|
25
|
+
- **Adversarial regression suite** (#3) — `test/fixtures/adversarial/` locks in
|
|
26
|
+
what the scanner catches (`caught/`) *and* what it is known to miss
|
|
27
|
+
(`missed/`, asserted zero findings): computed strings, computed capability
|
|
28
|
+
keys, generated registration, framework adapters, cross-module renames.
|
|
29
|
+
- **`mcp-vet fixtures [dir]`** (#4) — emits nine protocol-level conformance
|
|
30
|
+
fixtures + `CHECKLIST.md`: `server/discover`, per-request `_meta`,
|
|
31
|
+
`Mcp-Method`/`Mcp-Name` routing headers (incl. mismatch rejection), stateless
|
|
32
|
+
auth, task-handle lifecycle, duplicate deliveries, retry on another instance,
|
|
33
|
+
`tools/list` cache invalidation, and downgrade/refusal behavior. Also exported
|
|
34
|
+
programmatically (`CONFORMANCE_FIXTURES`, `emitConformanceFixtures`).
|
|
35
|
+
- **BENCHMARK.md** (#5) — the precision claim is now evidence: pinned corpus
|
|
36
|
+
SHAs, 447 files / ~44k LOC, every finding labeled (105 findings, 104 TP,
|
|
37
|
+
1 FP), labeled negatives, and an explicit recall discussion.
|
|
38
|
+
|
|
39
|
+
### Changed
|
|
40
|
+
|
|
41
|
+
- **Docs: spec-date semantics** (#6) — July 28 is a specification release, not
|
|
42
|
+
a remote kill switch; breakage appears when a client/server pair negotiates
|
|
43
|
+
the new revision. README and the post-scan notice now say so, and recommend
|
|
44
|
+
the dual-version (2025-11-25 + 2026-07-28) rollout test matrix.
|
|
45
|
+
- The README "0 false positives" claim is replaced by the measured, reproducible
|
|
46
|
+
numbers in BENCHMARK.md (1 FP in 44k LOC — an already-migrated negative
|
|
47
|
+
assertion in test code).
|
|
48
|
+
|
|
7
49
|
## [0.3.0]
|
|
8
50
|
|
|
9
51
|
The completeness release — full detection coverage, a real migration path, and a
|
package/README.md
CHANGED
|
@@ -15,11 +15,22 @@ npx @booyaka/mcp-vet .
|
|
|
15
15
|
```
|
|
16
16
|
|
|
17
17
|
<p align="center">
|
|
18
|
-
<img src="https://raw.githubusercontent.com/Booyaka101/mcp-vet/main/assets/demo.
|
|
18
|
+
<img src="https://raw.githubusercontent.com/Booyaka101/mcp-vet/main/assets/demo.png" alt="mcp-vet scanning a server — BREAKING and DEPRECATED findings with before/after fixes and confidence tags" width="720">
|
|
19
19
|
</p>
|
|
20
20
|
|
|
21
21
|
No account, no API key, no network calls — it parses your code locally (ts-morph for TS/JS, a bundled Python `ast` script for `.py`) and exits non-zero if it finds anything **BREAKING**, so you can drop it straight into CI.
|
|
22
22
|
|
|
23
|
+
## What actually happens on July 28
|
|
24
|
+
|
|
25
|
+
**July 28 is a specification release date, not a switch that remotely disables your deployment.** Nothing reaches into running servers and turns them off. Breakage appears when a **client and server pair negotiates or requires the new revision** — a client that sends `2026-07-28`-style requests (per-request `_meta`, no handshake, routing headers) against a server that still expects `2025-11-25` semantics, or vice versa.
|
|
26
|
+
|
|
27
|
+
Two practical consequences:
|
|
28
|
+
|
|
29
|
+
- **Your rollout is a window, not a day.** Until every client you care about has moved, keep **both** revisions in your production test matrix: a `2025-11-25` path and a `2026-07-28` path. `mcp-vet fixtures` emits wire-level test fixtures for exactly this (see [Runtime conformance fixtures](#runtime-conformance-fixtures)).
|
|
30
|
+
- **Silent acceptance is the worst failure mode.** A server that quietly processes an old-revision request under new semantics (or the reverse) corrupts behavior instead of failing loudly. Verify *refusal* behavior, not just the happy path.
|
|
31
|
+
|
|
32
|
+
The scan tells you *what to change in your source*; the date tells you *when clients start expecting it*.
|
|
33
|
+
|
|
23
34
|
---
|
|
24
35
|
|
|
25
36
|
## Real-world example
|
|
@@ -43,7 +54,7 @@ sse-polling.ts:107:13 BREAKING MCP_SESSION_ID [medium]
|
|
|
43
54
|
6 finding(s): 5 BREAKING, 1 DEPRECATED
|
|
44
55
|
```
|
|
45
56
|
|
|
46
|
-
Note it catches the `sessionIdGenerator` session usage — the real signal in SDK-based servers, which usually never write the literal `Mcp-Session-Id` string. And it stays quiet where it should: the `Mcp-Session-Id` mentioned in a *comment*, the `initialize` in a comment in `dual-era.ts`, and the `sampling/createMessage` in `sampling.ts` (which appears only in comments and behind the `requestSampling()` helper) are all left alone. That precision — structural AST checks, not text matching — is what keeps the noise down on a real codebase: **6 findings, 0 false positives.**
|
|
57
|
+
Note it catches the `sessionIdGenerator` session usage — the real signal in SDK-based servers, which usually never write the literal `Mcp-Session-Id` string. And it stays quiet where it should: the `Mcp-Session-Id` mentioned in a *comment*, the `initialize` in a comment in `dual-era.ts`, and the `sampling/createMessage` in `sampling.ts` (which appears only in comments and behind the `requestSampling()` helper) are all left alone. That precision — structural AST checks, not text matching — is what keeps the noise down on a real codebase: **6 findings, 0 false positives on these files.** (Across the full labeled corpus it's 104/105 true positives — see [BENCHMARK.md](./BENCHMARK.md).)
|
|
47
58
|
|
|
48
59
|
---
|
|
49
60
|
|
|
@@ -53,7 +64,7 @@ Note it catches the `sessionIdGenerator` session usage — the real signal in SD
|
|
|
53
64
|
|
|
54
65
|
| ID | Pattern |
|
|
55
66
|
| --- | --- |
|
|
56
|
-
| `MCP_SESSION_ID` | `Mcp-Session-Id` header / `mcpSessionId` variable |
|
|
67
|
+
| `MCP_SESSION_ID` | `Mcp-Session-Id` header / `mcpSessionId` variable / client-side session ownership (`sessionId` passed to or read from a client transport) |
|
|
57
68
|
| `INITIALIZE_HANDLER` | `initialize` / `notifications/initialized` handler registration |
|
|
58
69
|
| `ERROR_CODE_32002` | the numeric error code `-32002` |
|
|
59
70
|
| `TASKS_LEGACY` | `tasks/get` · `tasks/update` · `tasks/cancel` legacy method strings |
|
|
@@ -73,7 +84,7 @@ Note it catches the `sessionIdGenerator` session usage — the real signal in SD
|
|
|
73
84
|
Every finding carries a **confidence** so you can tune signal-to-noise with `--min-confidence`:
|
|
74
85
|
|
|
75
86
|
- **high** — exact/deterministic match (session id, `-32002`, tasks methods), a structurally-verified capability (the `roots`/`sampling`/`logging` key is really *inside* a `capabilities` object), or an `initialize` string used as a method name (handler registration, `switch` case, or `req.method === 'initialize'`).
|
|
76
|
-
- **medium** — a `roots`/`sampling`/`logging` key/string within 5 lines of a `capabilities` mention but not structurally verified.
|
|
87
|
+
- **medium** — a `roots`/`sampling`/`logging` key/string within 5 lines of a `capabilities` mention but not structurally verified; a real `sessionIdGenerator`; client-side session ownership (`sessionId`/`session_id` passed to or read from a transport/client).
|
|
77
88
|
- **low** — a bare `'initialize'` string with no registration context.
|
|
78
89
|
|
|
79
90
|
---
|
|
@@ -96,6 +107,19 @@ function handle(req) {
|
|
|
96
107
|
}
|
|
97
108
|
```
|
|
98
109
|
|
|
110
|
+
This cuts both ways — **client-side session ownership breaks too**, even against a server that scans clean. A lot of tool-reliability bugs only show up when the server is stateless but the client still behaves as if it owns a session:
|
|
111
|
+
|
|
112
|
+
```ts
|
|
113
|
+
// ❌ before — the client resumes a stored session
|
|
114
|
+
const transport = new StreamableHTTPClientTransport(url, { sessionId: stored });
|
|
115
|
+
persist(transport.sessionId);
|
|
116
|
+
|
|
117
|
+
// ✅ after — stateless: no stored session id, full _meta on every request
|
|
118
|
+
const transport = new StreamableHTTPClientTransport(url, { sessionId: undefined });
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
`mcp-vet` flags a client transport constructed with a real `sessionId`/`session_id` and reads of `transport.sessionId` (medium confidence). The migrated `sessionId: undefined` / `session_id=None` forms are recognized and left alone.
|
|
122
|
+
|
|
99
123
|
### 2. `initialize` / `notifications/initialized` — the handshake is removed
|
|
100
124
|
|
|
101
125
|
> *"The `initialize`/`initialized` handshake is removed. The protocol version, client info, and client capabilities that used to be exchanged once at connection time now travel in `_meta` on every request."*
|
|
@@ -167,6 +191,28 @@ case 'tasks/list': return listTasks();
|
|
|
167
191
|
|
|
168
192
|
The CLI prints a one-line reminder of these after every scan.
|
|
169
193
|
|
|
194
|
+
## Runtime conformance fixtures
|
|
195
|
+
|
|
196
|
+
Static analysis proves known legacy patterns are *absent* from your source. Only wire-level tests prove your running server actually *speaks* the 2026-07-28 contract. `mcp-vet` ships both halves:
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
npx @booyaka/mcp-vet fixtures ./mcp-fixtures
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
writes nine ready-to-fire JSON fixtures plus a `CHECKLIST.md`, covering the runtime behaviors a linter cannot see:
|
|
203
|
+
|
|
204
|
+
1. `server/discover` replaces the initialize handshake
|
|
205
|
+
2. per-request `_meta` (protocolVersion, clientInfo, capabilities) — including explicit refusal when `_meta` is missing
|
|
206
|
+
3. `Mcp-Method` / `Mcp-Name` routing headers, including the header/body-mismatch rejection case
|
|
207
|
+
4. stateless auth context (no session-bound token cache)
|
|
208
|
+
5. task-handle lifecycle: creation, `tasks/get` polling, resume on another instance, `tasks/list` and `tasks/result` returning method-not-found
|
|
209
|
+
6. duplicate request delivery (idempotency under retries)
|
|
210
|
+
7. retry against a different server instance (no sticky in-memory state)
|
|
211
|
+
8. `tools/list` cache invalidation
|
|
212
|
+
9. downgrade/refusal: old-revision requests get an explicit error, never silent acceptance under the wrong semantics
|
|
213
|
+
|
|
214
|
+
Each fixture is a plain JSON description (`send` headers + JSON-RPC body, `expect` notes) you can replay with curl, supertest, pytest + httpx, or any HTTP harness. The checklist also spells out the **dual-version rollout matrix** — run both `2025-11-25` and `2026-07-28` paths until your clients have all moved — and a **client-side assumptions** list (session resume, per-request `_meta`, retries landing on other instances, `tools/list` revalidation).
|
|
215
|
+
|
|
170
216
|
## Usage
|
|
171
217
|
|
|
172
218
|
```bash
|
|
@@ -174,6 +220,7 @@ npx @booyaka/mcp-vet [paths...] # scan directories and/or files (default:
|
|
|
174
220
|
npx @booyaka/mcp-vet . --fix # scan, and auto-apply the mechanical -32002 → -32602 rewrite
|
|
175
221
|
npx @booyaka/mcp-vet ./src ./packages # multiple roots
|
|
176
222
|
npx @booyaka/mcp-vet server.py # a single file
|
|
223
|
+
npx @booyaka/mcp-vet fixtures ./dir # write runtime conformance fixtures + checklist (default: ./mcp-vet-fixtures)
|
|
177
224
|
```
|
|
178
225
|
|
|
179
226
|
Globs `**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}` and `**/*.py`, skipping `node_modules`, `.git`, `__pycache__`, `dist`, and `build`.
|
|
@@ -325,15 +372,25 @@ It matches the ways real servers are actually written, not just raw method strin
|
|
|
325
372
|
- **SDK schema-constant registration** — `server.setRequestHandler(InitializeRequestSchema, …)` (how the official SDKs register handlers) maps `InitializeRequestSchema`, `ListRootsRequestSchema`, `CreateMessageRequestSchema`, `SetLevelRequestSchema`, `ListTasksRequestSchema`, `GetTaskResultRequestSchema`, … to the right rule.
|
|
326
373
|
- **SDK capability constructors** — the Python SDK's `ClientCapabilities(roots=RootsCapability())` is recognized structurally (high confidence), and `RootsCapability` / `SamplingCapability` / `LoggingCapability` are matched directly.
|
|
327
374
|
- **`sessionIdGenerator`** — flagged only when it's a real generator, not the migrated `sessionIdGenerator: undefined`.
|
|
375
|
+
- **aliased imports** — `import { InitializeRequestSchema as Init }` (TS) and `from mcp.types import RootsCapability as RC` (Python) are resolved back to their canonical names, so both the import line and the aliased usage sites are flagged. Namespace access (`types.InitializeRequestSchema`) is matched too.
|
|
376
|
+
- **client-side session ownership** — a client transport constructed with a real `sessionId`/`session_id`, or a read of `transport.sessionId`; the migrated `sessionId: undefined` / `session_id=None` forms are recognized as benign.
|
|
328
377
|
|
|
329
|
-
|
|
378
|
+
**Measured, not vibes:** scanned against the official MCP reference servers and both SDK example suites at pinned commits — 447 files / ~44k LOC — every finding manually labeled: **105 findings, 104 true positives, 1 false positive**. Corpus, commit SHAs, per-pattern counts, labeled negatives, and the recall discussion are in [BENCHMARK.md](./BENCHMARK.md).
|
|
330
379
|
|
|
331
380
|
### Known limitations
|
|
332
381
|
|
|
382
|
+
These are locked into the test suite as `test/fixtures/adversarial/missed/` — fixtures asserted to produce **zero** findings, so the claims below can't silently rot in either direction:
|
|
383
|
+
|
|
384
|
+
- **Split/computed method strings** — `"tasks" + "/list"`, `` `tasks/${op}` ``, or `f"tasks/{x}"` are not reconstructed.
|
|
385
|
+
- **Computed capability keys** — `{ ['roo'+'ts']: {} }` never exists as a single token.
|
|
386
|
+
- **Generated/loop-driven registration** — method tables assembled from string fragments at runtime.
|
|
387
|
+
- **Framework-adapter indirection** — routes built dynamically (`app.post('/rpc/' + ns + '/' + action, ...)`).
|
|
388
|
+
- **Cross-module renames** — a wrapper module re-exporting an SDK constant under a new name is flagged *in the wrapper file*, but a consumer importing only the new name scans clean on its own. Scan whole projects, not single files.
|
|
333
389
|
- **Python SDK decorator/method registration** — a handler wired purely as `@server.list_roots()` or a bare `session.list_roots()` call (with no capability declaration or method string in the file) is not matched, to avoid false positives on generic method names. The capability declaration in the same server is normally caught.
|
|
334
|
-
- **Split/computed method strings** — `"tasks" + "/list"` or `f"tasks/{x}"` are not reconstructed.
|
|
335
390
|
- The **regex fallback** (no Python interpreter) covers only the deterministic rules at reduced precision; install Python for full `.py` fidelity.
|
|
336
391
|
|
|
392
|
+
This is the recall boundary of static analysis: it proves known patterns are *absent*, not that the server *speaks the new wire contract*. Cover the difference with the [runtime conformance fixtures](#runtime-conformance-fixtures).
|
|
393
|
+
|
|
337
394
|
## Programmatic API
|
|
338
395
|
|
|
339
396
|
The scanner is usable as a library (typed) as well as a CLI — for editor extensions, custom CI steps, or migration harnesses:
|
package/dist/cli.js
CHANGED
|
@@ -44,6 +44,7 @@ const types_1 = require("./types");
|
|
|
44
44
|
const constants_1 = require("./constants");
|
|
45
45
|
const reporters_1 = require("./reporters");
|
|
46
46
|
const autofix_1 = require("./autofix");
|
|
47
|
+
const conformance_1 = require("./conformance");
|
|
47
48
|
const CONF_VALUES = ['high', 'medium', 'low'];
|
|
48
49
|
const FAILON_VALUES = ['breaking', 'any', 'none'];
|
|
49
50
|
function fail(msg) {
|
|
@@ -79,6 +80,23 @@ function readIgnoreFile(dir) {
|
|
|
79
80
|
return [];
|
|
80
81
|
}
|
|
81
82
|
}
|
|
83
|
+
// `mcp-vet fixtures [dir]` — emit the protocol-level conformance fixtures that
|
|
84
|
+
// pair with the static scan (runtime wire-contract checks a linter can't make).
|
|
85
|
+
// Dispatched before commander so the scan pipeline stays untouched; a directory
|
|
86
|
+
// literally named "fixtures" can still be scanned as `mcp-vet ./fixtures`.
|
|
87
|
+
if (process.argv[2] === 'fixtures') {
|
|
88
|
+
const rawDir = process.argv[3];
|
|
89
|
+
const dir = path.resolve(process.cwd(), rawDir && !rawDir.startsWith('-') ? rawDir : 'mcp-vet-fixtures');
|
|
90
|
+
try {
|
|
91
|
+
const res = (0, conformance_1.emitConformanceFixtures)(dir);
|
|
92
|
+
console.log(`mcp-vet: wrote ${res.files.length} conformance file(s) to ${res.dir}`);
|
|
93
|
+
console.log('Replay them against your running server and work through CHECKLIST.md — including the dual-version (2025-11-25 + 2026-07-28) rollout matrix.');
|
|
94
|
+
process.exit(0);
|
|
95
|
+
}
|
|
96
|
+
catch (err) {
|
|
97
|
+
fail(`could not write fixtures: ${err.message}`);
|
|
98
|
+
}
|
|
99
|
+
}
|
|
82
100
|
const program = new commander_1.Command();
|
|
83
101
|
program
|
|
84
102
|
.name('mcp-vet')
|
|
@@ -106,6 +124,7 @@ program
|
|
|
106
124
|
.option('--no-color', 'disable colored output')
|
|
107
125
|
.option('--quiet', 'suppress the human-readable terminal report')
|
|
108
126
|
.version((0, constants_1.getVersion)(), '-v, --version')
|
|
127
|
+
.addHelpText('after', '\nCommands:\n fixtures [dir] write protocol-level conformance fixtures + CHECKLIST.md (default: ./mcp-vet-fixtures)')
|
|
109
128
|
.showHelpAfterError();
|
|
110
129
|
program.parse(process.argv);
|
|
111
130
|
const opts = program.opts();
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
export interface ConformanceStep {
|
|
2
|
+
/** what to send — a JSON-RPC body, plus any required HTTP headers */
|
|
3
|
+
send: {
|
|
4
|
+
headers?: Record<string, string>;
|
|
5
|
+
body: unknown;
|
|
6
|
+
};
|
|
7
|
+
/** what a conformant 2026-07-28 server must do with it */
|
|
8
|
+
expect: string;
|
|
9
|
+
}
|
|
10
|
+
export interface ConformanceFixture {
|
|
11
|
+
/** file stem, e.g. "01-discover" */
|
|
12
|
+
id: string;
|
|
13
|
+
title: string;
|
|
14
|
+
/** why this fixture exists — the failure mode it exposes */
|
|
15
|
+
description: string;
|
|
16
|
+
steps: ConformanceStep[];
|
|
17
|
+
}
|
|
18
|
+
export declare const CONFORMANCE_FIXTURES: ConformanceFixture[];
|
|
19
|
+
export interface EmitResult {
|
|
20
|
+
dir: string;
|
|
21
|
+
files: string[];
|
|
22
|
+
}
|
|
23
|
+
/** Write all conformance fixtures plus CHECKLIST.md into `dir`. */
|
|
24
|
+
export declare function emitConformanceFixtures(dir: string): EmitResult;
|
|
@@ -0,0 +1,310 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
|
|
3
|
+
if (k2 === undefined) k2 = k;
|
|
4
|
+
var desc = Object.getOwnPropertyDescriptor(m, k);
|
|
5
|
+
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
|
|
6
|
+
desc = { enumerable: true, get: function() { return m[k]; } };
|
|
7
|
+
}
|
|
8
|
+
Object.defineProperty(o, k2, desc);
|
|
9
|
+
}) : (function(o, m, k, k2) {
|
|
10
|
+
if (k2 === undefined) k2 = k;
|
|
11
|
+
o[k2] = m[k];
|
|
12
|
+
}));
|
|
13
|
+
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
|
|
14
|
+
Object.defineProperty(o, "default", { enumerable: true, value: v });
|
|
15
|
+
}) : function(o, v) {
|
|
16
|
+
o["default"] = v;
|
|
17
|
+
});
|
|
18
|
+
var __importStar = (this && this.__importStar) || (function () {
|
|
19
|
+
var ownKeys = function(o) {
|
|
20
|
+
ownKeys = Object.getOwnPropertyNames || function (o) {
|
|
21
|
+
var ar = [];
|
|
22
|
+
for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
|
|
23
|
+
return ar;
|
|
24
|
+
};
|
|
25
|
+
return ownKeys(o);
|
|
26
|
+
};
|
|
27
|
+
return function (mod) {
|
|
28
|
+
if (mod && mod.__esModule) return mod;
|
|
29
|
+
var result = {};
|
|
30
|
+
if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
|
|
31
|
+
__setModuleDefault(result, mod);
|
|
32
|
+
return result;
|
|
33
|
+
};
|
|
34
|
+
})();
|
|
35
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
36
|
+
exports.CONFORMANCE_FIXTURES = void 0;
|
|
37
|
+
exports.emitConformanceFixtures = emitConformanceFixtures;
|
|
38
|
+
const fs = __importStar(require("node:fs"));
|
|
39
|
+
const path = __importStar(require("node:path"));
|
|
40
|
+
const constants_1 = require("./constants");
|
|
41
|
+
/**
|
|
42
|
+
* Protocol-level conformance fixtures — the runtime companion to the static
|
|
43
|
+
* scan. Static analysis proves known legacy patterns are *absent*; only
|
|
44
|
+
* wire-level tests prove the server actually *speaks* the 2026-07-28 contract.
|
|
45
|
+
* `mcp-vet fixtures <dir>` emits these as ready-to-fire JSON files plus a
|
|
46
|
+
* checklist, so a migration harness (curl, supertest, pytest, ...) can replay
|
|
47
|
+
* them against a real server during rollout.
|
|
48
|
+
*/
|
|
49
|
+
const NEW_REV = '2026-07-28';
|
|
50
|
+
const OLD_REV = '2025-11-25';
|
|
51
|
+
/** The per-request `_meta` that replaces the removed initialize handshake. */
|
|
52
|
+
const META = {
|
|
53
|
+
protocolVersion: NEW_REV,
|
|
54
|
+
clientInfo: { name: 'mcp-vet-conformance', version: '1.0.0' },
|
|
55
|
+
capabilities: { tools: {} },
|
|
56
|
+
};
|
|
57
|
+
const rpc = (id, method, params = {}) => ({
|
|
58
|
+
jsonrpc: '2.0',
|
|
59
|
+
id,
|
|
60
|
+
method,
|
|
61
|
+
params: { ...params, _meta: META },
|
|
62
|
+
});
|
|
63
|
+
const routingHeaders = (method, name) => ({
|
|
64
|
+
'Content-Type': 'application/json',
|
|
65
|
+
'Mcp-Method': method,
|
|
66
|
+
...(name ? { 'Mcp-Name': name } : {}),
|
|
67
|
+
});
|
|
68
|
+
exports.CONFORMANCE_FIXTURES = [
|
|
69
|
+
{
|
|
70
|
+
id: '01-discover',
|
|
71
|
+
title: 'server/discover replaces the initialize handshake',
|
|
72
|
+
description: 'A fresh connection must be able to discover server info and capabilities via server/discover with no prior handshake of any kind.',
|
|
73
|
+
steps: [
|
|
74
|
+
{
|
|
75
|
+
send: { headers: routingHeaders('server/discover'), body: rpc(1, 'server/discover') },
|
|
76
|
+
expect: 'HTTP 200 with a result carrying serverInfo, capabilities, and protocolVersion "' +
|
|
77
|
+
NEW_REV +
|
|
78
|
+
'". Any "session not initialized" style error is a conformance failure.',
|
|
79
|
+
},
|
|
80
|
+
],
|
|
81
|
+
},
|
|
82
|
+
{
|
|
83
|
+
id: '02-per-request-meta',
|
|
84
|
+
title: 'every request carries and honors _meta',
|
|
85
|
+
description: 'protocolVersion, clientInfo, and capabilities travel in _meta on every request. A cold request (no prior traffic) must succeed on _meta alone.',
|
|
86
|
+
steps: [
|
|
87
|
+
{
|
|
88
|
+
send: { headers: routingHeaders('tools/list'), body: rpc(1, 'tools/list') },
|
|
89
|
+
expect: 'HTTP 200 with the tool list. The server must not require any earlier request.',
|
|
90
|
+
},
|
|
91
|
+
{
|
|
92
|
+
send: {
|
|
93
|
+
headers: routingHeaders('tools/list'),
|
|
94
|
+
body: { jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} },
|
|
95
|
+
},
|
|
96
|
+
expect: 'Request WITHOUT _meta: the server must refuse explicitly (JSON-RPC error, e.g. -32602) — not silently assume a protocol revision.',
|
|
97
|
+
},
|
|
98
|
+
],
|
|
99
|
+
},
|
|
100
|
+
{
|
|
101
|
+
id: '03-http-routing-headers',
|
|
102
|
+
title: 'Mcp-Method / Mcp-Name headers must mirror the body',
|
|
103
|
+
description: 'Streamable HTTP requires routing headers that mirror the JSON-RPC body; a mismatch must be rejected, not routed by either half.',
|
|
104
|
+
steps: [
|
|
105
|
+
{
|
|
106
|
+
send: {
|
|
107
|
+
headers: routingHeaders('tools/call', 'echo'),
|
|
108
|
+
body: rpc(1, 'tools/call', { name: 'echo', arguments: {} }),
|
|
109
|
+
},
|
|
110
|
+
expect: 'HTTP 200 — headers and body agree.',
|
|
111
|
+
},
|
|
112
|
+
{
|
|
113
|
+
send: {
|
|
114
|
+
headers: routingHeaders('tools/list'),
|
|
115
|
+
body: rpc(2, 'tools/call', { name: 'echo', arguments: {} }),
|
|
116
|
+
},
|
|
117
|
+
expect: 'HTTP 4xx rejection — the Mcp-Method header says tools/list but the body says tools/call. Accepting either interpretation is a conformance failure.',
|
|
118
|
+
},
|
|
119
|
+
],
|
|
120
|
+
},
|
|
121
|
+
{
|
|
122
|
+
id: '04-stateless-auth',
|
|
123
|
+
title: 'auth context is derived per-request, not per-session',
|
|
124
|
+
description: 'With sessions gone there is nowhere to cache an auth handshake. Every request must be authorized from its own credentials.',
|
|
125
|
+
steps: [
|
|
126
|
+
{
|
|
127
|
+
send: {
|
|
128
|
+
headers: { ...routingHeaders('tools/list'), Authorization: 'Bearer <token>' },
|
|
129
|
+
body: rpc(1, 'tools/list'),
|
|
130
|
+
},
|
|
131
|
+
expect: 'HTTP 200. The same request against a server instance that has never seen this client before must behave identically (no session-bound token cache).',
|
|
132
|
+
},
|
|
133
|
+
{
|
|
134
|
+
send: { headers: routingHeaders('tools/list'), body: rpc(2, 'tools/list') },
|
|
135
|
+
expect: 'If the server requires auth: HTTP 401/403 on EVERY unauthenticated request — not just the first one of a "session".',
|
|
136
|
+
},
|
|
137
|
+
],
|
|
138
|
+
},
|
|
139
|
+
{
|
|
140
|
+
id: '05-task-handle-lifecycle',
|
|
141
|
+
title: 'task handles: explicit creation and resume',
|
|
142
|
+
description: 'tools/call returns a task handle; the client drives it with tasks/get / tasks/update / tasks/cancel using the new argument shapes. tasks/list and tasks/result no longer exist.',
|
|
143
|
+
steps: [
|
|
144
|
+
{
|
|
145
|
+
send: {
|
|
146
|
+
headers: routingHeaders('tools/call', 'long-running'),
|
|
147
|
+
body: rpc(1, 'tools/call', { name: 'long-running', arguments: {} }),
|
|
148
|
+
},
|
|
149
|
+
expect: 'Result contains a task handle (task.taskId per the ' + NEW_REV + ' schema).',
|
|
150
|
+
},
|
|
151
|
+
{
|
|
152
|
+
send: {
|
|
153
|
+
headers: routingHeaders('tasks/get'),
|
|
154
|
+
body: rpc(2, 'tasks/get', { taskId: '<handle-from-step-1>' }),
|
|
155
|
+
},
|
|
156
|
+
expect: 'Task status (and result once terminal) — including when this request lands on a DIFFERENT server instance than step 1. Poll tasks/get; there is no blocking tasks/result.',
|
|
157
|
+
},
|
|
158
|
+
{
|
|
159
|
+
send: { headers: routingHeaders('tasks/list'), body: rpc(3, 'tasks/list') },
|
|
160
|
+
expect: 'JSON-RPC method-not-found (-32601). tasks/list is removed.',
|
|
161
|
+
},
|
|
162
|
+
{
|
|
163
|
+
send: {
|
|
164
|
+
headers: routingHeaders('tasks/result'),
|
|
165
|
+
body: rpc(4, 'tasks/result', { taskId: '<handle-from-step-1>' }),
|
|
166
|
+
},
|
|
167
|
+
expect: 'JSON-RPC method-not-found (-32601). tasks/result is removed (SEP-2663).',
|
|
168
|
+
},
|
|
169
|
+
],
|
|
170
|
+
},
|
|
171
|
+
{
|
|
172
|
+
id: '06-duplicate-requests',
|
|
173
|
+
title: 'duplicate request delivery is safe',
|
|
174
|
+
description: 'Stateless HTTP means retried deliveries happen. Sending the identical request twice must not corrupt state or fail on the second delivery.',
|
|
175
|
+
steps: [
|
|
176
|
+
{
|
|
177
|
+
send: {
|
|
178
|
+
headers: routingHeaders('tools/call', 'echo'),
|
|
179
|
+
body: rpc(1, 'tools/call', { name: 'echo', arguments: { value: 'dup' } }),
|
|
180
|
+
},
|
|
181
|
+
expect: 'HTTP 200.',
|
|
182
|
+
},
|
|
183
|
+
{
|
|
184
|
+
send: {
|
|
185
|
+
headers: routingHeaders('tools/call', 'echo'),
|
|
186
|
+
body: rpc(1, 'tools/call', { name: 'echo', arguments: { value: 'dup' } }),
|
|
187
|
+
},
|
|
188
|
+
expect: 'Same request, same id, delivered again: a conformant server handles it without "already initialized" / duplicate-session errors.',
|
|
189
|
+
},
|
|
190
|
+
],
|
|
191
|
+
},
|
|
192
|
+
{
|
|
193
|
+
id: '07-retry-other-instance',
|
|
194
|
+
title: 'retry against a different server instance',
|
|
195
|
+
description: 'Run the same sequence against two separate instances (or restart the server between steps). Nothing may depend on in-memory per-client state.',
|
|
196
|
+
steps: [
|
|
197
|
+
{
|
|
198
|
+
send: { headers: routingHeaders('tools/list'), body: rpc(1, 'tools/list') },
|
|
199
|
+
expect: 'Send to instance A: HTTP 200.',
|
|
200
|
+
},
|
|
201
|
+
{
|
|
202
|
+
send: { headers: routingHeaders('tools/list'), body: rpc(2, 'tools/list') },
|
|
203
|
+
expect: 'Send to instance B (no prior traffic from this client): identical behavior. Divergence means hidden session state survived the migration.',
|
|
204
|
+
},
|
|
205
|
+
],
|
|
206
|
+
},
|
|
207
|
+
{
|
|
208
|
+
id: '08-tools-list-cache-invalidation',
|
|
209
|
+
title: 'tools/list cache invalidation',
|
|
210
|
+
description: 'Without a session there is no notifications channel to piggyback list_changed on outside of an active request. Clients must revalidate; servers must not serve a stale list after tool changes.',
|
|
211
|
+
steps: [
|
|
212
|
+
{
|
|
213
|
+
send: { headers: routingHeaders('tools/list'), body: rpc(1, 'tools/list') },
|
|
214
|
+
expect: 'Baseline tool list.',
|
|
215
|
+
},
|
|
216
|
+
{
|
|
217
|
+
send: { headers: routingHeaders('tools/list'), body: rpc(2, 'tools/list') },
|
|
218
|
+
expect: 'After the server adds/removes a tool (harness step): the new list. If the deployment caches tool lists, verify the cache is invalidated on change.',
|
|
219
|
+
},
|
|
220
|
+
],
|
|
221
|
+
},
|
|
222
|
+
{
|
|
223
|
+
id: '09-downgrade-refusal',
|
|
224
|
+
title: 'old-revision requests are refused, not silently accepted',
|
|
225
|
+
description: 'A request declaring protocolVersion ' +
|
|
226
|
+
OLD_REV +
|
|
227
|
+
' (or arriving in the old handshake style) must get an explicit refusal — processing it under new semantics silently is the worst failure mode.',
|
|
228
|
+
steps: [
|
|
229
|
+
{
|
|
230
|
+
send: {
|
|
231
|
+
headers: routingHeaders('tools/list'),
|
|
232
|
+
body: {
|
|
233
|
+
jsonrpc: '2.0',
|
|
234
|
+
id: 1,
|
|
235
|
+
method: 'tools/list',
|
|
236
|
+
params: { _meta: { ...META, protocolVersion: OLD_REV } },
|
|
237
|
+
},
|
|
238
|
+
},
|
|
239
|
+
expect: 'Explicit JSON-RPC error naming the supported revision(s) — unless the server intentionally supports both revisions during rollout, in which case it must apply ' +
|
|
240
|
+
OLD_REV +
|
|
241
|
+
' semantics consistently for this request.',
|
|
242
|
+
},
|
|
243
|
+
{
|
|
244
|
+
send: {
|
|
245
|
+
headers: routingHeaders('initialize'),
|
|
246
|
+
body: {
|
|
247
|
+
jsonrpc: '2.0',
|
|
248
|
+
id: 2,
|
|
249
|
+
method: 'initialize',
|
|
250
|
+
params: { protocolVersion: OLD_REV, clientInfo: META.clientInfo, capabilities: {} },
|
|
251
|
+
},
|
|
252
|
+
},
|
|
253
|
+
expect: 'A ' +
|
|
254
|
+
NEW_REV +
|
|
255
|
+
'-only server: method-not-found (-32601). A dual-revision server: a valid ' +
|
|
256
|
+
OLD_REV +
|
|
257
|
+
' initialize result. Anything else (crash, silent success under new semantics) fails.',
|
|
258
|
+
},
|
|
259
|
+
],
|
|
260
|
+
},
|
|
261
|
+
];
|
|
262
|
+
function checklist() {
|
|
263
|
+
const lines = [
|
|
264
|
+
'# mcp-vet conformance checklist (2026-07-28)',
|
|
265
|
+
'',
|
|
266
|
+
'The static scan proves known legacy patterns are absent from your source.',
|
|
267
|
+
'These fixtures prove the running server actually speaks the new wire contract.',
|
|
268
|
+
'Replay each `*.json` fixture against your server with your HTTP harness of',
|
|
269
|
+
'choice (curl, supertest, pytest + httpx, ...) and check the `expect` notes.',
|
|
270
|
+
'',
|
|
271
|
+
`Spec: ${constants_1.SPEC_URL}`,
|
|
272
|
+
'',
|
|
273
|
+
'## The dual-version rollout matrix',
|
|
274
|
+
'',
|
|
275
|
+
`**July 28 is a specification release date, not a switch that remotely disables`,
|
|
276
|
+
`existing deployments.** Breakage appears when a client and server negotiate or`,
|
|
277
|
+
`require the new revision. Until every client you care about has moved, your`,
|
|
278
|
+
`production test matrix needs BOTH paths:`,
|
|
279
|
+
'',
|
|
280
|
+
`- \`${OLD_REV}\` client -> your server (old semantics, or an explicit refusal)`,
|
|
281
|
+
`- \`${NEW_REV}\` client -> your server (new semantics)`,
|
|
282
|
+
'',
|
|
283
|
+
'Run fixture 09 in both configurations. Verify refusal is explicit — a server',
|
|
284
|
+
'that silently accepts a request under the wrong semantics is the failure mode',
|
|
285
|
+
'that reaches production.',
|
|
286
|
+
'',
|
|
287
|
+
'## Fixtures',
|
|
288
|
+
'',
|
|
289
|
+
];
|
|
290
|
+
for (const f of exports.CONFORMANCE_FIXTURES) {
|
|
291
|
+
lines.push(`- [ ] **${f.id}** — ${f.title}`);
|
|
292
|
+
lines.push(` ${f.description}`);
|
|
293
|
+
}
|
|
294
|
+
lines.push('', '## Client-side assumptions (test these too)', '', 'Reliability bugs also hide in clients that still behave as if they own a', 'session while the server is stateless:', '', '- [ ] client works with `sessionId: undefined` / no stored session id', '- [ ] client sends full `_meta` on every request, not just the first', '- [ ] client survives its next request landing on a different server instance', '- [ ] client retries do not depend on server-side per-client state', '- [ ] client revalidates tools/list instead of trusting list_changed pushes', '');
|
|
295
|
+
return lines.join('\n');
|
|
296
|
+
}
|
|
297
|
+
/** Write all conformance fixtures plus CHECKLIST.md into `dir`. */
|
|
298
|
+
function emitConformanceFixtures(dir) {
|
|
299
|
+
fs.mkdirSync(dir, { recursive: true });
|
|
300
|
+
const files = [];
|
|
301
|
+
for (const f of exports.CONFORMANCE_FIXTURES) {
|
|
302
|
+
const p = path.join(dir, `${f.id}.json`);
|
|
303
|
+
fs.writeFileSync(p, JSON.stringify(f, null, 2) + '\n', 'utf8');
|
|
304
|
+
files.push(p);
|
|
305
|
+
}
|
|
306
|
+
const cl = path.join(dir, 'CHECKLIST.md');
|
|
307
|
+
fs.writeFileSync(cl, checklist(), 'utf8');
|
|
308
|
+
files.push(cl);
|
|
309
|
+
return { dir, files };
|
|
310
|
+
}
|
package/dist/index.d.ts
CHANGED
|
@@ -23,6 +23,8 @@ export { applyFixes, isFixable } from './autofix';
|
|
|
23
23
|
export type { FixResult } from './autofix';
|
|
24
24
|
export { renderJson, renderMarkdown, renderSarif, toPublicFinding } from './reporters';
|
|
25
25
|
export { RULES } from './rules';
|
|
26
|
+
export { CONFORMANCE_FIXTURES, emitConformanceFixtures } from './conformance';
|
|
27
|
+
export type { ConformanceFixture, ConformanceStep, EmitResult } from './conformance';
|
|
26
28
|
export { IgnoreMatcher } from './ignore';
|
|
27
29
|
export { SPEC_URL, SPEC_DATE, CHANGELOG_URL, MANUAL_REVIEW, getVersion } from './constants';
|
|
28
30
|
export { ALL_PATTERN_IDS } from './types';
|
package/dist/index.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
-
exports.ALL_PATTERN_IDS = exports.getVersion = exports.MANUAL_REVIEW = exports.CHANGELOG_URL = exports.SPEC_DATE = exports.SPEC_URL = exports.IgnoreMatcher = exports.RULES = exports.toPublicFinding = exports.renderSarif = exports.renderMarkdown = exports.renderJson = exports.isFixable = exports.applyFixes = exports.ScanError = exports.scan = void 0;
|
|
3
|
+
exports.ALL_PATTERN_IDS = exports.getVersion = exports.MANUAL_REVIEW = exports.CHANGELOG_URL = exports.SPEC_DATE = exports.SPEC_URL = exports.IgnoreMatcher = exports.emitConformanceFixtures = exports.CONFORMANCE_FIXTURES = exports.RULES = exports.toPublicFinding = exports.renderSarif = exports.renderMarkdown = exports.renderJson = exports.isFixable = exports.applyFixes = exports.ScanError = exports.scan = void 0;
|
|
4
4
|
/**
|
|
5
5
|
* Programmatic API for mcp-vet.
|
|
6
6
|
*
|
|
@@ -33,6 +33,9 @@ Object.defineProperty(exports, "renderSarif", { enumerable: true, get: function
|
|
|
33
33
|
Object.defineProperty(exports, "toPublicFinding", { enumerable: true, get: function () { return reporters_1.toPublicFinding; } });
|
|
34
34
|
var rules_1 = require("./rules");
|
|
35
35
|
Object.defineProperty(exports, "RULES", { enumerable: true, get: function () { return rules_1.RULES; } });
|
|
36
|
+
var conformance_1 = require("./conformance");
|
|
37
|
+
Object.defineProperty(exports, "CONFORMANCE_FIXTURES", { enumerable: true, get: function () { return conformance_1.CONFORMANCE_FIXTURES; } });
|
|
38
|
+
Object.defineProperty(exports, "emitConformanceFixtures", { enumerable: true, get: function () { return conformance_1.emitConformanceFixtures; } });
|
|
36
39
|
var ignore_1 = require("./ignore");
|
|
37
40
|
Object.defineProperty(exports, "IgnoreMatcher", { enumerable: true, get: function () { return ignore_1.IgnoreMatcher; } });
|
|
38
41
|
var constants_1 = require("./constants");
|
|
@@ -23,7 +23,9 @@ CAP = {"roots", "sampling", "logging"}
|
|
|
23
23
|
INIT_STRINGS = {"initialize", "notifications/initialized"}
|
|
24
24
|
HANDLERISH = re.compile(r"handler|handle|register|route|request|notification|method|^on$", re.I)
|
|
25
25
|
CAPS_RE = re.compile(r"capabilit", re.I)
|
|
26
|
+
TRANSPORTISH = re.compile(r"transport|client", re.I)
|
|
26
27
|
METHODISH = ("method", "type")
|
|
28
|
+
SESSION_KWARGS = ("session_id", "sessionId")
|
|
27
29
|
|
|
28
30
|
|
|
29
31
|
def _func_mentions_caps(func):
|
|
@@ -103,10 +105,40 @@ def _is_registration(node):
|
|
|
103
105
|
return False
|
|
104
106
|
|
|
105
107
|
|
|
108
|
+
def _func_name(func):
|
|
109
|
+
if isinstance(func, ast.Attribute):
|
|
110
|
+
return func.attr
|
|
111
|
+
if isinstance(func, ast.Name):
|
|
112
|
+
return func.id
|
|
113
|
+
return ""
|
|
114
|
+
|
|
115
|
+
|
|
116
|
+
def _base_name(node):
|
|
117
|
+
"""Leftmost usable name of an attribute chain (`transport.session_id` -> 'transport')."""
|
|
118
|
+
if isinstance(node, ast.Attribute):
|
|
119
|
+
return _base_name(node.value) or node.attr
|
|
120
|
+
if isinstance(node, ast.Name):
|
|
121
|
+
return node.id
|
|
122
|
+
return ""
|
|
123
|
+
|
|
124
|
+
|
|
125
|
+
def _collect_aliases(tree):
|
|
126
|
+
"""Map local alias -> canonical imported name, so `from mcp.types import
|
|
127
|
+
RootsCapability as RC` still flags `RC()` usage sites (and the import line)."""
|
|
128
|
+
aliases = {}
|
|
129
|
+
for n in ast.walk(tree):
|
|
130
|
+
if isinstance(n, (ast.Import, ast.ImportFrom)):
|
|
131
|
+
for a in n.names:
|
|
132
|
+
if a.asname and a.asname != a.name:
|
|
133
|
+
aliases[a.asname] = a.name.rsplit(".", 1)[-1]
|
|
134
|
+
return aliases
|
|
135
|
+
|
|
136
|
+
|
|
106
137
|
class Scanner:
|
|
107
|
-
def __init__(self, lines):
|
|
138
|
+
def __init__(self, lines, aliases=None):
|
|
108
139
|
self.tokens = []
|
|
109
140
|
self.lines = lines
|
|
141
|
+
self.aliases = aliases or {}
|
|
110
142
|
|
|
111
143
|
def _col(self, node):
|
|
112
144
|
c = getattr(node, "col_offset", None)
|
|
@@ -136,8 +168,24 @@ class Scanner:
|
|
|
136
168
|
)
|
|
137
169
|
elif isinstance(node, ast.Name):
|
|
138
170
|
self.tokens.append({"kind": "name", "value": node.id, "line": node.lineno, "col": self._col(node)})
|
|
171
|
+
# An aliased identifier also counts as its canonical imported name.
|
|
172
|
+
original = self.aliases.get(node.id)
|
|
173
|
+
if original:
|
|
174
|
+
self.tokens.append({"kind": "name", "value": original, "line": node.lineno, "col": self._col(node)})
|
|
139
175
|
elif isinstance(node, ast.Attribute):
|
|
140
|
-
|
|
176
|
+
tok = {"kind": "name", "value": node.attr, "line": node.lineno, "col": self._col(node)}
|
|
177
|
+
# `transport.session_id` read = client-side session ownership.
|
|
178
|
+
if node.attr in SESSION_KWARGS and TRANSPORTISH.search(_base_name(node.value)):
|
|
179
|
+
tok["clientSession"] = True
|
|
180
|
+
self.tokens.append(tok)
|
|
181
|
+
elif isinstance(node, (ast.Import, ast.ImportFrom)):
|
|
182
|
+
# Surface imported names so `from mcp.types import X as Y` still
|
|
183
|
+
# flags the import line even though usages only say `Y`.
|
|
184
|
+
for a in node.names:
|
|
185
|
+
line = getattr(a, "lineno", None) or node.lineno
|
|
186
|
+
self.tokens.append(
|
|
187
|
+
{"kind": "name", "value": a.name.rsplit(".", 1)[-1], "line": line, "col": self._col(a) or self._col(node)}
|
|
188
|
+
)
|
|
141
189
|
|
|
142
190
|
def visit(self, node, in_caps):
|
|
143
191
|
self.emit_for(node, in_caps)
|
|
@@ -168,6 +216,13 @@ class Scanner:
|
|
|
168
216
|
tok = {"kind": "key", "value": kw.arg, "line": line, "col": self._col(kw)}
|
|
169
217
|
if kw.arg in CAP:
|
|
170
218
|
tok["inCapabilities"] = caps_ctx
|
|
219
|
+
# `session_id=` into a transport/client factory = the client
|
|
220
|
+
# resuming/owning a session. `session_id=None` is migrated.
|
|
221
|
+
if kw.arg in SESSION_KWARGS and TRANSPORTISH.search(_func_name(node.func)):
|
|
222
|
+
if isinstance(kw.value, ast.Constant) and kw.value.value is None:
|
|
223
|
+
tok["benign"] = True
|
|
224
|
+
else:
|
|
225
|
+
tok["clientSession"] = True
|
|
171
226
|
self.tokens.append(tok)
|
|
172
227
|
child_caps = caps_ctx or (kw.arg == "capabilities")
|
|
173
228
|
self.visit(kw.value, child_caps)
|
|
@@ -185,7 +240,7 @@ def scan_source(src):
|
|
|
185
240
|
for n in ast.walk(tree):
|
|
186
241
|
for c in ast.iter_child_nodes(n):
|
|
187
242
|
c.parent = n
|
|
188
|
-
scanner = Scanner(src.split("\n"))
|
|
243
|
+
scanner = Scanner(src.split("\n"), _collect_aliases(tree))
|
|
189
244
|
try:
|
|
190
245
|
scanner.visit(tree, False)
|
|
191
246
|
except RecursionError:
|
package/dist/reporters.js
CHANGED
|
@@ -117,6 +117,7 @@ function reportTerminal(result, opts = {}) {
|
|
|
117
117
|
/** One-line pointer to the changes static analysis can't catch — keeps the tool honest. */
|
|
118
118
|
function printManualReview(c) {
|
|
119
119
|
console.error(c.gray(`note: ${constants_1.MANUAL_REVIEW.length} more 2026-07-28 changes need manual review (SSE push, required headers, auth, JSON Schema 2020-12) — see the README "Needs manual review" section.`));
|
|
120
|
+
console.error(c.gray('note: July 28 is a spec release, not a remote kill switch — breakage appears when a client/server pair negotiates the new revision. Test both 2025-11-25 and 2026-07-28 paths during rollout (`mcp-vet fixtures`).'));
|
|
120
121
|
}
|
|
121
122
|
function mdEscape(s) {
|
|
122
123
|
return s.replace(/\|/g, '\\|').replace(/\r?\n/g, ' ');
|
package/dist/rules.js
CHANGED
|
@@ -232,6 +232,16 @@ function applyRules(relPath, lines, tokens, opts) {
|
|
|
232
232
|
!t.benign) {
|
|
233
233
|
push('MCP_SESSION_ID', t, 'medium');
|
|
234
234
|
}
|
|
235
|
+
// Rule 11 — client-side session ownership. A client transport constructed
|
|
236
|
+
// with a real sessionId / session_id, or a read of transport.sessionId,
|
|
237
|
+
// means the client still behaves as if it owns a session — which breaks
|
|
238
|
+
// against a stateless 2026-07-28 server even when the server scans clean.
|
|
239
|
+
if ((t.kind === 'key' || t.kind === 'name') &&
|
|
240
|
+
(v === 'sessionId' || v === 'session_id') &&
|
|
241
|
+
t.clientSession &&
|
|
242
|
+
!t.benign) {
|
|
243
|
+
push('MCP_SESSION_ID', t, 'medium');
|
|
244
|
+
}
|
|
235
245
|
// Rules 5-7 — deprecated capabilities.
|
|
236
246
|
// High confidence when structurally inside a `capabilities` object (AST);
|
|
237
247
|
// medium when only within 5 lines of a "capabilities" mention.
|
package/dist/ts-analyzer.js
CHANGED
|
@@ -41,6 +41,26 @@ function safePropName(node) {
|
|
|
41
41
|
return undefined;
|
|
42
42
|
}
|
|
43
43
|
}
|
|
44
|
+
const TRANSPORTISH = /transport|client/i;
|
|
45
|
+
/**
|
|
46
|
+
* Is `node` inside a call/constructor argument of something transport/client
|
|
47
|
+
* shaped (e.g. `new StreamableHTTPClientTransport(url, { sessionId })`)?
|
|
48
|
+
* Drives the client-side session-ownership check.
|
|
49
|
+
*/
|
|
50
|
+
function isClientTransportContext(node) {
|
|
51
|
+
let depth = 0;
|
|
52
|
+
for (const anc of node.getAncestors()) {
|
|
53
|
+
if (depth++ > 8)
|
|
54
|
+
break;
|
|
55
|
+
const k = anc.getKind();
|
|
56
|
+
if (k === ts_morph_1.SyntaxKind.CallExpression || k === ts_morph_1.SyntaxKind.NewExpression) {
|
|
57
|
+
const expr = anc.getExpression?.();
|
|
58
|
+
if (expr && TRANSPORTISH.test(expr.getText()))
|
|
59
|
+
return true;
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
return false;
|
|
63
|
+
}
|
|
44
64
|
/** Is `node` structurally inside a `capabilities` object / call argument? */
|
|
45
65
|
function isInCapabilities(node) {
|
|
46
66
|
let depth = 0;
|
|
@@ -143,6 +163,22 @@ function analyzeTs(absPath, text) {
|
|
|
143
163
|
return { line: node.getStartLineNumber(), col: undefined };
|
|
144
164
|
}
|
|
145
165
|
};
|
|
166
|
+
// Aliased named imports (`import { InitializeRequestSchema as Init }`): map the
|
|
167
|
+
// local alias back to its canonical SDK name so *usage sites* are flagged too,
|
|
168
|
+
// not just the import line where the original identifier happens to appear.
|
|
169
|
+
const aliases = new Map();
|
|
170
|
+
try {
|
|
171
|
+
for (const imp of sf.getImportDeclarations()) {
|
|
172
|
+
for (const spec of imp.getNamedImports()) {
|
|
173
|
+
const aliasNode = spec.getAliasNode();
|
|
174
|
+
if (aliasNode)
|
|
175
|
+
aliases.set(aliasNode.getText(), spec.getName());
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
}
|
|
179
|
+
catch {
|
|
180
|
+
/* ignore malformed imports */
|
|
181
|
+
}
|
|
146
182
|
try {
|
|
147
183
|
sf.forEachDescendant((node) => {
|
|
148
184
|
const kind = node.getKind();
|
|
@@ -183,10 +219,33 @@ function analyzeTs(absPath, text) {
|
|
|
183
219
|
}
|
|
184
220
|
if (kind === ts_morph_1.SyntaxKind.Identifier) {
|
|
185
221
|
const { line, col } = posOf(node);
|
|
186
|
-
|
|
222
|
+
const text = node.getText();
|
|
223
|
+
tokens.push({ kind: 'name', value: text, line, col });
|
|
224
|
+
// An aliased identifier also counts as its canonical imported name —
|
|
225
|
+
// except inside the import specifier itself, where the original
|
|
226
|
+
// identifier is already present (avoids a duplicate import-line finding).
|
|
227
|
+
const original = aliases.get(text);
|
|
228
|
+
if (original && node.getParent()?.getKind() !== ts_morph_1.SyntaxKind.ImportSpecifier) {
|
|
229
|
+
tokens.push({ kind: 'name', value: original, line, col });
|
|
230
|
+
}
|
|
187
231
|
return;
|
|
188
232
|
}
|
|
189
233
|
});
|
|
234
|
+
// Client-side session ownership: reads of `<transport>.sessionId` mean the
|
|
235
|
+
// client still behaves as if it owns a session against a stateless server.
|
|
236
|
+
for (const pae of sf.getDescendantsOfKind(ts_morph_1.SyntaxKind.PropertyAccessExpression)) {
|
|
237
|
+
try {
|
|
238
|
+
if (pae.getName() !== 'sessionId')
|
|
239
|
+
continue;
|
|
240
|
+
if (!TRANSPORTISH.test(pae.getExpression().getText()))
|
|
241
|
+
continue;
|
|
242
|
+
const { line, col } = posOf(pae.getNameNode());
|
|
243
|
+
tokens.push({ kind: 'name', value: 'sessionId', line, col, clientSession: true });
|
|
244
|
+
}
|
|
245
|
+
catch {
|
|
246
|
+
/* ignore */
|
|
247
|
+
}
|
|
248
|
+
}
|
|
190
249
|
// Object literal keys (roots:, "sampling":, logging shorthand, ...)
|
|
191
250
|
const emitKey = (nameNode, value, benign = false) => {
|
|
192
251
|
const { line, col } = posOf(nameNode);
|
|
@@ -195,14 +254,20 @@ function analyzeTs(absPath, text) {
|
|
|
195
254
|
tok.inCapabilities = isInCapabilities(nameNode);
|
|
196
255
|
if (benign)
|
|
197
256
|
tok.benign = true;
|
|
257
|
+
// `sessionId` passed into a client transport constructor/factory = the
|
|
258
|
+
// client resuming/owning a session.
|
|
259
|
+
if (value === 'sessionId' && isClientTransportContext(nameNode)) {
|
|
260
|
+
tok.clientSession = true;
|
|
261
|
+
}
|
|
198
262
|
tokens.push(tok);
|
|
199
263
|
};
|
|
200
264
|
for (const pa of sf.getDescendantsOfKind(ts_morph_1.SyntaxKind.PropertyAssignment)) {
|
|
201
265
|
try {
|
|
202
266
|
const name = pa.getName();
|
|
203
|
-
// `sessionIdGenerator: undefined` (or null) is
|
|
267
|
+
// `sessionIdGenerator: undefined` / `sessionId: undefined` (or null) is
|
|
268
|
+
// the migrated, stateless form.
|
|
204
269
|
let benign = false;
|
|
205
|
-
if (name === 'sessionIdGenerator') {
|
|
270
|
+
if (name === 'sessionIdGenerator' || name === 'sessionId') {
|
|
206
271
|
const init = pa.getInitializer()?.getText();
|
|
207
272
|
benign = init === 'undefined' || init === 'null';
|
|
208
273
|
}
|
package/dist/types.d.ts
CHANGED
|
@@ -31,6 +31,13 @@ export interface Token {
|
|
|
31
31
|
* e.g. `sessionIdGenerator: undefined`, the documented stateless migration.
|
|
32
32
|
*/
|
|
33
33
|
benign?: boolean;
|
|
34
|
+
/**
|
|
35
|
+
* True when a `sessionId`/`session_id` token sits in *client-side* session
|
|
36
|
+
* ownership context — a client transport constructed with a session id, or a
|
|
37
|
+
* read of `transport.sessionId`. Client code that still owns a session breaks
|
|
38
|
+
* against a stateless server even when the server itself scans clean.
|
|
39
|
+
*/
|
|
40
|
+
clientSession?: boolean;
|
|
34
41
|
}
|
|
35
42
|
export interface Finding {
|
|
36
43
|
/** path relative to the scan root, forward-slashed */
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@booyaka/mcp-vet",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Scan MCP server source code for patterns that break under the 2026-07-28 Model Context Protocol spec release candidate.",
|
|
5
5
|
"type": "commonjs",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -21,6 +21,7 @@
|
|
|
21
21
|
"schema",
|
|
22
22
|
"README.md",
|
|
23
23
|
"CHANGELOG.md",
|
|
24
|
+
"BENCHMARK.md",
|
|
24
25
|
"LICENSE"
|
|
25
26
|
],
|
|
26
27
|
"engines": {
|