@booyaka/mcp-vet 0.7.0 → 0.9.0

package/BENCHMARK.md

@@ -1,112 +1,161 @@
-# Benchmark: corpus, methodology, and honest limits
-
-The README claims high precision on real MCP code. This file is the evidence
-behind that claim — corpus, pinned commits, counts, labels, and what the
-scanner is *known to miss* — so the claim is checkable rather than vibes.
-
-> Prompted by community feedback on the launch post: *"'0 false positives' is
-> encouraging but incomplete without corpus size, commit SHAs, labeled
-> negatives, and recall."* Correct. Here they are.
-
-## Corpus (pinned)
-
-Scanned with `mcp-vet` v0.4.0 (`node dist/cli.js <roots> --json`), all rules
-enabled, default confidence (`low`), on 2026-07-23:
-
-| Repo | Commit | Scanned root | Files | LOC |
-| --- | --- | --- | --- | --- |
-| [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) | `d31124c982401739917fd817c2a59db344529c16` | `src/` | 78 | 14,742 |
-| [modelcontextprotocol/typescript-sdk](https://github.com/modelcontextprotocol/typescript-sdk) | `1e1392e3f91583884fe82a0b4b91335875c3fba6` | `examples/` | 144 | 17,224 |
-| [modelcontextprotocol/python-sdk](https://github.com/modelcontextprotocol/python-sdk) | `3a6f2996cdd8358957479791e8b26198c07d6a75` | `examples/` | 225 | 12,013 |
-| **Total** | | | **447** | **43,979** |
-
-File counts are candidate files (`.ts/.tsx/.js/.mjs/.cjs/.py`) under the
-scanned roots, excluding `node_modules`.
-
-## Results
-
-**105 findings across 41 files** (TypeScript/JavaScript: 93, Python: 12).
-By confidence: 66 high, 38 medium, 1 low.
-
-| Pattern | Findings |
-| --- | --- |
-| `MCP_SESSION_ID` | 49 |
-| `LOGGING_CAP` | 17 |
-| `SAMPLING_CAP` | 16 |
-| `ROOTS_CAP` | 15 |
-| `INITIALIZE_HANDLER` | 4 |
-| `TASKS_LEGACY` | 2 |
-| `TASKS_RESULT_REMOVED` | 2 |
-
-### Labeling
-
-Every finding was manually reviewed against its source line:
-
-- **104 / 105 true positives** — real references to a removed or deprecated
-  protocol surface (session headers/ids, handshake registration, legacy task
-  methods, deprecated capability declarations and method strings).
-- **1 / 105 false positive (0.95%)** —
-  `stories/json_response/client.py:62` in the typescript-sdk examples:
-  `assert "mcp-session-id" not in response.headers`. That line is
-  *already-migrated* test code asserting the header is **absent**; flagging it
-  as "will break" is wrong. It is exactly what inline suppression
-  (`# mcp-vet-disable-line MCP_SESSION_ID`) is for, but we count it as a false
-  positive rather than defining it away. So the honest headline is
-  **"1 false positive in 44k LOC"**, not zero.
-
-Notes on reading the numbers:
-
-- Two occurrences on one line (e.g. `transport.sessionId && sessions.delete(transport.sessionId)`)
-  are reported as two findings — column-level dedup, not line-level.
-- Findings in test files (`__tests__/…`) are counted as true positives: a test
-  that registers `sampling/createMessage` breaks the same way production code
-  does.
-
-### Labeled negatives
-
-Files asserted to stay **clean** are part of the repo's test suite and run in CI:
-
-- `test/fixtures/clean/` — a full server written in the 2026-07-28 style
-  (per-request `_meta`, `sessionIdGenerator: undefined`, `-32602`).
-- `test/fixtures/negatives/` — "false friend" patterns: `sessionId` on plain
-  app-level objects, `-32002` inside strings/comments, capability-like words
-  with no capabilities context.
-- `test/fixtures/adversarial/caught/` — obfuscations the scanner **must**
-  catch: aliased imports (TS + Python), namespace-qualified SDK constants,
-  client transports resuming a `sessionId`.
-
-Additionally, in the corpus above, comment-only mentions (e.g. `Mcp-Session-Id`
-in a comment, `initialize` in prose) produced zero findings — the AST layer
-distinguishes executable tokens from comments by construction.
-
-## Recall — what the scanner is known to miss
-
-Static token analysis proves known patterns are **absent**; it cannot prove
-your server **speaks the new wire contract**. Recall is bounded by
-construction, and the misses are locked into the test suite
-(`test/fixtures/adversarial/missed/`, asserted to produce zero findings so any
-silent claim-inflation fails CI):
-
-- split/computed method strings — `'tasks' + '/list'`, `` `tasks/${op}` ``, f-strings
-- computed capability keys — `{ ['roo'+'ts']: {} }`
-- generated/loop-driven registration from string fragments
-- framework-adapter indirection (route tables built at runtime)
-- cross-module renames — a wrapper re-exporting an SDK constant under a new
-  name is flagged in the wrapper file, but a consumer importing only the new
-  name scans clean on its own
-
-There is no corpus-wide recall *percentage*: that would require a labeled set
-of every legacy usage in the wild, which nobody has. What we can say is: for
-the pattern shapes listed in the README, detection is exact; for the shapes
-above, it is zero, and the tool says so — pair the scan with runtime checks
-(`mcp-vet fixtures`) to cover the difference.
-
-## Reproducing
-
-```bash
-git clone --depth 1 https://github.com/modelcontextprotocol/servers
-git clone --depth 1 https://github.com/modelcontextprotocol/typescript-sdk
-git clone --depth 1 https://github.com/modelcontextprotocol/python-sdk
-# check out the pinned SHAs above, then:
-npx @booyaka/mcp-vet servers/src typescript-sdk/examples python-sdk/examples --json --no-files
-```
+# Benchmark: corpus, methodology, and honest limits
+
+The README claims high precision on real MCP code. This file is the evidence
+behind that claim — corpus, pinned commits, counts, labels, and what the
+scanner is *known to miss* — so the claim is checkable rather than vibes.
+
+> Prompted by community feedback on the launch post: *"'0 false positives' is
+> encouraging but incomplete without corpus size, commit SHAs, labeled
+> negatives, and recall."* Correct. Here they are.
+
+## Corpus (pinned)
+
+Originally scanned with `mcp-vet` v0.4.0 (re-scanned with v0.9.0 above) (`node dist/cli.js <roots> --json`), all rules
+enabled, default confidence (`low`), on 2026-07-23:
+
+| Repo | Commit | Scanned root | Files | LOC |
+| --- | --- | --- | --- | --- |
+| [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) | `d31124c982401739917fd817c2a59db344529c16` | `src/` | 78 | 14,742 |
+| [modelcontextprotocol/typescript-sdk](https://github.com/modelcontextprotocol/typescript-sdk) | `1e1392e3f91583884fe82a0b4b91335875c3fba6` | `examples/` | 144 | 17,224 |
+| [modelcontextprotocol/python-sdk](https://github.com/modelcontextprotocol/python-sdk) | `3a6f2996cdd8358957479791e8b26198c07d6a75` | `examples/` | 225 | 12,013 |
+| **Total** | | | **447** | **43,979** |
+
+File counts are candidate files (`.ts/.tsx/.js/.mjs/.cjs/.py`) under the
+scanned roots, excluding `node_modules`.
+
+## Results — v0.9.0 (18-rule engine, final 2026-07-28 changelog)
+
+Re-run 2026-07-28 with `mcp-vet` v0.9.0 on the SAME pinned corpus:
+**243 findings across 66 files** (was 105/41 with the 9-rule engine — the
+final changelog's removals fire heavily on the SDKs' own pre-final examples).
+By confidence: 100 high, 142 medium, 1 low.
+
+| Pattern | Findings |
+| --- | --- |
+| `SSE_RESUMABILITY_REMOVED` | 65 |
+| `MCP_SESSION_ID` | 49 |
+| `ELICITATION_COMPLETE_REMOVED` | 44 |
+| `RESOURCE_SUBSCRIBE_REMOVED` | 23 |
+| `SAMPLING_CAP` | 16 |
+| `LOGGING_CAP` | 16 |
+| `ROOTS_CAP` | 10 |
+| `ROOTS_LIST_CHANGED_REMOVED` | 5 |
+| `INITIALIZE_HANDLER` | 4 |
+| `PING_REMOVED` | 3 |
+| `TASKS_LEGACY` | 2 |
+| `TASKS_RESULT_REMOVED` | 2 |
+| `OAUTH_DCR` | 2 |
+| `ERROR_CODE_RENUMBERED` | 1 |
+| `LOGGING_SETLEVEL_REMOVED` | 1 |
+
+Labeling of the 138 NEW findings (spot-reviewed per category against source):
+
+- **136 true positives.** The corpus repos genuinely implement the removed
+  surfaces: the typescript-sdk `everything` example ships an
+  `InMemoryEventStore` + `Last-Event-ID` resumability transport (all 65 SSE
+  findings sit in those transport/resumability files), the elicitation examples
+  register `notifications/elicitation/complete` handlers and read
+  `elicitationId`, `everything/resources/subscriptions.ts` and friends register
+  `SubscribeRequestSchema`/`UnsubscribeRequestSchema`, and the bearer-auth
+  clients send `method: 'ping'`.
+- **2 counted as false positives (honest reading):**
+  `guides/serving/sessions-state-scaling.examples.ts:62` returns
+  `code: -32001` for "Session not found" — an *implementation-defined* use the
+  final policy grandfathers; static analysis cannot distinguish it from the
+  renumbered `HeaderMismatch`, so ERROR_CODE_RENUMBERED flags it and we count
+  it against ourselves. Plus the pre-existing `mcp-session-id` negative
+  assertion (below). During this run a `registerTool('ping', ...)` false
+  positive (a tool merely NAMED ping) was found and FIXED before release —
+  strict registration context now excludes tool/prompt/resource registration
+  calls, locked into `negatives/`.
+
+So the v0.9.0 headline on this corpus is **241/243 true positives (2 FP,
+0.8%)** — same discipline as before: FPs are counted, not defined away.
+
+## Results — v0.4.0 (9-rule engine, release-candidate era)
+
+**105 findings across 41 files** (TypeScript/JavaScript: 93, Python: 12).
+By confidence: 66 high, 38 medium, 1 low.
+
+| Pattern | Findings |
+| --- | --- |
+| `MCP_SESSION_ID` | 49 |
+| `LOGGING_CAP` | 17 |
+| `SAMPLING_CAP` | 16 |
+| `ROOTS_CAP` | 15 |
+| `INITIALIZE_HANDLER` | 4 |
+| `TASKS_LEGACY` | 2 |
+| `TASKS_RESULT_REMOVED` | 2 |
+
+### Labeling
+
+Every finding was manually reviewed against its source line:
+
+- **104 / 105 true positives** — real references to a removed or deprecated
+  protocol surface (session headers/ids, handshake registration, legacy task
+  methods, deprecated capability declarations and method strings).
+- **1 / 105 false positive (0.95%)** —
+  `stories/json_response/client.py:62` in the typescript-sdk examples:
+  `assert "mcp-session-id" not in response.headers`. That line is
+  *already-migrated* test code asserting the header is **absent**; flagging it
+  as "will break" is wrong. It is exactly what inline suppression
+  (`# mcp-vet-disable-line MCP_SESSION_ID`) is for, but we count it as a false
+  positive rather than defining it away. So the honest headline is
+  **"1 false positive in 44k LOC"**, not zero.
+
+Notes on reading the numbers:
+
+- Two occurrences on one line (e.g. `transport.sessionId && sessions.delete(transport.sessionId)`)
+  are reported as two findings — column-level dedup, not line-level.
+- Findings in test files (`__tests__/…`) are counted as true positives: a test
+  that registers `sampling/createMessage` breaks the same way production code
+  does.
+
+### Labeled negatives
+
+Files asserted to stay **clean** are part of the repo's test suite and run in CI:
+
+- `test/fixtures/clean/` — a full server written in the 2026-07-28 style
+  (per-request `_meta`, `sessionIdGenerator: undefined`, `-32602`).
+- `test/fixtures/negatives/` — "false friend" patterns: `sessionId` on plain
+  app-level objects, `-32002` inside strings/comments, capability-like words
+  with no capabilities context.
+- `test/fixtures/adversarial/caught/` — obfuscations the scanner **must**
+  catch: aliased imports (TS + Python), namespace-qualified SDK constants,
+  client transports resuming a `sessionId`.
+
+Additionally, in the corpus above, comment-only mentions (e.g. `Mcp-Session-Id`
+in a comment, `initialize` in prose) produced zero findings — the AST layer
+distinguishes executable tokens from comments by construction.
+
+## Recall — what the scanner is known to miss
+
+Static token analysis proves known patterns are **absent**; it cannot prove
+your server **speaks the new wire contract**. Recall is bounded by
+construction, and the misses are locked into the test suite
+(`test/fixtures/adversarial/missed/`, asserted to produce zero findings so any
+silent claim-inflation fails CI):
+
+- split/computed method strings — `'tasks' + '/list'`, `` `tasks/${op}` ``, f-strings
+- computed capability keys — `{ ['roo'+'ts']: {} }`
+- generated/loop-driven registration from string fragments
+- framework-adapter indirection (route tables built at runtime)
+- cross-module renames — a wrapper re-exporting an SDK constant under a new
+  name is flagged in the wrapper file, but a consumer importing only the new
+  name scans clean on its own
+
+There is no corpus-wide recall *percentage*: that would require a labeled set
+of every legacy usage in the wild, which nobody has. What we can say is: for
+the pattern shapes listed in the README, detection is exact; for the shapes
+above, it is zero, and the tool says so — pair the scan with runtime checks
+(`mcp-vet fixtures`) to cover the difference.
+
+## Reproducing
+
+```bash
+git clone --depth 1 https://github.com/modelcontextprotocol/servers
+git clone --depth 1 https://github.com/modelcontextprotocol/typescript-sdk
+git clone --depth 1 https://github.com/modelcontextprotocol/python-sdk
+# check out the pinned SHAs above, then:
+npx @booyaka/mcp-vet servers/src typescript-sdk/examples python-sdk/examples --json --no-files
+```

package/CHANGELOG.md

@@ -4,6 +4,92 @@ All notable changes to `mcp-vet` are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres
 to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0]
+
+The final-specification release. mcp-vet was built against the 2026-07-28
+release candidate; the FINAL Key Changes list published 2026-07-28 at
+<https://modelcontextprotocol.io/specification/draft/changelog> is materially
+longer, so a clean 0.8.0 scan was a false all-clear. This release closes that
+gap — every rule now cites a sentence pinned verbatim in
+`docs/SPEC-2026-07-28.md`. (The dated URL
+`/specification/2026-07-28` still returns 404; the final text is served under
+`/specification/draft/`.)
+
+### Added — seven BREAKING static rules (exit 1), in BOTH analyzers
+
+- **`PING_REMOVED`** — `ping` in MCP method-registration context,
+  `PingRequestSchema`, Python `types.PingRequest`. A `/ping` health route, a
+  bare `'ping'` string, or a tool merely *named* ping stays clean.
+- **`RESOURCE_SUBSCRIBE_REMOVED`** — `resources/subscribe`,
+  `resources/unsubscribe`, `SubscribeRequestSchema`,
+  `UnsubscribeRequestSchema`; the fix points at `subscriptions/listen` and its
+  four opt-in types.
+- **`ROOTS_LIST_CHANGED_REMOVED`** and **`LOGGING_SETLEVEL_REMOVED`** — a
+  RECLASSIFICATION: 0.8.0 mapped `notifications/roots/list_changed` and
+  `logging/setLevel` (+ their SDK schema constants) to the DEPRECATED
+  capability rules, reporting two hard removals as exit-0 warnings with a
+  grace-period label. They are BREAKING now; the `roots`/`logging` capability
+  *keys* stay DEPRECATED, and a test locks the severity split.
+- **`SSE_RESUMABILITY_REMOVED`** — the `Last-Event-ID` header string,
+  `lastEventId`, and `eventStore`/`resumptionToken`/`onresumptiontoken` passed
+  to a Streamable HTTP transport. Gated on file-level MCP context, so a plain
+  SSE client stays clean (locked by `negatives/sse-client.ts`).
+- **`ELICITATION_COMPLETE_REMOVED`** — `notifications/elicitation/complete`
+  and the `elicitationId` field.
+- **`ERROR_CODE_RENUMBERED`** — `-32001`/`-32003`/`-32004` → `-32020`/`-32021`/
+  `-32022`, flagged ONLY in a JSON-RPC error `code` position (`code:` key,
+  `*Error(...)` construction, comparison against `code`) — the changelog
+  grandfathers `-32000..-32019` for implementation-defined codes, so a bare
+  negative constant is never flagged.
+
+### Added — two DEPRECATED static rules (exit 0)
+
+- **`INCLUDE_CONTEXT_VALUES`** — `includeContext` set to `"thisServer"` /
+  `"allServers"` (medium; removal "Follows Sampling").
+- **`OAUTH_DCR`** — RFC7591 dynamic-client-registration surfaces
+  (`registration_endpoint`, `registration_access_token`,
+  `client_id_issued_at`) in favour of Client ID Metadata Documents (medium).
+
+### Added — probe & fixtures
+
+- Four checks join the opt-in `mcp-vet probe --spec 2026-07-28` suite (now ten):
+  **`missing-result-type`** (ERROR, SEP-2322), **`missing-cacheable-fields`**
+  (WARN, SEP-2549 `ttlMs` + `cacheScope`), **`legacy-error-code-renumbered`**
+  (ERROR — still answering `-32001`/`-32003`/`-32004`), and
+  **`ping-still-answered`** (WARN — `ping` returns a result instead of
+  `-32601`). All cross-checked: a dead or non-MCP server is exit 2, never a
+  false violation, and every inconclusive outcome is a note.
+- `mcp-vet fixtures` gains **`10-subscriptions-listen`** (opt-in +
+  `io.modelcontextprotocol/subscriptionId` tagging + `resources/subscribe` →
+  -32601) and **`11-mrtr`** (`resultType: "input_required"` + `inputRequests`,
+  retry with `inputResponses`) — eleven fixtures total.
+- `--fix` now rewrites the renumbered codes next to the existing `-32002` →
+  `-32602` (same-length, column-anchored); `--dry-run` lists every rewrite.
+- DEPRECATED findings now print the registry's exact removal window ("First
+  revision released on or after 2027-07-28", "Follows Sampling") instead of a
+  hardcoded 12 months.
+- New `test/fixtures/dirty/` (TS + Python, one instance of every new pattern),
+  migrated forms in `clean/`, new true-negatives, computed/split forms in
+  `adversarial/missed/`. 78 tests (was 71), none skipped.
+- BENCHMARK.md re-measured with the 18-rule engine on the same pinned corpus.
+
+## [0.8.0]
+
+Maintenance release — dependency and CI currency; no rule or probe changes.
+
+### Changed
+
+- Dropped `chalk` for a local colouriser; moved to `commander` 15,
+  `ts-morph` 28, `@types/node` 26.
+- CI tests on supported Node only (22/24/26); `engines.node` >= 22.
+- `mcp-vet probe` lets the event loop drain instead of calling
+  `process.exit()` — fixes an intermittent Windows libuv crash (0xC0000409)
+  on process teardown.
+- GitHub Actions moved to latest majors; Dependabot groups minor/patch and
+  splits majors; the blocked TypeScript 7 major is ignored until the
+  Compiler-API crash upstream clears; our own guards (npm-script-lens audit +
+  allowlist drift, ts7-compat-guard, pnpm11-ci-guard) run against this repo.
+
 ## [0.7.0]
 
 An opt-in `--spec 2026-07-28` compliance suite for `mcp-vet probe` — six