@yawlabs/mcp-compliance 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +77 -12
- package/dist/{chunk-U66YZGE5.js → chunk-KNOSZ3TD.js} +86 -43
- package/dist/index.js +208 -62
- package/dist/mcp/server.d.ts +11 -1
- package/dist/mcp/server.js +26 -13
- package/dist/runner.d.ts +1 -0
- package/dist/runner.js +1 -1
- package/package.json +13 -5
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Yaw Labs
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -7,17 +7,31 @@
|
|
|
7
7
|
|
|
8
8
|
**Test any MCP server for spec compliance.** 43-test suite covering transport, lifecycle, tools, resources, prompts, error handling, and schema validation against the [MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). CLI, MCP server, and programmatic API.
|
|
9
9
|
|
|
10
|
-
Built and maintained by [
|
|
10
|
+
Built and maintained by [Yaw Labs](https://yaw.sh).
|
|
11
11
|
|
|
12
|
-
##
|
|
12
|
+
## Why this tool?
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.
|
|
15
|
+
|
|
16
|
+
This tool solves that:
|
|
17
|
+
|
|
18
|
+
- **43 tests across 7 categories** — transport, lifecycle, tools, resources, prompts, error handling, and schema validation. No gaps.
|
|
19
|
+
- **Capability-driven** — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
|
|
20
|
+
- **Graded scoring** — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
|
|
21
|
+
- **CI-ready** — `--strict` mode exits with code 1 on required test failures. Drop it into any pipeline.
|
|
22
|
+
- **Spec-referenced** — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
|
|
23
|
+
- **Three interfaces** — CLI for humans, MCP server for AI assistants, programmatic API for integration.
|
|
24
|
+
- **Published specification** — the [testing methodology](./MCP_COMPLIANCE_SPEC.md) and [rule catalog](./mcp-compliance-rules.json) are open (CC BY 4.0) so anyone can implement compatible tooling.
|
|
25
|
+
|
|
26
|
+
## Quick start
|
|
27
|
+
|
|
28
|
+
**Run against any MCP server:**
|
|
15
29
|
|
|
16
30
|
```bash
|
|
17
31
|
npx @yawlabs/mcp-compliance test https://my-server.com/mcp
|
|
18
32
|
```
|
|
19
33
|
|
|
20
|
-
**
|
|
34
|
+
**Or install globally:**
|
|
21
35
|
|
|
22
36
|
```bash
|
|
23
37
|
npm install -g @yawlabs/mcp-compliance
|
|
@@ -26,7 +40,7 @@ mcp-compliance test https://my-server.com/mcp
|
|
|
26
40
|
|
|
27
41
|
That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.
|
|
28
42
|
|
|
29
|
-
## CLI
|
|
43
|
+
## CLI usage
|
|
30
44
|
|
|
31
45
|
```bash
|
|
32
46
|
# Terminal output with colors and grade
|
|
@@ -35,6 +49,9 @@ mcp-compliance test https://my-server.com/mcp
|
|
|
35
49
|
# JSON output (for scripting)
|
|
36
50
|
mcp-compliance test https://my-server.com/mcp --format json
|
|
37
51
|
|
|
52
|
+
# SARIF output (for GitHub Code Scanning)
|
|
53
|
+
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif
|
|
54
|
+
|
|
38
55
|
# Strict mode — exits with code 1 on required test failure (for CI)
|
|
39
56
|
mcp-compliance test https://my-server.com/mcp --strict
|
|
40
57
|
|
|
@@ -65,7 +82,7 @@ mcp-compliance test https://my-server.com/mcp --verbose
|
|
|
65
82
|
|
|
66
83
|
| Option | Description |
|
|
67
84
|
|--------|-------------|
|
|
68
|
-
| `--format <format>` | Output format: `terminal` or `
|
|
85
|
+
| `--format <format>` | Output format: `terminal`, `json`, or `sarif` (default: `terminal`) |
|
|
69
86
|
| `--strict` | Exit with code 1 on any required test failure (for CI) |
|
|
70
87
|
| `-H, --header <header>` | Add header to all requests, format `"Key: Value"` (repeatable) |
|
|
71
88
|
| `--auth <token>` | Shorthand for `-H "Authorization: <token>"` |
|
|
@@ -180,9 +197,9 @@ Outputs the markdown embed for a compliance badge hosted at [mcp.hosting](https:
|
|
|
180
197
|
| D | 40-59 |
|
|
181
198
|
| F | 0-39 |
|
|
182
199
|
|
|
183
|
-
Required tests are worth 70% of the score, optional tests 30%.
|
|
200
|
+
Required tests are worth 70% of the score, optional tests 30%. See the [full scoring algorithm](./MCP_COMPLIANCE_SPEC.md#2-scoring-algorithm) in the specification.
|
|
184
201
|
|
|
185
|
-
## CI
|
|
202
|
+
## CI integration
|
|
186
203
|
|
|
187
204
|
```yaml
|
|
188
205
|
# GitHub Actions example
|
|
@@ -204,13 +221,29 @@ Required tests are worth 70% of the score, optional tests 30%.
|
|
|
204
221
|
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --strict --retries 2 --timeout 30000
|
|
205
222
|
```
|
|
206
223
|
|
|
207
|
-
|
|
224
|
+
```yaml
|
|
225
|
+
# SARIF output for GitHub Code Scanning
|
|
226
|
+
- name: MCP Compliance Check
|
|
227
|
+
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --format sarif > compliance.sarif
|
|
228
|
+
- name: Upload SARIF
|
|
229
|
+
uses: github/codeql-action/upload-sarif@v3
|
|
230
|
+
with:
|
|
231
|
+
sarif_file: compliance.sarif
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## MCP server (for Claude Code, Cursor, etc.)
|
|
208
235
|
|
|
209
236
|
This package also exposes an MCP server with 3 tools that can be used from Claude Code, Cursor, or any MCP client.
|
|
210
237
|
|
|
211
238
|
### Setup
|
|
212
239
|
|
|
213
|
-
|
|
240
|
+
**Claude Code (one-liner):**
|
|
241
|
+
|
|
242
|
+
```bash
|
|
243
|
+
claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance mcp
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
**Or create `.mcp.json` in your project root:**
|
|
214
247
|
|
|
215
248
|
macOS / Linux / WSL:
|
|
216
249
|
|
|
@@ -250,7 +283,7 @@ Restart your MCP client and approve the server when prompted.
|
|
|
250
283
|
|
|
251
284
|
All tools have [MCP tool annotations](https://modelcontextprotocol.io/specification/2025-11-25/server/tools#annotations) (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) so MCP clients can skip confirmation dialogs for safe operations.
|
|
252
285
|
|
|
253
|
-
## Programmatic
|
|
286
|
+
## Programmatic usage
|
|
254
287
|
|
|
255
288
|
```typescript
|
|
256
289
|
import { runComplianceSuite } from '@yawlabs/mcp-compliance';
|
|
@@ -267,16 +300,48 @@ const report2 = await runComplianceSuite('https://my-server.com/mcp', {
|
|
|
267
300
|
});
|
|
268
301
|
```
|
|
269
302
|
|
|
303
|
+
## Specification
|
|
304
|
+
|
|
305
|
+
The compliance testing methodology is published as an open specification:
|
|
306
|
+
|
|
307
|
+
- **[MCP Compliance Testing Specification](./MCP_COMPLIANCE_SPEC.md)** — test execution model, scoring algorithm, all 43 test rules with pass/fail criteria (CC BY 4.0)
|
|
308
|
+
- **[Machine-readable rule catalog](./mcp-compliance-rules.json)** — JSON Schema-compliant catalog for programmatic consumption
|
|
309
|
+
|
|
310
|
+
These are complementary to (not competing with) the [official MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). The MCP spec defines what servers must do; this spec defines how to verify compliance.
|
|
311
|
+
|
|
270
312
|
## Requirements
|
|
271
313
|
|
|
272
314
|
- Node.js 18+
|
|
273
315
|
|
|
316
|
+
## Contributing
|
|
317
|
+
|
|
318
|
+
```bash
|
|
319
|
+
git clone https://github.com/YawLabs/mcp-compliance.git
|
|
320
|
+
cd mcp-compliance
|
|
321
|
+
npm install
|
|
322
|
+
npm run build
|
|
323
|
+
npm test
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
**Development commands:**
|
|
327
|
+
|
|
328
|
+
| Command | Description |
|
|
329
|
+
|---------|-------------|
|
|
330
|
+
| `npm run build` | Compile with tsup |
|
|
331
|
+
| `npm run dev` | Watch mode |
|
|
332
|
+
| `npm test` | Run tests (vitest) |
|
|
333
|
+
| `npm run lint` | Check with Biome |
|
|
334
|
+
| `npm run lint:fix` | Auto-fix with Biome |
|
|
335
|
+
| `npm run typecheck` | TypeScript type checking |
|
|
336
|
+
| `npm run test:ci` | Build + test (CI-safe) |
|
|
337
|
+
|
|
274
338
|
## Links
|
|
275
339
|
|
|
276
340
|
- [mcp.hosting](https://mcp.hosting) — Hosted MCP server infrastructure
|
|
277
341
|
- [MCP Specification](https://modelcontextprotocol.io/specification/2025-11-25)
|
|
342
|
+
- [MCP Compliance Testing Spec](./MCP_COMPLIANCE_SPEC.md)
|
|
278
343
|
- [Yaw Labs](https://yaw.sh)
|
|
279
344
|
|
|
280
345
|
## License
|
|
281
346
|
|
|
282
|
-
MIT
|
|
347
|
+
MIT — see [LICENSE](./LICENSE).
|
|
@@ -65,7 +65,8 @@ var TEST_DEFINITIONS = [
|
|
|
65
65
|
category: "transport",
|
|
66
66
|
required: true,
|
|
67
67
|
specRef: "basic/transports#streamable-http",
|
|
68
|
-
description: "Verifies the server accepts HTTP POST requests and returns a 2xx status code. This is the fundamental transport requirement for Streamable HTTP MCP servers."
|
|
68
|
+
description: "Verifies the server accepts HTTP POST requests and returns a 2xx status code. This is the fundamental transport requirement for Streamable HTTP MCP servers.",
|
|
69
|
+
recommendation: "Ensure your server listens for POST requests on the MCP endpoint. If you see 401/403, pass --auth with a valid token. Check that the URL is correct and the server is running."
|
|
69
70
|
},
|
|
70
71
|
{
|
|
71
72
|
id: "transport-content-type",
|
|
@@ -73,7 +74,8 @@ var TEST_DEFINITIONS = [
|
|
|
73
74
|
category: "transport",
|
|
74
75
|
required: true,
|
|
75
76
|
specRef: "basic/transports#streamable-http",
|
|
76
|
-
description: "Checks that the server responds with Content-Type application/json or text/event-stream. MCP servers must use one of these two content types."
|
|
77
|
+
description: "Checks that the server responds with Content-Type application/json or text/event-stream. MCP servers must use one of these two content types.",
|
|
78
|
+
recommendation: 'Set the Content-Type response header to "application/json" for synchronous responses or "text/event-stream" for streaming. Do not use text/html or other types.'
|
|
77
79
|
},
|
|
78
80
|
{
|
|
79
81
|
id: "transport-notification-202",
|
|
@@ -81,7 +83,8 @@ var TEST_DEFINITIONS = [
|
|
|
81
83
|
category: "transport",
|
|
82
84
|
required: false,
|
|
83
85
|
specRef: "basic/transports#streamable-http",
|
|
84
|
-
description: "Verifies that sending a JSON-RPC notification (no id field) returns HTTP 202 Accepted with no body. Per spec, servers MUST return 202 for notifications."
|
|
86
|
+
description: "Verifies that sending a JSON-RPC notification (no id field) returns HTTP 202 Accepted with no body. Per spec, servers MUST return 202 for notifications.",
|
|
87
|
+
recommendation: "Detect JSON-RPC messages without an id field and return HTTP 202 with an empty body. Do not attempt to send a JSON-RPC response for notifications."
|
|
85
88
|
},
|
|
86
89
|
{
|
|
87
90
|
id: "transport-session-id",
|
|
@@ -89,7 +92,8 @@ var TEST_DEFINITIONS = [
|
|
|
89
92
|
category: "transport",
|
|
90
93
|
required: false,
|
|
91
94
|
specRef: "basic/transports#streamable-http",
|
|
92
|
-
description: "Tests that the server returns HTTP 400 when MCP-Session-Id header is missing on requests after initialization (when the server issued a session ID)."
|
|
95
|
+
description: "Tests that the server returns HTTP 400 when MCP-Session-Id header is missing on requests after initialization (when the server issued a session ID).",
|
|
96
|
+
recommendation: "If your server issues an MCP-Session-Id header in the initialize response, reject subsequent requests that omit this header with HTTP 400."
|
|
93
97
|
},
|
|
94
98
|
{
|
|
95
99
|
id: "transport-get",
|
|
@@ -97,7 +101,8 @@ var TEST_DEFINITIONS = [
|
|
|
97
101
|
category: "transport",
|
|
98
102
|
required: false,
|
|
99
103
|
specRef: "basic/transports#streamable-http",
|
|
100
|
-
description: "Tests the GET endpoint for server-initiated messages. Server should return text/event-stream or 405 Method Not Allowed."
|
|
104
|
+
description: "Tests the GET endpoint for server-initiated messages. Server should return text/event-stream or 405 Method Not Allowed.",
|
|
105
|
+
recommendation: "If your server supports server-initiated messages, handle GET with text/event-stream. Otherwise, return 405 Method Not Allowed."
|
|
101
106
|
},
|
|
102
107
|
{
|
|
103
108
|
id: "transport-delete",
|
|
@@ -105,7 +110,8 @@ var TEST_DEFINITIONS = [
|
|
|
105
110
|
category: "transport",
|
|
106
111
|
required: false,
|
|
107
112
|
specRef: "basic/transports#streamable-http",
|
|
108
|
-
description: "Tests the DELETE endpoint for session termination. Server should accept the request or return 405 Method Not Allowed."
|
|
113
|
+
description: "Tests the DELETE endpoint for session termination. Server should accept the request or return 405 Method Not Allowed.",
|
|
114
|
+
recommendation: "Handle DELETE requests for session cleanup, or return 405 if session termination is not supported. Do not return 500."
|
|
109
115
|
},
|
|
110
116
|
{
|
|
111
117
|
id: "transport-batch-reject",
|
|
@@ -113,7 +119,8 @@ var TEST_DEFINITIONS = [
|
|
|
113
119
|
category: "transport",
|
|
114
120
|
required: true,
|
|
115
121
|
specRef: "basic/transports#streamable-http",
|
|
116
|
-
description: "Sends a JSON-RPC batch request (array of messages) and verifies the server rejects it with an error. MCP does not support JSON-RPC batch requests."
|
|
122
|
+
description: "Sends a JSON-RPC batch request (array of messages) and verifies the server rejects it with an error. MCP does not support JSON-RPC batch requests.",
|
|
123
|
+
recommendation: "Check if the parsed JSON body is an array. If so, return a JSON-RPC error or HTTP 400. Do not process batch requests \u2014 MCP explicitly forbids them."
|
|
117
124
|
},
|
|
118
125
|
// ── Lifecycle (10 tests) ─────────────────────────────────────────
|
|
119
126
|
{
|
|
@@ -122,7 +129,8 @@ var TEST_DEFINITIONS = [
|
|
|
122
129
|
category: "lifecycle",
|
|
123
130
|
required: true,
|
|
124
131
|
specRef: "basic/lifecycle#initialization",
|
|
125
|
-
description: "Tests the initialize handshake by sending an initialize request with client capabilities. The server must return a result with protocolVersion."
|
|
132
|
+
description: "Tests the initialize handshake by sending an initialize request with client capabilities. The server must return a result with protocolVersion.",
|
|
133
|
+
recommendation: 'Implement the "initialize" method handler. Return a result object with at least protocolVersion, capabilities, and serverInfo fields.'
|
|
126
134
|
},
|
|
127
135
|
{
|
|
128
136
|
id: "lifecycle-proto-version",
|
|
@@ -130,7 +138,8 @@ var TEST_DEFINITIONS = [
|
|
|
130
138
|
category: "lifecycle",
|
|
131
139
|
required: true,
|
|
132
140
|
specRef: "basic/lifecycle#version-negotiation",
|
|
133
|
-
description: "Validates that the protocolVersion returned by the server matches the YYYY-MM-DD date format required by the spec."
|
|
141
|
+
description: "Validates that the protocolVersion returned by the server matches the YYYY-MM-DD date format required by the spec.",
|
|
142
|
+
recommendation: `Return protocolVersion as a YYYY-MM-DD string (e.g., "2025-11-25"). The server should negotiate based on the client's requested version.`
|
|
134
143
|
},
|
|
135
144
|
{
|
|
136
145
|
id: "lifecycle-server-info",
|
|
@@ -138,7 +147,8 @@ var TEST_DEFINITIONS = [
|
|
|
138
147
|
category: "lifecycle",
|
|
139
148
|
required: false,
|
|
140
149
|
specRef: "basic/lifecycle#initialization",
|
|
141
|
-
description: "Checks that the server includes a serverInfo object with at least a name field in its initialize response. While recommended, this is not strictly required."
|
|
150
|
+
description: "Checks that the server includes a serverInfo object with at least a name field in its initialize response. While recommended, this is not strictly required.",
|
|
151
|
+
recommendation: 'Add a serverInfo object to your initialize response: { name: "your-server", version: "1.0.0" }. This helps clients identify your server.'
|
|
142
152
|
},
|
|
143
153
|
{
|
|
144
154
|
id: "lifecycle-capabilities",
|
|
@@ -146,7 +156,8 @@ var TEST_DEFINITIONS = [
|
|
|
146
156
|
category: "lifecycle",
|
|
147
157
|
required: true,
|
|
148
158
|
specRef: "basic/lifecycle#capability-negotiation",
|
|
149
|
-
description: "Verifies the server returns a capabilities object in its initialize response. An empty object is valid (no optional features declared)."
|
|
159
|
+
description: "Verifies the server returns a capabilities object in its initialize response. An empty object is valid (no optional features declared).",
|
|
160
|
+
recommendation: "Include a capabilities object in your initialize response. Declare the features your server supports (tools, resources, prompts, logging, etc.). An empty object {} is valid."
|
|
150
161
|
},
|
|
151
162
|
{
|
|
152
163
|
id: "lifecycle-jsonrpc",
|
|
@@ -154,7 +165,8 @@ var TEST_DEFINITIONS = [
|
|
|
154
165
|
category: "lifecycle",
|
|
155
166
|
required: true,
|
|
156
167
|
specRef: "basic",
|
|
157
|
-
description: 'Validates that the initialize response is a proper JSON-RPC 2.0 message with jsonrpc="2.0", an id field, and either a result or error field.'
|
|
168
|
+
description: 'Validates that the initialize response is a proper JSON-RPC 2.0 message with jsonrpc="2.0", an id field, and either a result or error field.',
|
|
169
|
+
recommendation: 'Ensure every response includes jsonrpc: "2.0", the matching id from the request, and either a result or error field. Never omit the jsonrpc field.'
|
|
158
170
|
},
|
|
159
171
|
{
|
|
160
172
|
id: "lifecycle-ping",
|
|
@@ -162,7 +174,8 @@ var TEST_DEFINITIONS = [
|
|
|
162
174
|
category: "lifecycle",
|
|
163
175
|
required: true,
|
|
164
176
|
specRef: "basic/utilities#ping",
|
|
165
|
-
description: "Tests that the server responds to the ping method with an empty result object. This is a required utility method."
|
|
177
|
+
description: "Tests that the server responds to the ping method with an empty result object. This is a required utility method.",
|
|
178
|
+
recommendation: 'Implement a "ping" method handler that returns an empty result object {}. This is required by the MCP spec for keepalive and connectivity checking.'
|
|
166
179
|
},
|
|
167
180
|
{
|
|
168
181
|
id: "lifecycle-instructions",
|
|
@@ -170,7 +183,8 @@ var TEST_DEFINITIONS = [
|
|
|
170
183
|
category: "lifecycle",
|
|
171
184
|
required: false,
|
|
172
185
|
specRef: "basic/lifecycle#initialization",
|
|
173
|
-
description: "If the server includes an instructions field in the initialize response, validates it is a string. Instructions provide guidance for how the client should interact with the server."
|
|
186
|
+
description: "If the server includes an instructions field in the initialize response, validates it is a string. Instructions provide guidance for how the client should interact with the server.",
|
|
187
|
+
recommendation: "If you include an instructions field in the initialize response, ensure it is a string. Remove the field or fix the type if it is not a string."
|
|
174
188
|
},
|
|
175
189
|
{
|
|
176
190
|
id: "lifecycle-id-match",
|
|
@@ -178,7 +192,8 @@ var TEST_DEFINITIONS = [
|
|
|
178
192
|
category: "lifecycle",
|
|
179
193
|
required: true,
|
|
180
194
|
specRef: "basic",
|
|
181
|
-
description: "Verifies that the JSON-RPC response id matches the request id sent by the client. This is a fundamental JSON-RPC 2.0 requirement."
|
|
195
|
+
description: "Verifies that the JSON-RPC response id matches the request id sent by the client. This is a fundamental JSON-RPC 2.0 requirement.",
|
|
196
|
+
recommendation: "Copy the id field from the request into the response. This is a core JSON-RPC 2.0 requirement. Check that your framework does not modify or discard the request ID."
|
|
182
197
|
},
|
|
183
198
|
{
|
|
184
199
|
id: "lifecycle-logging",
|
|
@@ -186,7 +201,8 @@ var TEST_DEFINITIONS = [
|
|
|
186
201
|
category: "lifecycle",
|
|
187
202
|
required: false,
|
|
188
203
|
specRef: "server/utilities#logging",
|
|
189
|
-
description: "If the server declares logging capability, tests that logging/setLevel method is accepted with a valid log level."
|
|
204
|
+
description: "If the server declares logging capability, tests that logging/setLevel method is accepted with a valid log level.",
|
|
205
|
+
recommendation: 'If you declare logging in capabilities, implement the "logging/setLevel" handler. Accept standard log levels: debug, info, notice, warning, error, critical, alert, emergency.'
|
|
190
206
|
},
|
|
191
207
|
{
|
|
192
208
|
id: "lifecycle-completions",
|
|
@@ -194,7 +210,8 @@ var TEST_DEFINITIONS = [
|
|
|
194
210
|
category: "lifecycle",
|
|
195
211
|
required: false,
|
|
196
212
|
specRef: "server/utilities#completion",
|
|
197
|
-
description: "If the server declares completions capability, tests that the completion/complete method is accepted."
|
|
213
|
+
description: "If the server declares completions capability, tests that the completion/complete method is accepted.",
|
|
214
|
+
recommendation: 'If you declare completions in capabilities, implement the "completion/complete" handler. Return a completion object with a values array, even if empty.'
|
|
198
215
|
},
|
|
199
216
|
// ── Tools (4 tests) ──────────────────────────────────────────────
|
|
200
217
|
{
|
|
@@ -203,7 +220,8 @@ var TEST_DEFINITIONS = [
|
|
|
203
220
|
category: "tools",
|
|
204
221
|
required: false,
|
|
205
222
|
specRef: "server/tools#listing-tools",
|
|
206
|
-
description: "Calls tools/list and validates it returns an array of tool definitions. Required if the server declares tools capability."
|
|
223
|
+
description: "Calls tools/list and validates it returns an array of tool definitions. Required if the server declares tools capability.",
|
|
224
|
+
recommendation: "Implement the tools/list handler to return { tools: [...] } with an array of tool definition objects. Each tool needs at least a name and inputSchema."
|
|
207
225
|
},
|
|
208
226
|
{
|
|
209
227
|
id: "tools-call",
|
|
@@ -211,7 +229,8 @@ var TEST_DEFINITIONS = [
|
|
|
211
229
|
category: "tools",
|
|
212
230
|
required: false,
|
|
213
231
|
specRef: "server/tools#calling-tools",
|
|
214
|
-
description: "Calls the first tool with empty arguments and verifies the response format. Accepts both successful results and InvalidParams errors."
|
|
232
|
+
description: "Calls the first tool with empty arguments and verifies the response format. Accepts both successful results and InvalidParams errors.",
|
|
233
|
+
recommendation: "Ensure tools/call returns { content: [...] } with an array of content objects, each having a type field. Return isError: true for tool execution errors."
|
|
215
234
|
},
|
|
216
235
|
{
|
|
217
236
|
id: "tools-pagination",
|
|
@@ -219,7 +238,8 @@ var TEST_DEFINITIONS = [
|
|
|
219
238
|
category: "tools",
|
|
220
239
|
required: false,
|
|
221
240
|
specRef: "server/tools#listing-tools",
|
|
222
|
-
description: "Tests cursor-based pagination on tools/list. Validates nextCursor is a string if present and that fetching the next page returns a valid response."
|
|
241
|
+
description: "Tests cursor-based pagination on tools/list. Validates nextCursor is a string if present and that fetching the next page returns a valid response.",
|
|
242
|
+
recommendation: "If your server has many tools, include a nextCursor string in the response. Ensure passing this cursor back in a subsequent request returns the next page."
|
|
223
243
|
},
|
|
224
244
|
{
|
|
225
245
|
id: "tools-content-types",
|
|
@@ -227,7 +247,8 @@ var TEST_DEFINITIONS = [
|
|
|
227
247
|
category: "tools",
|
|
228
248
|
required: false,
|
|
229
249
|
specRef: "server/tools#calling-tools",
|
|
230
|
-
description: "Validates that content items returned by tools/call have a recognized type field (text, image, audio, resource, resource_link)."
|
|
250
|
+
description: "Validates that content items returned by tools/call have a recognized type field (text, image, audio, resource, resource_link).",
|
|
251
|
+
recommendation: 'Every content item returned by tools/call must have a type field set to one of: "text", "image", "audio", "resource", or "resource_link". Check for typos or missing type fields.'
|
|
231
252
|
},
|
|
232
253
|
// ── Resources (5 tests) ──────────────────────────────────────────
|
|
233
254
|
{
|
|
@@ -236,7 +257,8 @@ var TEST_DEFINITIONS = [
|
|
|
236
257
|
category: "resources",
|
|
237
258
|
required: false,
|
|
238
259
|
specRef: "server/resources#listing-resources",
|
|
239
|
-
description: "Calls resources/list and validates it returns an array. Required if the server declares resources capability."
|
|
260
|
+
description: "Calls resources/list and validates it returns an array. Required if the server declares resources capability.",
|
|
261
|
+
recommendation: "Implement resources/list to return { resources: [...] } with an array of resource objects. Each resource needs at least a uri and name."
|
|
240
262
|
},
|
|
241
263
|
{
|
|
242
264
|
id: "resources-read",
|
|
@@ -244,7 +266,8 @@ var TEST_DEFINITIONS = [
|
|
|
244
266
|
category: "resources",
|
|
245
267
|
required: false,
|
|
246
268
|
specRef: "server/resources#reading-resources",
|
|
247
|
-
description: "Reads the first resource and validates the response contains a contents array with proper uri and text/blob fields."
|
|
269
|
+
description: "Reads the first resource and validates the response contains a contents array with proper uri and text/blob fields.",
|
|
270
|
+
recommendation: "Implement resources/read to return { contents: [...] } where each item has a uri and either a text or blob field. Ensure the uri matches the requested resource."
|
|
248
271
|
},
|
|
249
272
|
{
|
|
250
273
|
id: "resources-templates",
|
|
@@ -252,7 +275,8 @@ var TEST_DEFINITIONS = [
|
|
|
252
275
|
category: "resources",
|
|
253
276
|
required: false,
|
|
254
277
|
specRef: "server/resources#resource-templates",
|
|
255
|
-
description: "Tests the resource templates endpoint. Accepts Method not found (-32601) since templates are optional."
|
|
278
|
+
description: "Tests the resource templates endpoint. Accepts Method not found (-32601) since templates are optional.",
|
|
279
|
+
recommendation: "If your server supports resource templates, implement resources/templates/list returning { resourceTemplates: [...] }. Otherwise, return error code -32601."
|
|
256
280
|
},
|
|
257
281
|
{
|
|
258
282
|
id: "resources-pagination",
|
|
@@ -260,7 +284,8 @@ var TEST_DEFINITIONS = [
|
|
|
260
284
|
category: "resources",
|
|
261
285
|
required: false,
|
|
262
286
|
specRef: "server/resources#listing-resources",
|
|
263
|
-
description: "Tests cursor-based pagination on resources/list. Validates nextCursor is a string if present and that fetching the next page works."
|
|
287
|
+
description: "Tests cursor-based pagination on resources/list. Validates nextCursor is a string if present and that fetching the next page works.",
|
|
288
|
+
recommendation: "If you return nextCursor in resources/list, ensure it is a string and that passing it back as cursor in the next request returns valid results."
|
|
264
289
|
},
|
|
265
290
|
{
|
|
266
291
|
id: "resources-subscribe",
|
|
@@ -268,7 +293,8 @@ var TEST_DEFINITIONS = [
|
|
|
268
293
|
category: "resources",
|
|
269
294
|
required: false,
|
|
270
295
|
specRef: "server/resources#subscriptions",
|
|
271
|
-
description: "If the server declares resources.subscribe capability, tests that resources/subscribe and resources/unsubscribe methods are accepted."
|
|
296
|
+
description: "If the server declares resources.subscribe capability, tests that resources/subscribe and resources/unsubscribe methods are accepted.",
|
|
297
|
+
recommendation: "If you declare resources.subscribe capability, implement both resources/subscribe and resources/unsubscribe handlers. Both should accept a uri parameter."
|
|
272
298
|
},
|
|
273
299
|
// ── Prompts (3 tests) ────────────────────────────────────────────
|
|
274
300
|
{
|
|
@@ -277,7 +303,8 @@ var TEST_DEFINITIONS = [
|
|
|
277
303
|
category: "prompts",
|
|
278
304
|
required: false,
|
|
279
305
|
specRef: "server/prompts#listing-prompts",
|
|
280
|
-
description: "Calls prompts/list and validates it returns an array. Required if the server declares prompts capability."
|
|
306
|
+
description: "Calls prompts/list and validates it returns an array. Required if the server declares prompts capability.",
|
|
307
|
+
recommendation: "Implement prompts/list to return { prompts: [...] } with an array of prompt objects. Each prompt needs at least a name field."
|
|
281
308
|
},
|
|
282
309
|
{
|
|
283
310
|
id: "prompts-get",
|
|
@@ -285,7 +312,8 @@ var TEST_DEFINITIONS = [
|
|
|
285
312
|
category: "prompts",
|
|
286
313
|
required: false,
|
|
287
314
|
specRef: "server/prompts#getting-a-prompt",
|
|
288
|
-
description: "Gets the first prompt and validates the response contains a messages array with proper role and content fields."
|
|
315
|
+
description: "Gets the first prompt and validates the response contains a messages array with proper role and content fields.",
|
|
316
|
+
recommendation: 'Implement prompts/get to return { messages: [...] } where each message has a role ("user" or "assistant") and a content field.'
|
|
289
317
|
},
|
|
290
318
|
{
|
|
291
319
|
id: "prompts-pagination",
|
|
@@ -293,7 +321,8 @@ var TEST_DEFINITIONS = [
|
|
|
293
321
|
category: "prompts",
|
|
294
322
|
required: false,
|
|
295
323
|
specRef: "server/prompts#listing-prompts",
|
|
296
|
-
description: "Tests cursor-based pagination on prompts/list. Validates nextCursor is a string if present and that fetching the next page works."
|
|
324
|
+
description: "Tests cursor-based pagination on prompts/list. Validates nextCursor is a string if present and that fetching the next page works.",
|
|
325
|
+
recommendation: "If you return nextCursor in prompts/list, ensure it is a string and that passing it back as cursor in the next request returns valid results."
|
|
297
326
|
},
|
|
298
327
|
// ── Error Handling (8 tests) ─────────────────────────────────────
|
|
299
328
|
{
|
|
@@ -302,7 +331,8 @@ var TEST_DEFINITIONS = [
|
|
|
302
331
|
category: "errors",
|
|
303
332
|
required: true,
|
|
304
333
|
specRef: "basic",
|
|
305
|
-
description: "Sends an unknown method and verifies the server returns a JSON-RPC error. The spec requires error code -32601 (Method not found)."
|
|
334
|
+
description: "Sends an unknown method and verifies the server returns a JSON-RPC error. The spec requires error code -32601 (Method not found).",
|
|
335
|
+
recommendation: "Return a JSON-RPC error with code -32601 (Method not found) for any unrecognized method name. Do not silently ignore unknown methods."
|
|
306
336
|
},
|
|
307
337
|
{
|
|
308
338
|
id: "error-method-code",
|
|
@@ -310,7 +340,8 @@ var TEST_DEFINITIONS = [
|
|
|
310
340
|
category: "errors",
|
|
311
341
|
required: false,
|
|
312
342
|
specRef: "basic",
|
|
313
|
-
description: "Checks the error code is specifically -32601 (Method not found) for unknown methods, as required by JSON-RPC 2.0."
|
|
343
|
+
description: "Checks the error code is specifically -32601 (Method not found) for unknown methods, as required by JSON-RPC 2.0.",
|
|
344
|
+
recommendation: "Use exactly error code -32601 for unknown methods. Do not use generic error codes like -32000. This is required by JSON-RPC 2.0."
|
|
314
345
|
},
|
|
315
346
|
{
|
|
316
347
|
id: "error-invalid-jsonrpc",
|
|
@@ -318,7 +349,8 @@ var TEST_DEFINITIONS = [
|
|
|
318
349
|
category: "errors",
|
|
319
350
|
required: true,
|
|
320
351
|
specRef: "basic",
|
|
321
|
-
description: "Sends a malformed JSON-RPC message (missing required fields) and verifies the server returns an error or 4xx status."
|
|
352
|
+
description: "Sends a malformed JSON-RPC message (missing required fields) and verifies the server returns an error or 4xx status.",
|
|
353
|
+
recommendation: "Validate incoming JSON-RPC messages for required fields (jsonrpc, method). Return error code -32600 (Invalid Request) or HTTP 400 for malformed messages."
|
|
322
354
|
},
|
|
323
355
|
{
|
|
324
356
|
id: "error-invalid-json",
|
|
@@ -326,7 +358,8 @@ var TEST_DEFINITIONS = [
|
|
|
326
358
|
category: "errors",
|
|
327
359
|
required: false,
|
|
328
360
|
specRef: "basic",
|
|
329
|
-
description: "Sends invalid JSON and verifies the server returns a parse error (-32700) or 4xx status code."
|
|
361
|
+
description: "Sends invalid JSON and verifies the server returns a parse error (-32700) or 4xx status code.",
|
|
362
|
+
recommendation: "Catch JSON parse errors and return error code -32700 (Parse error) with a descriptive message. Do not return 500 for malformed input."
|
|
330
363
|
},
|
|
331
364
|
{
|
|
332
365
|
id: "error-missing-params",
|
|
@@ -334,7 +367,8 @@ var TEST_DEFINITIONS = [
|
|
|
334
367
|
category: "errors",
|
|
335
368
|
required: false,
|
|
336
369
|
specRef: "server/tools#error-handling",
|
|
337
|
-
description: "Calls tools/call with an empty params object (missing required name field) and verifies an error is returned."
|
|
370
|
+
description: "Calls tools/call with an empty params object (missing required name field) and verifies an error is returned.",
|
|
371
|
+
recommendation: "Validate tools/call params and return error code -32602 (Invalid params) when the required name field is missing."
|
|
338
372
|
},
|
|
339
373
|
{
|
|
340
374
|
id: "error-parse-code",
|
|
@@ -342,7 +376,8 @@ var TEST_DEFINITIONS = [
|
|
|
342
376
|
category: "errors",
|
|
343
377
|
required: false,
|
|
344
378
|
specRef: "basic",
|
|
345
|
-
description: "Checks that the server returns the specific JSON-RPC error code -32700 (Parse error) when receiving invalid JSON, as required by the JSON-RPC 2.0 specification."
|
|
379
|
+
description: "Checks that the server returns the specific JSON-RPC error code -32700 (Parse error) when receiving invalid JSON, as required by the JSON-RPC 2.0 specification.",
|
|
380
|
+
recommendation: "Return exactly error code -32700 for JSON parse failures. Most JSON-RPC frameworks handle this automatically \u2014 check yours does not override the code."
|
|
346
381
|
},
|
|
347
382
|
{
|
|
348
383
|
id: "error-invalid-request-code",
|
|
@@ -350,7 +385,8 @@ var TEST_DEFINITIONS = [
|
|
|
350
385
|
category: "errors",
|
|
351
386
|
required: false,
|
|
352
387
|
specRef: "basic",
|
|
353
|
-
description: "Checks that the server returns the specific JSON-RPC error code -32600 (Invalid Request) for malformed JSON-RPC messages missing required fields."
|
|
388
|
+
description: "Checks that the server returns the specific JSON-RPC error code -32600 (Invalid Request) for malformed JSON-RPC messages missing required fields.",
|
|
389
|
+
recommendation: "Return exactly error code -32600 for structurally invalid JSON-RPC messages (e.g., missing method field). Check your JSON-RPC middleware configuration."
|
|
354
390
|
},
|
|
355
391
|
{
|
|
356
392
|
id: "tools-call-unknown",
|
|
@@ -358,7 +394,8 @@ var TEST_DEFINITIONS = [
|
|
|
358
394
|
category: "errors",
|
|
359
395
|
required: false,
|
|
360
396
|
specRef: "server/tools#error-handling",
|
|
361
|
-
description: "Calls tools/call with a nonexistent tool name and verifies the server returns an error response."
|
|
397
|
+
description: "Calls tools/call with a nonexistent tool name and verifies the server returns an error response.",
|
|
398
|
+
recommendation: "Return a JSON-RPC error or set isError: true when tools/call receives an unrecognized tool name. Do not return an empty success response."
|
|
362
399
|
},
|
|
363
400
|
// ── Schema Validation (6 tests) ──────────────────────────────────
|
|
364
401
|
{
|
|
@@ -367,7 +404,8 @@ var TEST_DEFINITIONS = [
|
|
|
367
404
|
category: "schema",
|
|
368
405
|
required: false,
|
|
369
406
|
specRef: "server/tools#data-types",
|
|
370
|
-
description: 'Validates every tool has a valid name (1-128 chars, alphanumeric/underscore/hyphen/dot) and a required inputSchema of type "object".'
|
|
407
|
+
description: 'Validates every tool has a valid name (1-128 chars, alphanumeric/underscore/hyphen/dot) and a required inputSchema of type "object".',
|
|
408
|
+
recommendation: 'Ensure every tool has a name (1-128 chars, [A-Za-z0-9_.-]) and an inputSchema with type: "object". Add descriptions to tools for better AI assistant integration.'
|
|
371
409
|
},
|
|
372
410
|
{
|
|
373
411
|
id: "tools-annotations",
|
|
@@ -375,7 +413,8 @@ var TEST_DEFINITIONS = [
|
|
|
375
413
|
category: "schema",
|
|
376
414
|
required: false,
|
|
377
415
|
specRef: "server/tools#annotations",
|
|
378
|
-
description: "Validates tool annotation fields if present: readOnlyHint, destructiveHint, idempotentHint, openWorldHint should be booleans; title should be a string."
|
|
416
|
+
description: "Validates tool annotation fields if present: readOnlyHint, destructiveHint, idempotentHint, openWorldHint should be booleans; title should be a string.",
|
|
417
|
+
recommendation: "If you include annotations on tools, ensure readOnlyHint, destructiveHint, idempotentHint, and openWorldHint are booleans. Title must be a string."
|
|
379
418
|
},
|
|
380
419
|
{
|
|
381
420
|
id: "tools-title-field",
|
|
@@ -383,7 +422,8 @@ var TEST_DEFINITIONS = [
|
|
|
383
422
|
category: "schema",
|
|
384
423
|
required: false,
|
|
385
424
|
specRef: "server/tools#data-types",
|
|
386
|
-
description: "Checks if tools include the optional title field for human-readable display names. Added in spec version 2025-11-25."
|
|
425
|
+
description: "Checks if tools include the optional title field for human-readable display names. Added in spec version 2025-11-25.",
|
|
426
|
+
recommendation: "Add a title field (human-readable string) to each tool definition. This helps MCP clients display your tools in a user-friendly way."
|
|
387
427
|
},
|
|
388
428
|
{
|
|
389
429
|
id: "tools-output-schema",
|
|
@@ -391,7 +431,8 @@ var TEST_DEFINITIONS = [
|
|
|
391
431
|
category: "schema",
|
|
392
432
|
required: false,
|
|
393
433
|
specRef: "server/tools#structured-content",
|
|
394
|
-
description: 'If tools declare an outputSchema, validates it is a valid JSON Schema object with type "object". Used for structured output validation.'
|
|
434
|
+
description: 'If tools declare an outputSchema, validates it is a valid JSON Schema object with type "object". Used for structured output validation.',
|
|
435
|
+
recommendation: 'If you declare outputSchema on a tool, ensure it is a valid JSON Schema object with type: "object". Remove outputSchema if you do not need structured output.'
|
|
395
436
|
},
|
|
396
437
|
{
|
|
397
438
|
id: "prompts-schema",
|
|
@@ -399,7 +440,8 @@ var TEST_DEFINITIONS = [
|
|
|
399
440
|
category: "schema",
|
|
400
441
|
required: false,
|
|
401
442
|
specRef: "server/prompts#data-types",
|
|
402
|
-
description: "Validates every prompt has a name and that any arguments array contains items with name fields."
|
|
443
|
+
description: "Validates every prompt has a name and that any arguments array contains items with name fields.",
|
|
444
|
+
recommendation: "Ensure every prompt has a name field. If the prompt has arguments, each argument object must include a name field."
|
|
403
445
|
},
|
|
404
446
|
{
|
|
405
447
|
id: "resources-schema",
|
|
@@ -407,7 +449,8 @@ var TEST_DEFINITIONS = [
|
|
|
407
449
|
category: "schema",
|
|
408
450
|
required: false,
|
|
409
451
|
specRef: "server/resources#data-types",
|
|
410
|
-
description: "Validates every resource has a valid URI (parseable as a URL) and a name field."
|
|
452
|
+
description: "Validates every resource has a valid URI (parseable as a URL) and a name field.",
|
|
453
|
+
recommendation: "Ensure every resource has a valid, parseable URI and a name field. Add description and mimeType for better client integration."
|
|
411
454
|
}
|
|
412
455
|
];
|
|
413
456
|
|