@yawlabs/mcp-compliance 0.13.0 → 0.13.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +602 -591
- package/dist/{chunk-G5K7CRWU.js → chunk-X5CVUDPW.js} +65 -27
- package/dist/index.js +124 -38
- package/dist/mcp/server.js +12 -4
- package/dist/runner.d.ts +9 -1
- package/dist/runner.js +3 -1
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -1,591 +1,602 @@
|
|
|
1
|
-
# @yawlabs/mcp-compliance
|
|
2
|
-
|
|
3
|
-
[](https://www.npmjs.com/package/@yawlabs/mcp-compliance)
|
|
4
|
-
[](https://opensource.org/licenses/MIT)
|
|
5
|
-
[](https://github.com/YawLabs/mcp-compliance/stargazers)
|
|
6
|
-
[](https://github.com/YawLabs/mcp-compliance/actions/workflows/ci.yml)
|
|
7
|
-
|
|
8
|
-
**Test any MCP server for spec compliance.** 88-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the [MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). Works against **HTTP endpoints** (`https://my-server.com/mcp`) and **stdio servers** (`npx @modelcontextprotocol/server-filesystem /tmp`) alike. CLI, MCP server, and programmatic API.
|
|
9
|
-
|
|
10
|
-
Built and maintained by [Yaw Labs](https://yaw.sh).
|
|
11
|
-
|
|
12
|
-
## Why this tool?
|
|
13
|
-
|
|
14
|
-
MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.
|
|
15
|
-
|
|
16
|
-
This tool solves that:
|
|
17
|
-
|
|
18
|
-
- **88 tests across 8 categories** — transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. No gaps. (HTTP runs all 85 transport-applicable tests; stdio runs ~75 — HTTP-specific tests like CORS, TLS, session headers, and rate limiting are gated out.)
|
|
19
|
-
- **Capability-driven** — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
|
|
20
|
-
- **Graded scoring** — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
|
|
21
|
-
- **CI-ready** — `--strict` mode exits with code 1 on required test failures. Drop it into any pipeline.
|
|
22
|
-
- **Spec-referenced** — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
|
|
23
|
-
- **Three interfaces** — CLI for humans, MCP server for AI assistants, programmatic API for integration.
|
|
24
|
-
- **Published
|
|
25
|
-
|
|
26
|
-
## Quick start
|
|
27
|
-
|
|
28
|
-
**Remote HTTP server:**
|
|
29
|
-
|
|
30
|
-
```bash
|
|
31
|
-
npx @yawlabs/mcp-compliance test https://my-server.com/mcp
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
**Local stdio server** (the vast majority of MCP servers on npm):
|
|
35
|
-
|
|
36
|
-
```bash
|
|
37
|
-
# Pass the command directly, Inspector-style
|
|
38
|
-
npx @yawlabs/mcp-compliance test npx @modelcontextprotocol/server-filesystem /tmp
|
|
39
|
-
|
|
40
|
-
# Or a local build
|
|
41
|
-
npx @yawlabs/mcp-compliance test node ./dist/server.js
|
|
42
|
-
|
|
43
|
-
# With env vars
|
|
44
|
-
npx @yawlabs/mcp-compliance test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
**Install globally:**
|
|
48
|
-
|
|
49
|
-
```bash
|
|
50
|
-
npm install -g @yawlabs/mcp-compliance
|
|
51
|
-
mcp-compliance test https://my-server.com/mcp
|
|
52
|
-
```
|
|
53
|
-
|
|
54
|
-
That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.
|
|
55
|
-
|
|
56
|
-
## CLI usage
|
|
57
|
-
|
|
58
|
-
### HTTP targets
|
|
59
|
-
|
|
60
|
-
```bash
|
|
61
|
-
# Terminal output with colors and grade
|
|
62
|
-
mcp-compliance test https://my-server.com/mcp
|
|
63
|
-
|
|
64
|
-
# JSON / SARIF for scripting + GitHub Code Scanning
|
|
65
|
-
mcp-compliance test https://my-server.com/mcp --format json
|
|
66
|
-
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif
|
|
67
|
-
|
|
68
|
-
# Strict mode for CI — exits 1 on required-test failure
|
|
69
|
-
mcp-compliance test https://my-server.com/mcp --strict
|
|
70
|
-
|
|
71
|
-
# Auth (shorthand or full header)
|
|
72
|
-
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"
|
|
73
|
-
mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"
|
|
74
|
-
|
|
75
|
-
# Focus the run
|
|
76
|
-
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
|
|
77
|
-
mcp-compliance test https://my-server.com/mcp --skip prompts,resources
|
|
78
|
-
mcp-compliance test https://my-server.com/mcp --verbose
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
### stdio targets
|
|
82
|
-
|
|
83
|
-
Pass the command and its args as positional arguments (MCP Inspector-style). Use `--` to disambiguate when the target needs flags that collide with ours.
|
|
84
|
-
|
|
85
|
-
```bash
|
|
86
|
-
# npm-distributed stdio servers
|
|
87
|
-
mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp
|
|
88
|
-
mcp-compliance test uvx mcp-server-git
|
|
89
|
-
|
|
90
|
-
# Local build
|
|
91
|
-
mcp-compliance test node ./dist/server.js
|
|
92
|
-
|
|
93
|
-
# With env vars (repeatable -E, or --env-file)
|
|
94
|
-
mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server
|
|
95
|
-
mcp-compliance test --env-file .env -- node ./server.js
|
|
96
|
-
|
|
97
|
-
# Set working directory
|
|
98
|
-
mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js
|
|
99
|
-
|
|
100
|
-
# Target uses a flag that collides with ours — use `--` to separate
|
|
101
|
-
mcp-compliance test --verbose -- node ./server.js --verbose
|
|
102
|
-
```
|
|
103
|
-
|
|
104
|
-
On Windows, `npx` and other `.cmd` shims are handled automatically by spawning through the shell.
|
|
105
|
-
|
|
106
|
-
### Options
|
|
107
|
-
|
|
108
|
-
| Option | Applies to | Description |
|
|
109
|
-
|--------|-----------|-------------|
|
|
110
|
-
| `--format <format>` | both | Output format: `terminal`, `json`, `sarif`, `github`, or `markdown` (default: `terminal`) |
|
|
111
|
-
| `--config <path>` | both | Load defaults from a config file (default: `mcp-compliance.config.json` in cwd) |
|
|
112
|
-
| `--output <file>` | both | Write a local SVG badge to the given path after the run |
|
|
113
|
-
| `--list` | both | Print test IDs that would run given current filters, then exit (no connection) |
|
|
114
|
-
| `--transport <kind>` | both | Filter by `http` or `stdio` (only used with `--list` when no target is provided) |
|
|
115
|
-
| `--strict` | both | Exit with code 1 on any required test failure (for CI) |
|
|
116
|
-
| `--min-grade <grade>` | both | Exit with code 1 if grade is below this threshold (`A`–`F`) |
|
|
117
|
-
| `-H, --header <h>` | HTTP | Add header to all requests, format `"Key: Value"` (repeatable) |
|
|
118
|
-
| `--auth <token>` | HTTP | Shorthand for `-H "Authorization: <token>"` |
|
|
119
|
-
| `-E, --env <var>` | stdio | Set env var for stdio command, format `"KEY=VALUE"` (repeatable) |
|
|
120
|
-
| `--env-file <path>` | stdio | Load env vars from a file (one `KEY=VALUE` per line) |
|
|
121
|
-
| `--cwd <dir>` | stdio | Working directory for the stdio command |
|
|
122
|
-
| `--timeout <ms>` | both | Request timeout in milliseconds (default: `15000`) |
|
|
123
|
-
| `--preflight-timeout <ms>` | HTTP | Preflight connectivity check timeout (HTTP only) |
|
|
124
|
-
| `--retries <n>` | both | Number of retries for failed tests (default: `0`) |
|
|
125
|
-
| `--only <items>` | both | Only run tests matching these categories or test IDs (comma-separated) |
|
|
126
|
-
| `--skip <items>` | both | Skip tests matching these categories or test IDs (comma-separated) |
|
|
127
|
-
| `--
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
mcp-compliance
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
"
|
|
196
|
-
"
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
"
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
"
|
|
210
|
-
"
|
|
211
|
-
"
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
"
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
|
232
|
-
| `--
|
|
233
|
-
| `--
|
|
234
|
-
| `--
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
mcp-compliance
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
<
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
-
|
|
268
|
-
- **transport-
|
|
269
|
-
- **transport-content-type
|
|
270
|
-
- **transport-
|
|
271
|
-
- **transport-
|
|
272
|
-
- **transport-
|
|
273
|
-
- **transport-
|
|
274
|
-
- **transport-
|
|
275
|
-
- **transport-
|
|
276
|
-
- **transport-
|
|
277
|
-
- **transport-
|
|
278
|
-
- **transport-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
- **
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
- **lifecycle-
|
|
293
|
-
- **lifecycle-
|
|
294
|
-
- **lifecycle-
|
|
295
|
-
- **lifecycle-
|
|
296
|
-
- **lifecycle-
|
|
297
|
-
- **lifecycle-
|
|
298
|
-
- **lifecycle-
|
|
299
|
-
- **lifecycle-
|
|
300
|
-
- **lifecycle-
|
|
301
|
-
- **lifecycle-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
- **
|
|
309
|
-
- **
|
|
310
|
-
- **
|
|
311
|
-
- **
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
<
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
- **
|
|
320
|
-
- **
|
|
321
|
-
- **
|
|
322
|
-
- **
|
|
323
|
-
|
|
324
|
-
</details>
|
|
325
|
-
|
|
326
|
-
<details>
|
|
327
|
-
<summary><strong>
|
|
328
|
-
|
|
329
|
-
- **
|
|
330
|
-
- **
|
|
331
|
-
- **
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
- **
|
|
341
|
-
- **
|
|
342
|
-
- **
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
- **
|
|
355
|
-
- **
|
|
356
|
-
- **tools-
|
|
357
|
-
- **
|
|
358
|
-
- **
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
<
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
- **
|
|
367
|
-
- **
|
|
368
|
-
- **
|
|
369
|
-
- **
|
|
370
|
-
- **
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
- **security-
|
|
378
|
-
- **security-
|
|
379
|
-
- **security-
|
|
380
|
-
- **security-
|
|
381
|
-
- **security-
|
|
382
|
-
- **security-
|
|
383
|
-
- **security-
|
|
384
|
-
- **security-
|
|
385
|
-
- **security-
|
|
386
|
-
- **security-
|
|
387
|
-
- **security-
|
|
388
|
-
- **security-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
#
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
{
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
}
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
-
|
|
546
|
-
-
|
|
547
|
-
-
|
|
548
|
-
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
1
|
+
# @yawlabs/mcp-compliance
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/@yawlabs/mcp-compliance)
|
|
4
|
+
[](https://opensource.org/licenses/MIT)
|
|
5
|
+
[](https://github.com/YawLabs/mcp-compliance/stargazers)
|
|
6
|
+
[](https://github.com/YawLabs/mcp-compliance/actions/workflows/ci.yml)
|
|
7
|
+
|
|
8
|
+
**Test any MCP server for spec compliance.** 88-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the [MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). Works against **HTTP endpoints** (`https://my-server.com/mcp`) and **stdio servers** (`npx @modelcontextprotocol/server-filesystem /tmp`) alike. CLI, MCP server, and programmatic API.
|
|
9
|
+
|
|
10
|
+
Built and maintained by [Yaw Labs](https://yaw.sh).
|
|
11
|
+
|
|
12
|
+
## Why this tool?
|
|
13
|
+
|
|
14
|
+
MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.
|
|
15
|
+
|
|
16
|
+
This tool solves that:
|
|
17
|
+
|
|
18
|
+
- **88 tests across 8 categories** — transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. No gaps. (HTTP runs all 85 transport-applicable tests; stdio runs ~75 — HTTP-specific tests like CORS, TLS, session headers, and rate limiting are gated out.)
|
|
19
|
+
- **Capability-driven** — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
|
|
20
|
+
- **Graded scoring** — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
|
|
21
|
+
- **CI-ready** — `--strict` mode exits with code 1 on required test failures. Drop it into any pipeline.
|
|
22
|
+
- **Spec-referenced** — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
|
|
23
|
+
- **Three interfaces** — CLI for humans, MCP server for AI assistants, programmatic API for integration.
|
|
24
|
+
- **Published methodology** — the [testing methodology](./COMPLIANCE_RUBRIC.md) and [rule catalog](./mcp-compliance-rules.json) are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules.
|
|
25
|
+
|
|
26
|
+
## Quick start
|
|
27
|
+
|
|
28
|
+
**Remote HTTP server:**
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
npx @yawlabs/mcp-compliance test https://my-server.com/mcp
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
**Local stdio server** (the vast majority of MCP servers on npm):
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
# Pass the command directly, Inspector-style
|
|
38
|
+
npx @yawlabs/mcp-compliance test npx @modelcontextprotocol/server-filesystem /tmp
|
|
39
|
+
|
|
40
|
+
# Or a local build
|
|
41
|
+
npx @yawlabs/mcp-compliance test node ./dist/server.js
|
|
42
|
+
|
|
43
|
+
# With env vars
|
|
44
|
+
npx @yawlabs/mcp-compliance test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Install globally:**
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npm install -g @yawlabs/mcp-compliance
|
|
51
|
+
mcp-compliance test https://my-server.com/mcp
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.
|
|
55
|
+
|
|
56
|
+
## CLI usage
|
|
57
|
+
|
|
58
|
+
### HTTP targets
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
# Terminal output with colors and grade
|
|
62
|
+
mcp-compliance test https://my-server.com/mcp
|
|
63
|
+
|
|
64
|
+
# JSON / SARIF for scripting + GitHub Code Scanning
|
|
65
|
+
mcp-compliance test https://my-server.com/mcp --format json
|
|
66
|
+
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif
|
|
67
|
+
|
|
68
|
+
# Strict mode for CI — exits 1 on required-test failure
|
|
69
|
+
mcp-compliance test https://my-server.com/mcp --strict
|
|
70
|
+
|
|
71
|
+
# Auth (shorthand or full header)
|
|
72
|
+
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"
|
|
73
|
+
mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"
|
|
74
|
+
|
|
75
|
+
# Focus the run
|
|
76
|
+
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
|
|
77
|
+
mcp-compliance test https://my-server.com/mcp --skip prompts,resources
|
|
78
|
+
mcp-compliance test https://my-server.com/mcp --verbose
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### stdio targets
|
|
82
|
+
|
|
83
|
+
Pass the command and its args as positional arguments (MCP Inspector-style). Use `--` to disambiguate when the target needs flags that collide with ours.
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
# npm-distributed stdio servers
|
|
87
|
+
mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp
|
|
88
|
+
mcp-compliance test uvx mcp-server-git
|
|
89
|
+
|
|
90
|
+
# Local build
|
|
91
|
+
mcp-compliance test node ./dist/server.js
|
|
92
|
+
|
|
93
|
+
# With env vars (repeatable -E, or --env-file)
|
|
94
|
+
mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server
|
|
95
|
+
mcp-compliance test --env-file .env -- node ./server.js
|
|
96
|
+
|
|
97
|
+
# Set working directory
|
|
98
|
+
mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js
|
|
99
|
+
|
|
100
|
+
# Target uses a flag that collides with ours — use `--` to separate
|
|
101
|
+
mcp-compliance test --verbose -- node ./server.js --verbose
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
On Windows, `npx` and other `.cmd` shims are handled automatically by spawning through the shell.
|
|
105
|
+
|
|
106
|
+
### Options
|
|
107
|
+
|
|
108
|
+
| Option | Applies to | Description |
|
|
109
|
+
|--------|-----------|-------------|
|
|
110
|
+
| `--format <format>` | both | Output format: `terminal`, `json`, `sarif`, `github`, or `markdown` (default: `terminal`) |
|
|
111
|
+
| `--config <path>` | both | Load defaults from a config file (default: `mcp-compliance.config.json` in cwd) |
|
|
112
|
+
| `--output <file>` | both | Write a local SVG badge to the given path after the run |
|
|
113
|
+
| `--list` | both | Print test IDs that would run given current filters, then exit (no connection) |
|
|
114
|
+
| `--transport <kind>` | both | Filter by `http` or `stdio` (only used with `--list` when no target is provided) |
|
|
115
|
+
| `--strict` | both | Exit with code 1 on any required test failure (for CI) |
|
|
116
|
+
| `--min-grade <grade>` | both | Exit with code 1 if grade is below this threshold (`A`–`F`) |
|
|
117
|
+
| `-H, --header <h>` | HTTP | Add header to all requests, format `"Key: Value"` (repeatable) |
|
|
118
|
+
| `--auth <token>` | HTTP | Shorthand for `-H "Authorization: <token>"` |
|
|
119
|
+
| `-E, --env <var>` | stdio | Set env var for stdio command, format `"KEY=VALUE"` (repeatable) |
|
|
120
|
+
| `--env-file <path>` | stdio | Load env vars from a file (one `KEY=VALUE` per line) |
|
|
121
|
+
| `--cwd <dir>` | stdio | Working directory for the stdio command |
|
|
122
|
+
| `--timeout <ms>` | both | Request timeout in milliseconds (default: `15000`) |
|
|
123
|
+
| `--preflight-timeout <ms>` | HTTP | Preflight connectivity check timeout (HTTP only) |
|
|
124
|
+
| `--retries <n>` | both | Number of retries for failed tests (default: `0`) |
|
|
125
|
+
| `--only <items>` | both | Only run tests matching these categories or test IDs (comma-separated) |
|
|
126
|
+
| `--skip <items>` | both | Skip tests matching these categories or test IDs (comma-separated) |
|
|
127
|
+
| `--concurrency <n>` | both | Max parallel-safe tests in flight (default: `1`; raising reduces wall time but can perturb timing-sensitive servers) |
|
|
128
|
+
| `--verbose` | both | Print each test result as it runs (also forwards stdio stderr) |
|
|
129
|
+
|
|
130
|
+
### CI integration
|
|
131
|
+
|
|
132
|
+
**GitHub Action** (drop into any `.github/workflows/*.yml`):
|
|
133
|
+
|
|
134
|
+
```yaml
|
|
135
|
+
- uses: YawLabs/mcp-compliance@v0
|
|
136
|
+
with:
|
|
137
|
+
target: 'node ./dist/server.js' # or a URL like https://my-server.com/mcp
|
|
138
|
+
format: github # ::error / ::warning annotations on the PR
|
|
139
|
+
strict: 'true' # exit non-zero if any required test fails
|
|
140
|
+
min-grade: 'A' # also exit if grade slips
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
**Manual CLI invocation:**
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
# GitHub Actions: emits ::error / ::warning annotations inline on the PR
|
|
147
|
+
mcp-compliance test https://my-server.com/mcp --format github --strict
|
|
148
|
+
|
|
149
|
+
# Slack/Linear/PR comment: drop the body straight into a comment
|
|
150
|
+
mcp-compliance test https://my-server.com/mcp --format markdown > report.md
|
|
151
|
+
|
|
152
|
+
# HTML report (self-contained, share anywhere — issue comments, S3, GitHub Pages)
|
|
153
|
+
mcp-compliance test https://my-server.com/mcp --format html > report.html
|
|
154
|
+
|
|
155
|
+
# Block release if grade slips below B
|
|
156
|
+
mcp-compliance test https://my-server.com/mcp --min-grade B
|
|
157
|
+
|
|
158
|
+
# Preview which tests will run before connecting (handy for --only/--skip authoring)
|
|
159
|
+
mcp-compliance test --list --transport stdio --skip security
|
|
160
|
+
|
|
161
|
+
# Diff two runs — exit 1 if anything that was passing is now failing
|
|
162
|
+
mcp-compliance test https://my-server.com/mcp --format json > current.json
|
|
163
|
+
mcp-compliance diff baseline.json current.json
|
|
164
|
+
|
|
165
|
+
# Watch mode for stdio dev loop — re-runs on file changes in cwd
|
|
166
|
+
mcp-compliance test --watch -- node ./dist/server.js
|
|
167
|
+
|
|
168
|
+
# Latency benchmark
|
|
169
|
+
mcp-compliance benchmark -- node ./dist/server.js -r 200 -c 4
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
**Docker:**
|
|
173
|
+
|
|
174
|
+
```bash
|
|
175
|
+
docker run --rm ghcr.io/yawlabs/mcp-compliance test https://my-server.com/mcp
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Scaffold a config
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
mcp-compliance init
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Interactive prompts walk you through transport (http/stdio), command/url, env vars, timeout, and strict mode — then write a `mcp-compliance.config.json` you can commit.
|
|
185
|
+
|
|
186
|
+
### Config file
|
|
187
|
+
|
|
188
|
+
Check in a `mcp-compliance.config.json` so CI and your dev loop can run `mcp-compliance test` with no arguments. Supported locations (searched in order): `mcp-compliance.config.json`, `.mcp-compliancerc.json`, `.mcp-compliancerc`, and the `"mcp-compliance"` field of `package.json`. Pass `--config <path>` to load an explicit file.
|
|
189
|
+
|
|
190
|
+
**HTTP:**
|
|
191
|
+
|
|
192
|
+
```json
|
|
193
|
+
{
|
|
194
|
+
"target": {
|
|
195
|
+
"type": "http",
|
|
196
|
+
"url": "https://my-server.com/mcp",
|
|
197
|
+
"headers": { "Authorization": "Bearer tok123" }
|
|
198
|
+
},
|
|
199
|
+
"timeout": 20000,
|
|
200
|
+
"strict": true
|
|
201
|
+
}
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
**stdio:**
|
|
205
|
+
|
|
206
|
+
```json
|
|
207
|
+
{
|
|
208
|
+
"target": {
|
|
209
|
+
"type": "stdio",
|
|
210
|
+
"command": "node",
|
|
211
|
+
"args": ["./dist/server.js"],
|
|
212
|
+
"env": { "LOG_LEVEL": "error" }
|
|
213
|
+
},
|
|
214
|
+
"skip": ["security"],
|
|
215
|
+
"strict": true
|
|
216
|
+
}
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
Precedence: CLI flags > config file > defaults. Any field can be overridden on the command line.
|
|
220
|
+
|
|
221
|
+
### Publish a shareable badge (HTTP only)
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
mcp-compliance badge https://my-server.com/mcp
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Runs the compliance suite, publishes the report to [mcp.hosting](https://mcp.hosting), and prints the markdown embed for your README. The badge image reflects the real grade (A–F) and links to the full report.
|
|
228
|
+
|
|
229
|
+
| Option | Description |
|
|
230
|
+
|--------|-------------|
|
|
231
|
+
| `-H, --header <header>` | Add header to all requests, format `"Key: Value"` (repeatable) |
|
|
232
|
+
| `--auth <token>` | Shorthand for `-H "Authorization: <token>"` |
|
|
233
|
+
| `--timeout <ms>` | Request timeout in milliseconds (default: `15000`) |
|
|
234
|
+
| `--no-publish` | Skip publishing; print a local badge markdown only |
|
|
235
|
+
| `--output <file>` | Also write a local SVG badge to the given path |
|
|
236
|
+
|
|
237
|
+
Reports are kept for 90 days from last submission; resubmitting the same URL overwrites the previous report. Auth headers are stripped client-side before upload. Private/loopback URLs (`localhost`, `127.0.0.1`, `192.168.*`, etc.) trigger an interactive confirmation before publishing, and are rejected by the server in any case.
|
|
238
|
+
|
|
239
|
+
A delete token is returned at publish time and stored at `~/.mcp-compliance/tokens.json` (mode `0600`). Use it to take a report down:
|
|
240
|
+
|
|
241
|
+
```bash
|
|
242
|
+
mcp-compliance unpublish https://my-server.com/mcp
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
### Local SVG badge (any transport)
|
|
246
|
+
|
|
247
|
+
Stdio servers can't be published (no public URL to key on), but you can commit a local SVG reflecting the real grade:
|
|
248
|
+
|
|
249
|
+
```bash
|
|
250
|
+
mcp-compliance test node ./dist/server.js --output badge.svg
|
|
251
|
+
mcp-compliance badge npx -y @modelcontextprotocol/server-filesystem /tmp --output badge.svg
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Then embed it in your README:
|
|
255
|
+
|
|
256
|
+
```markdown
|
|
257
|
+

|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
The `test` command never publishes — use it for CI, debugging, and local iteration. `badge` is the only command that publishes to mcp.hosting.
|
|
261
|
+
|
|
262
|
+
## What the 88 tests check
|
|
263
|
+
|
|
264
|
+
<details>
|
|
265
|
+
<summary><strong>Transport (16 tests)</strong></summary>
|
|
266
|
+
|
|
267
|
+
HTTP-only (13):
|
|
268
|
+
- **transport-post** — Server accepts HTTP POST requests (required)
|
|
269
|
+
- **transport-content-type** — Responds with application/json or text/event-stream (required)
|
|
270
|
+
- **transport-notification-202** — Notifications return exactly 202 Accepted
|
|
271
|
+
- **transport-content-type-reject** — Rejects non-JSON request Content-Type
|
|
272
|
+
- **transport-session-id** — Enforces MCP-Session-Id after initialization
|
|
273
|
+
- **transport-session-invalid** — Returns 404 for unknown session ID
|
|
274
|
+
- **transport-get** — GET returns SSE stream or 405
|
|
275
|
+
- **transport-delete** — DELETE accepted or returns 405
|
|
276
|
+
- **transport-batch-reject** — Rejects JSON-RPC batch requests (required)
|
|
277
|
+
- **transport-content-type-init** — Initialize response has valid content type
|
|
278
|
+
- **transport-get-stream** — GET with session returns SSE or 405
|
|
279
|
+
- **transport-concurrent** — Handles concurrent requests
|
|
280
|
+
- **transport-sse-event-field** — SSE responses include required event: message field
|
|
281
|
+
|
|
282
|
+
stdio-only (3):
|
|
283
|
+
- **stdio-framing** — Newline-delimited JSON framing (required)
|
|
284
|
+
- **stdio-unicode** — UTF-8 unicode roundtrip preserves non-ASCII payloads
|
|
285
|
+
- **stdio-unknown-method-recovers** — Returns -32601 for unknown methods and keeps serving
|
|
286
|
+
|
|
287
|
+
</details>
|
|
288
|
+
|
|
289
|
+
<details>
|
|
290
|
+
<summary><strong>Lifecycle (21 tests)</strong></summary>
|
|
291
|
+
|
|
292
|
+
- **lifecycle-init** — Initialize handshake succeeds (required)
|
|
293
|
+
- **lifecycle-proto-version** — Returns valid YYYY-MM-DD protocol version (required)
|
|
294
|
+
- **lifecycle-server-info** — Includes serverInfo with name
|
|
295
|
+
- **lifecycle-capabilities** — Returns capabilities object (required)
|
|
296
|
+
- **lifecycle-jsonrpc** — Response is valid JSON-RPC 2.0 (required)
|
|
297
|
+
- **lifecycle-ping** — Responds to ping method (required)
|
|
298
|
+
- **lifecycle-instructions** — Instructions field is valid string if present
|
|
299
|
+
- **lifecycle-id-match** — Response ID matches request ID (required)
|
|
300
|
+
- **lifecycle-string-id** — Supports string request IDs (JSON-RPC 2.0)
|
|
301
|
+
- **lifecycle-version-negotiate** — Handles unknown protocol version gracefully
|
|
302
|
+
- **lifecycle-reinit-reject** — Rejects second initialize request
|
|
303
|
+
- **lifecycle-logging** — logging/setLevel accepted (required if logging capability declared)
|
|
304
|
+
- **lifecycle-completions** — completion/complete accepted (required if completions capability declared)
|
|
305
|
+
- **lifecycle-cancellation** — Handles cancellation notifications
|
|
306
|
+
- **lifecycle-progress** — Handles progress notifications gracefully
|
|
307
|
+
- **lifecycle-list-changed** — Accepts listChanged notifications for declared capabilities
|
|
308
|
+
- **lifecycle-progress-token** — Supports progress tokens in requests via SSE
|
|
309
|
+
- **lifecycle-sampling-capability** — Advisory check for server-side use of the client sampling capability
|
|
310
|
+
- **lifecycle-roots-capability** — Advisory check for server-side use of the client roots capability
|
|
311
|
+
- **lifecycle-elicitation-capability** — Advisory check for the 2025-11-25 client elicitation capability
|
|
312
|
+
- **lifecycle-meta-tolerance** — Server ignores unknown `_meta` fields on incoming requests
|
|
313
|
+
|
|
314
|
+
</details>
|
|
315
|
+
|
|
316
|
+
<details>
|
|
317
|
+
<summary><strong>Tools (4 tests)</strong></summary>
|
|
318
|
+
|
|
319
|
+
- **tools-list** — tools/list returns valid array (required if tools capability declared)
|
|
320
|
+
- **tools-call** — tools/call responds with correct format
|
|
321
|
+
- **tools-pagination** — tools/list supports cursor-based pagination
|
|
322
|
+
- **tools-content-types** — Tool content items have valid types
|
|
323
|
+
|
|
324
|
+
</details>
|
|
325
|
+
|
|
326
|
+
<details>
|
|
327
|
+
<summary><strong>Resources (5 tests)</strong></summary>
|
|
328
|
+
|
|
329
|
+
- **resources-list** — resources/list returns valid array (required if resources capability declared)
|
|
330
|
+
- **resources-read** — resources/read returns content items
|
|
331
|
+
- **resources-templates** — resources/templates/list works or returns method-not-found
|
|
332
|
+
- **resources-pagination** — resources/list supports cursor-based pagination
|
|
333
|
+
- **resources-subscribe** — Resource subscribe/unsubscribe (required if subscribe capability declared)
|
|
334
|
+
|
|
335
|
+
</details>
|
|
336
|
+
|
|
337
|
+
<details>
|
|
338
|
+
<summary><strong>Prompts (3 tests)</strong></summary>
|
|
339
|
+
|
|
340
|
+
- **prompts-list** — prompts/list returns valid array (required if prompts capability declared)
|
|
341
|
+
- **prompts-get** — prompts/get returns valid messages
|
|
342
|
+
- **prompts-pagination** — prompts/list supports cursor-based pagination
|
|
343
|
+
|
|
344
|
+
</details>
|
|
345
|
+
|
|
346
|
+
<details>
|
|
347
|
+
<summary><strong>Error Handling (10 tests)</strong></summary>
|
|
348
|
+
|
|
349
|
+
- **error-unknown-method** — Returns JSON-RPC error for unknown method (required)
|
|
350
|
+
- **error-method-code** — Uses correct -32601 error code
|
|
351
|
+
- **error-invalid-jsonrpc** — Handles malformed JSON-RPC (required)
|
|
352
|
+
- **error-invalid-json** — Handles invalid JSON body
|
|
353
|
+
- **error-missing-params** — Returns error for tools/call without name
|
|
354
|
+
- **error-parse-code** — Returns -32700 for invalid JSON
|
|
355
|
+
- **error-invalid-request-code** — Returns -32600 for invalid request
|
|
356
|
+
- **tools-call-unknown** — Returns error for nonexistent tool name
|
|
357
|
+
- **error-capability-gated** — Rejects methods for undeclared capabilities
|
|
358
|
+
- **error-invalid-cursor** — Handles invalid pagination cursor gracefully
|
|
359
|
+
|
|
360
|
+
</details>
|
|
361
|
+
|
|
362
|
+
<details>
|
|
363
|
+
<summary><strong>Schema Validation (6 tests)</strong></summary>
|
|
364
|
+
|
|
365
|
+
- **tools-schema** — All tools have valid name and inputSchema (required if tools capability declared)
|
|
366
|
+
- **tools-annotations** — Tool annotations are valid if present
|
|
367
|
+
- **tools-title-field** — Tools include title field (2025-11-25)
|
|
368
|
+
- **tools-output-schema** — Tools with outputSchema are valid (2025-11-25)
|
|
369
|
+
- **prompts-schema** — Prompts have valid name field (required if prompts capability declared)
|
|
370
|
+
- **resources-schema** — Resources have valid uri and name (required if resources capability declared)
|
|
371
|
+
|
|
372
|
+
</details>
|
|
373
|
+
|
|
374
|
+
<details>
|
|
375
|
+
<summary><strong>Security (23 tests)</strong></summary>
|
|
376
|
+
|
|
377
|
+
- **security-auth-required** — Rejects unauthenticated requests
|
|
378
|
+
- **security-www-authenticate** — 401 responses include WWW-Authenticate header
|
|
379
|
+
- **security-auth-malformed** — Rejects malformed auth credentials
|
|
380
|
+
- **security-tls-required** — Enforces HTTPS/TLS
|
|
381
|
+
- **security-session-entropy** — Session IDs are high-entropy
|
|
382
|
+
- **security-session-not-auth** — Session ID does not bypass auth
|
|
383
|
+
- **security-oauth-metadata** — Protected Resource Metadata endpoint exists (RFC 9728)
|
|
384
|
+
- **security-token-in-uri** — Rejects auth tokens in query string
|
|
385
|
+
- **security-cors-headers** — CORS headers are restrictive
|
|
386
|
+
- **security-origin-validation** — Validates Origin header for DNS rebinding protection
|
|
387
|
+
- **security-command-injection** — Resists command injection in tool params
|
|
388
|
+
- **security-sql-injection** — Resists SQL injection in tool params
|
|
389
|
+
- **security-path-traversal** — Resists path traversal in tool params
|
|
390
|
+
- **security-ssrf-internal** — Resists SSRF to internal networks
|
|
391
|
+
- **security-oversized-input** — Handles oversized inputs gracefully
|
|
392
|
+
- **security-extra-params** — Rejects or ignores extra tool params
|
|
393
|
+
- **security-tool-schema-defined** — All tools define inputSchema
|
|
394
|
+
- **security-tool-rug-pull** — Tool definitions are stable across calls
|
|
395
|
+
- **security-tool-description-poisoning** — Tool descriptions free of injection patterns
|
|
396
|
+
- **security-tool-cross-reference** — Tools do not reference other tools by name
|
|
397
|
+
- **security-error-no-stacktrace** — Error responses do not leak stack traces
|
|
398
|
+
- **security-error-no-internal-ip** — Error responses do not leak internal IPs
|
|
399
|
+
- **security-rate-limiting** — Rate limiting is enforced
|
|
400
|
+
|
|
401
|
+
</details>
|
|
402
|
+
|
|
403
|
+
## Grading
|
|
404
|
+
|
|
405
|
+
| Grade | Score |
|
|
406
|
+
|-------|--------|
|
|
407
|
+
| A | 90-100 |
|
|
408
|
+
| B | 75-89 |
|
|
409
|
+
| C | 60-74 |
|
|
410
|
+
| D | 40-59 |
|
|
411
|
+
| F | 0-39 |
|
|
412
|
+
|
|
413
|
+
Required tests are worth 70% of the score, optional tests 30%. See the [full scoring algorithm](./COMPLIANCE_RUBRIC.md#2-scoring-algorithm) in the methodology doc.
|
|
414
|
+
|
|
415
|
+
## CI integration
|
|
416
|
+
|
|
417
|
+
```yaml
|
|
418
|
+
# GitHub Actions example
|
|
419
|
+
- name: MCP Compliance Check
|
|
420
|
+
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --strict
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
```yaml
|
|
424
|
+
# With JSON output for parsing
|
|
425
|
+
- name: MCP Compliance Check
|
|
426
|
+
run: |
|
|
427
|
+
npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --format json > compliance.json
|
|
428
|
+
cat compliance.json | jq '.grade'
|
|
429
|
+
```
|
|
430
|
+
|
|
431
|
+
```yaml
|
|
432
|
+
# With retries for flaky network conditions
|
|
433
|
+
- name: MCP Compliance Check
|
|
434
|
+
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --strict --retries 2 --timeout 30000
|
|
435
|
+
```
|
|
436
|
+
|
|
437
|
+
```yaml
|
|
438
|
+
# SARIF output for GitHub Code Scanning
|
|
439
|
+
- name: MCP Compliance Check
|
|
440
|
+
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --format sarif > compliance.sarif
|
|
441
|
+
- name: Upload SARIF
|
|
442
|
+
uses: github/codeql-action/upload-sarif@v3
|
|
443
|
+
with:
|
|
444
|
+
sarif_file: compliance.sarif
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
## MCP server (for Claude Code, Cursor, etc.)
|
|
448
|
+
|
|
449
|
+
This package also exposes an MCP server with 3 tools that can be used from Claude Code, Cursor, or any MCP client.
|
|
450
|
+
|
|
451
|
+
### Setup
|
|
452
|
+
|
|
453
|
+
**Claude Code (one-liner):**
|
|
454
|
+
|
|
455
|
+
```bash
|
|
456
|
+
claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance mcp
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
**Or create `.mcp.json` in your project root:**
|
|
460
|
+
|
|
461
|
+
macOS / Linux / WSL:
|
|
462
|
+
|
|
463
|
+
```json
|
|
464
|
+
{
|
|
465
|
+
"mcpServers": {
|
|
466
|
+
"mcp-compliance": {
|
|
467
|
+
"command": "npx",
|
|
468
|
+
"args": ["-y", "@yawlabs/mcp-compliance", "mcp"]
|
|
469
|
+
}
|
|
470
|
+
}
|
|
471
|
+
}
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
Windows:
|
|
475
|
+
|
|
476
|
+
```json
|
|
477
|
+
{
|
|
478
|
+
"mcpServers": {
|
|
479
|
+
"mcp-compliance": {
|
|
480
|
+
"command": "cmd",
|
|
481
|
+
"args": ["/c", "npx", "-y", "@yawlabs/mcp-compliance", "mcp"]
|
|
482
|
+
}
|
|
483
|
+
}
|
|
484
|
+
}
|
|
485
|
+
```
|
|
486
|
+
|
|
487
|
+
> **Tip:** This file is safe to commit — it contains no secrets.
|
|
488
|
+
|
|
489
|
+
Restart your MCP client and approve the server when prompted.
|
|
490
|
+
|
|
491
|
+
### Tools
|
|
492
|
+
|
|
493
|
+
- **mcp_compliance_test** — Run the full 88-test suite against a URL or stdio command. Supports auth, custom headers, env vars, timeout, retries, and category/test filtering. Returns grade, score, and detailed results.
|
|
494
|
+
- **mcp_compliance_badge** — Get the badge markdown/HTML for a server. Supports auth and custom headers.
|
|
495
|
+
- **mcp_compliance_explain** — Explain what a specific test ID checks and why it matters.
|
|
496
|
+
|
|
497
|
+
All tools have [MCP tool annotations](https://modelcontextprotocol.io/specification/2025-11-25/server/tools#annotations) (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) so MCP clients can skip confirmation dialogs for safe operations.
|
|
498
|
+
|
|
499
|
+
## Programmatic usage
|
|
500
|
+
|
|
501
|
+
```typescript
|
|
502
|
+
import { runComplianceSuite } from '@yawlabs/mcp-compliance';
|
|
503
|
+
|
|
504
|
+
const report = await runComplianceSuite('https://my-server.com/mcp');
|
|
505
|
+
console.log(`Grade: ${report.grade} (${report.score}%)`);
|
|
506
|
+
|
|
507
|
+
// With options
|
|
508
|
+
const report2 = await runComplianceSuite('https://my-server.com/mcp', {
|
|
509
|
+
headers: { 'Authorization': 'Bearer tok123' },
|
|
510
|
+
timeout: 30000,
|
|
511
|
+
retries: 1,
|
|
512
|
+
only: ['transport', 'lifecycle'],
|
|
513
|
+
});
|
|
514
|
+
|
|
515
|
+
// Live progress for streaming UIs (e.g. server-sent-events to a browser)
|
|
516
|
+
await runComplianceSuite('https://my-server.com/mcp', {
|
|
517
|
+
onTestComplete: (result) => {
|
|
518
|
+
// result has the full TestResult: id, name, category, required,
|
|
519
|
+
// passed, details, durationMs, specRef. Push it to your client.
|
|
520
|
+
sendToClient(result);
|
|
521
|
+
},
|
|
522
|
+
});
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
## Report schema
|
|
526
|
+
|
|
527
|
+
The JSON output of the test suite is a stable, versioned contract. Every report includes a `schemaVersion` field at the top level. The full JSON Schema lives at [`schemas/report.v1.json`](./schemas/report.v1.json) and is shipped with the npm package.
|
|
528
|
+
|
|
529
|
+
```jsonc
|
|
530
|
+
{
|
|
531
|
+
"schemaVersion": "1", // bumped on breaking changes to the report shape
|
|
532
|
+
"specVersion": "2025-11-25", // MCP spec version tested against
|
|
533
|
+
"toolVersion": "0.10.0", // mcp-compliance version that produced the report
|
|
534
|
+
"url": "...",
|
|
535
|
+
"timestamp": "...",
|
|
536
|
+
"grade": "A",
|
|
537
|
+
"score": 92.5,
|
|
538
|
+
"tests": [ ... ],
|
|
539
|
+
// ...
|
|
540
|
+
}
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
Consumer guidance:
|
|
544
|
+
|
|
545
|
+
- Pin against `schemaVersion`. Reject reports with an unknown version rather than guessing at the shape.
|
|
546
|
+
- The schema validates with any Draft 2020-12 validator (e.g. `ajv`).
|
|
547
|
+
- Within a major version, additions are non-breaking. Renames, removals, or type changes bump the version.
|
|
548
|
+
- Two runs against the same server produce equivalent grade, score, and per-test pass/fail (modulo timings/timestamps).
|
|
549
|
+
|
|
550
|
+
## Methodology & docs
|
|
551
|
+
|
|
552
|
+
The testing methodology is published openly so the grading is auditable:
|
|
553
|
+
|
|
554
|
+
- **[Testing methodology](./COMPLIANCE_RUBRIC.md)** — test execution model, scoring algorithm, all 88 test rules with pass/fail criteria (CC BY 4.0)
|
|
555
|
+
- **[Machine-readable rule catalog](./mcp-compliance-rules.json)** — JSON Schema-compliant catalog for programmatic consumption
|
|
556
|
+
- **[Why `mcp-compliance`](./docs/WHY.md)** — the problem, existing alternatives, what this tool does differently
|
|
557
|
+
- **[Fixing common failures](./docs/FIXES.md)** — recipes for the most frequent test failures with code snippets
|
|
558
|
+
- **[Spec version migration policy](./docs/SPEC_VERSION_MIGRATION.md)** — how this tool evolves with MCP spec releases
|
|
559
|
+
- **[mcp.hosting external API](./docs/EXT_API.md)** — public submit/retrieve/badge/delete endpoints used by `mcp-compliance badge` and any custom integrations
|
|
560
|
+
- **[Enterprise tier (draft)](./docs/ENTERPRISE.md)** — paid tier structure for organizations with scheduled/private/audit-track compliance needs
|
|
561
|
+
- **[Performance deep-dive](./docs/PERFORMANCE.md)** — why the suite is sequential and what parallel execution would cost
|
|
562
|
+
- **[Spec PR drafts](./docs/spec-prs/)** — our proposed MCP spec clarifications for ambiguous cases we've hit
|
|
563
|
+
- **[mcp.hosting integration spec](./docs/mcp-hosting-integration.md)** — the contract between this engine and the mcp.hosting platform: URL surfaces, data flow, storage model, badge API, leaderboard, router integration
|
|
564
|
+
|
|
565
|
+
The methodology is not an authoritative conformance standard — it's one tool's choices, published so they can be inspected, adopted, or forked. The [official MCP specification](https://modelcontextprotocol.io/specification/2025-11-25) defines what servers must do; this document describes how `@yawlabs/mcp-compliance` verifies it.
|
|
566
|
+
|
|
567
|
+
## Requirements
|
|
568
|
+
|
|
569
|
+
- Node.js 18+
|
|
570
|
+
|
|
571
|
+
## Contributing
|
|
572
|
+
|
|
573
|
+
```bash
|
|
574
|
+
git clone https://github.com/YawLabs/mcp-compliance.git
|
|
575
|
+
cd mcp-compliance
|
|
576
|
+
npm install
|
|
577
|
+
npm run build
|
|
578
|
+
npm test
|
|
579
|
+
```
|
|
580
|
+
|
|
581
|
+
**Development commands:**
|
|
582
|
+
|
|
583
|
+
| Command | Description |
|
|
584
|
+
|---------|-------------|
|
|
585
|
+
| `npm run build` | Compile with tsup |
|
|
586
|
+
| `npm run dev` | Watch mode |
|
|
587
|
+
| `npm test` | Run tests (vitest) |
|
|
588
|
+
| `npm run lint` | Check with Biome |
|
|
589
|
+
| `npm run lint:fix` | Auto-fix with Biome |
|
|
590
|
+
| `npm run typecheck` | TypeScript type checking |
|
|
591
|
+
| `npm run test:ci` | Build + test (CI-safe) |
|
|
592
|
+
|
|
593
|
+
## Links
|
|
594
|
+
|
|
595
|
+
- [mcp.hosting](https://mcp.hosting) — Hosted MCP server infrastructure
|
|
596
|
+
- [MCP Specification](https://modelcontextprotocol.io/specification/2025-11-25)
|
|
597
|
+
- [Testing methodology](./COMPLIANCE_RUBRIC.md)
|
|
598
|
+
- [Yaw Labs](https://yaw.sh)
|
|
599
|
+
|
|
600
|
+
## License
|
|
601
|
+
|
|
602
|
+
MIT — see [LICENSE](./LICENSE).
|