@bryan-thompson/inspector-assessment 1.23.5 → 1.23.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +216 -1148
- package/cli/build/assess-full.js +2 -2
- package/client/dist/assets/{OAuthCallback-DeXwqSKV.js → OAuthCallback-Cdmh8W1V.js} +1 -1
- package/client/dist/assets/{OAuthDebugCallback-DyxX1t_p.js → OAuthDebugCallback-3i-pMsFQ.js} +1 -1
- package/client/dist/assets/{index-5ZFeiSLu.js → index-BOarHV9b.js} +4 -4
- package/client/dist/index.html +1 -1
- package/client/lib/lib/securityPatterns.d.ts +1 -1
- package/client/lib/lib/securityPatterns.d.ts.map +1 -1
- package/client/lib/lib/securityPatterns.js +4 -2
- package/client/lib/services/assessment/modules/SecurityAssessor.d.ts +9 -0
- package/client/lib/services/assessment/modules/SecurityAssessor.d.ts.map +1 -1
- package/client/lib/services/assessment/modules/SecurityAssessor.js +51 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -3,1299 +3,367 @@
|
|
|
3
3
|
[](https://www.npmjs.com/package/@bryan-thompson/inspector-assessment)
|
|
4
4
|
[](https://www.npmjs.com/package/@bryan-thompson/inspector-assessment)
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
**Comprehensive MCP server validation with 16 automated assessment modules.**
|
|
7
|
+
Test functionality, security, documentation, and policy compliance from the command line.
|
|
7
8
|
|
|
8
9
|

|
|
9
10
|
|
|
10
|
-
|
|
11
|
+
---
|
|
11
12
|
|
|
12
|
-
|
|
13
|
+
## Installation
|
|
13
14
|
|
|
14
15
|
```bash
|
|
16
|
+
# Install globally
|
|
15
17
|
npm install -g @bryan-thompson/inspector-assessment
|
|
16
|
-
```
|
|
17
|
-
|
|
18
|
-
**Or use directly with bunx (no installation):**
|
|
19
18
|
|
|
20
|
-
|
|
19
|
+
# Or use directly with bunx (no installation)
|
|
21
20
|
bunx @bryan-thompson/inspector-assessment
|
|
22
21
|
```
|
|
23
22
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
git clone https://github.com/triepod-ai/inspector-assessment.git
|
|
28
|
-
cd inspector-assessment
|
|
29
|
-
npm install
|
|
30
|
-
npm run build
|
|
31
|
-
npm run dev
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
## Quick Start
|
|
35
|
-
|
|
36
|
-
After installation, launch the inspector:
|
|
37
|
-
|
|
38
|
-
```bash
|
|
39
|
-
# Using global install
|
|
40
|
-
mcp-inspector-assess
|
|
41
|
-
|
|
42
|
-
# Using bunx
|
|
43
|
-
bunx @bryan-thompson/inspector-assessment
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
The web interface will open at http://localhost:6274
|
|
47
|
-
|
|
48
|
-
## For MCP Directory Reviewers
|
|
49
|
-
|
|
50
|
-
If you're reviewing MCP servers for the Anthropic MCP Directory, see our **[Reviewer Quick Start Guide](docs/REVIEWER_QUICK_START.md)** for:
|
|
51
|
-
|
|
52
|
-
- **60-second fast screening** workflow for approve/reject decisions
|
|
53
|
-
- **5-minute detailed review** process for borderline cases
|
|
54
|
-
- **Common pitfalls** explanation (false positives in security, informational vs scored tests)
|
|
55
|
-
- **Decision matrix** with clear approval criteria
|
|
56
|
-
- **Fast CLI analysis** commands for troubleshooting
|
|
57
|
-
|
|
58
|
-
The quick start guide is optimized for fast reviewer onboarding and provides clear guidance on interpreting assessment results.
|
|
59
|
-
|
|
60
|
-
## About This Fork
|
|
61
|
-
|
|
62
|
-
This is an enhanced fork of [Anthropic's MCP Inspector](https://github.com/modelcontextprotocol/inspector) with significantly expanded assessment capabilities for MCP server validation and testing.
|
|
63
|
-
|
|
64
|
-
**Original Repository**: https://github.com/modelcontextprotocol/inspector
|
|
65
|
-
**Our Enhanced Fork**: https://github.com/triepod-ai/inspector-assessment
|
|
66
|
-
|
|
67
|
-
**⚠️ Important**: This is a published fork with assessment enhancements. If you want the official Anthropic inspector without assessment features, use `npx @modelcontextprotocol/inspector`.
|
|
68
|
-
|
|
69
|
-
### What We Added
|
|
70
|
-
|
|
71
|
-
We've built a comprehensive assessment framework on top of the original inspector that transforms it from a debugging tool into a full validation suite for MCP servers. Our enhancements focus on accuracy, depth, and actionable insights for MCP server developers.
|
|
72
|
-
|
|
73
|
-
## Key Features
|
|
74
|
-
|
|
75
|
-
- **Interactive Testing**: Visual interface for testing MCP server tools, resources, and prompts
|
|
76
|
-
- **Comprehensive Assessment**: Automated validation of server functionality, error handling, documentation, security, and usability using multi-scenario testing with progressive complexity
|
|
77
|
-
- **Business Logic Validation**: Distinguishes between proper error handling and unintended failures
|
|
78
|
-
- **Detailed Test Reports**: Confidence scoring, test scenario details, and actionable recommendations
|
|
79
|
-
- **Multiple Transport Support**: STDIO, SSE, and Streamable HTTP transports
|
|
80
|
-
|
|
81
|
-
## Quality Metrics
|
|
82
|
-
|
|
83
|
-
Our enhanced fork maintains high code quality standards with comprehensive testing and validation:
|
|
84
|
-
|
|
85
|
-
- **Test Coverage**: ✅ 665/665 tests passing (100% pass rate)
|
|
86
|
-
- **Assessment Module Tests**: 291 tests specifically validating our assessment enhancements (including 83 new MCP Directory compliance tests)
|
|
87
|
-
- Business logic error detection with confidence scoring
|
|
88
|
-
- Progressive complexity testing (2 levels: minimal → simple)
|
|
89
|
-
- Context-aware security testing with zero false positives
|
|
90
|
-
- Realistic test data generation and boundary testing
|
|
91
|
-
- **Total Project Tests**: 582 tests including assessment modules, UI components, and core inspector functionality
|
|
92
|
-
- All tests updated to reflect focused backend testing (8 security patterns × 3 payloads per tool)
|
|
93
|
-
- Test files: `client/src/services/__tests__/` and `client/src/services/assessment/__tests__/`
|
|
94
|
-
- **Code Quality**: ✅ Production code uses proper TypeScript types
|
|
95
|
-
- 229 lint issues remaining (down 18% from 280 after recent cleanup)
|
|
96
|
-
- All source files migrated from `any` to `unknown` or proper types
|
|
97
|
-
- React Hooks follow best practices with proper dependency arrays
|
|
98
|
-
- **Build Status**: ✅ Production builds pass cleanly
|
|
99
|
-
- TypeScript compilation successful for all production code
|
|
100
|
-
- Vite build optimized and validated
|
|
101
|
-
- **Upstream Sync**: ✅ Up-to-date with v0.17.0
|
|
102
|
-
- Successfully integrated 121 commits from upstream
|
|
103
|
-
- New features: CustomHeaders, OAuth improvements, parameter validation
|
|
104
|
-
- All enhancements preserved during merge
|
|
105
|
-
|
|
106
|
-
**Testing Commands**:
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
npm test # Run all 665 tests
|
|
110
|
-
npm test -- assessment # Run all 291 assessment module tests
|
|
111
|
-
npm test -- assessmentService # Run assessment service integration tests (54 tests)
|
|
112
|
-
npm test -- SecurityAssessor # Run security assessment tests (16 tests)
|
|
113
|
-
npm test -- FunctionalityAssessor # Run functionality tests (11 tests)
|
|
114
|
-
npm test -- AUPCompliance # Run AUP compliance tests (26 tests)
|
|
115
|
-
npm test -- ToolAnnotation # Run tool annotation tests (13 tests)
|
|
116
|
-
npm run coverage # Generate coverage report
|
|
117
|
-
npm run lint # Check code quality
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
## Our Enhancements to the MCP Inspector
|
|
121
|
-
|
|
122
|
-
We've significantly expanded the original MCP Inspector's capabilities with advanced assessment features that go far beyond basic debugging. Here's what makes our fork unique:
|
|
123
|
-
|
|
124
|
-
### 1. Enhanced Business Logic Error Detection
|
|
125
|
-
|
|
126
|
-
**Problem**: The original inspector couldn't distinguish between broken tools and tools that correctly validate input. A tool returning "user not found" would be marked as broken.
|
|
127
|
-
|
|
128
|
-
**Our Solution**: Confidence-based validation system (ResponseValidator.ts:client/src/services/assessment/ResponseValidator.ts)
|
|
129
|
-
|
|
130
|
-
- **MCP Standard Error Code Recognition**: Properly identifies error codes like `-32602` (Invalid params) as successful validation
|
|
131
|
-
- **Confidence Scoring**: Multi-factor analysis determines if errors represent proper business logic
|
|
132
|
-
- **Tool Type Awareness**: Different validation thresholds for CRUD vs utility tools
|
|
133
|
-
- **Impact**: Estimated 80% reduction in false positives for resource-based tools (based on analysis in [FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md](docs/FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md#key-problems-addressed))
|
|
134
|
-
|
|
135
|
-
### 2. Optimized Progressive Complexity Testing
|
|
136
|
-
|
|
137
|
-
**Problem**: Testing tools with only complex inputs makes it hard to identify where functionality breaks down.
|
|
138
|
-
|
|
139
|
-
**Our Solution**: Two-level progressive diagnostic testing (TestScenarioEngine.ts:client/src/services/assessment/TestScenarioEngine.ts)
|
|
140
|
-
|
|
141
|
-
1. **Minimal**: Only required fields with simplest values - diagnoses basic setup issues
|
|
142
|
-
2. **Simple**: Required fields with realistic simple values - validates core functionality
|
|
143
|
-
|
|
144
|
-
Combined with multi-scenario comprehensive testing (Happy Path, Edge Cases, Boundary Testing) for full coverage.
|
|
145
|
-
|
|
146
|
-
**Benefits**:
|
|
147
|
-
|
|
148
|
-
- **50% faster** than previous 4-level approach (removed redundant typical/maximum tests)
|
|
149
|
-
- Identifies exact complexity level where tools fail (minimal vs simple)
|
|
150
|
-
- Provides specific, actionable recommendations
|
|
151
|
-
- Zero coverage loss - comprehensive scenarios provide full validation
|
|
152
|
-
- Helps developers understand tool limitations and requirements
|
|
153
|
-
|
|
154
|
-
**Performance**: For 10-tool servers, comprehensive testing now runs in ~4.2-8.3 minutes (down from ~7.5-11.7 minutes)
|
|
155
|
-
|
|
156
|
-
### 3. Realistic Test Data Generation
|
|
157
|
-
|
|
158
|
-
**Problem**: Generic test data like "test_value" and fake IDs trigger validation errors, causing false failures.
|
|
159
|
-
|
|
160
|
-
**Our Solution**: Context-aware test data generation (TestDataGenerator.ts:client/src/services/assessment/TestDataGenerator.ts)
|
|
161
|
-
|
|
162
|
-
- **Publicly Accessible URLs**: `https://www.google.com`, `https://api.github.com/users/octocat`
|
|
163
|
-
- **Real API Endpoints**: Uses actual test APIs like jsonplaceholder.typicode.com
|
|
164
|
-
- **Valid UUIDs**: Properly formatted identifiers that won't trigger format validation
|
|
165
|
-
- **Context Awareness**: Generates appropriate data based on field names (email, url, id, etc.)
|
|
166
|
-
|
|
167
|
-
### 4. Context-Aware Security Assessment with Zero False Positives
|
|
168
|
-
|
|
169
|
-
**Problem**: Traditional security scanners flag tools as vulnerable when they safely store or echo malicious input as data. For example, a database tool that stores `"<script>alert(1)"` would be incorrectly flagged as vulnerable to XSS.
|
|
170
|
-
|
|
171
|
-
**Our Solution**: Intelligent reflection detection (SecurityAssessor.ts:client/src/services/assessment/modules/SecurityAssessor.ts)
|
|
172
|
-
|
|
173
|
-
**Key Innovation**: Distinguishing **data reflection** (safe) from **command execution** (vulnerable)
|
|
174
|
-
|
|
175
|
-
**Examples**:
|
|
176
|
-
|
|
177
|
-
✅ **SAFE - Data Reflection**:
|
|
178
|
-
|
|
179
|
-
```
|
|
180
|
-
Payload: "<script>alert(1)</script>"
|
|
181
|
-
Response: "Stored in collection: <script>alert(1)</script>"
|
|
182
|
-
→ Tool is just storing data, not executing it
|
|
183
|
-
```
|
|
184
|
-
|
|
185
|
-
❌ **VULNERABLE - Command Execution** (Calculator Injection):
|
|
186
|
-
|
|
187
|
-
```
|
|
188
|
-
Payload: "2+2"
|
|
189
|
-
Response: "The answer is 4"
|
|
190
|
-
→ Tool executed the arithmetic expression via eval()!
|
|
191
|
-
```
|
|
192
|
-
|
|
193
|
-
**Detection Approach**:
|
|
194
|
-
|
|
195
|
-
1. **Reflection Pattern Recognition**: Identifies safe data operations through patterns like "stored", "created", "error getting info for", "collection doesn't exist"
|
|
196
|
-
2. **Execution Evidence Detection**: Only flags as vulnerable when actual execution is detected (calculator returning "4", API keys leaked, admin mode activated)
|
|
197
|
-
3. **Error Message Handling**: Recognizes that error messages echoing invalid input are safe reflection, not vulnerabilities
|
|
198
|
-
|
|
199
|
-
**Impact**:
|
|
200
|
-
|
|
201
|
-
- **Zero false positives** on data storage/retrieval tools (qdrant, databases, file systems)
|
|
202
|
-
- **18 injection patterns tested** (9 original + 9 advanced patterns)
|
|
203
|
-
- **Dual-mode testing**: Reviewer mode (3 critical patterns, fast) + Developer mode (all 13 patterns, comprehensive)
|
|
204
|
-
- **Real vulnerabilities still detected**: 100% test pass rate on detecting actual command injection, calculator injection, role override, data exfiltration
|
|
205
|
-
|
|
206
|
-
**Supported Injection Types**:
|
|
207
|
-
|
|
208
|
-
- **Command Injection**: System commands (whoami, ls -la, pwd)
|
|
209
|
-
- **Calculator Injection**: Arithmetic expressions and code injection via eval() (NEW - 7 payloads)
|
|
210
|
-
- **SQL Injection**: Database command injection
|
|
211
|
-
- **Path Traversal**: File system access outside intended directory
|
|
212
|
-
- Plus 9 additional patterns (Unicode Bypass, Nested Injection, Package Squatting, etc.)
|
|
213
|
-
|
|
214
|
-
**Validation**: See [VULNERABILITY_TESTING.md](VULNERABILITY_TESTING.md) for detailed testing guide and examples.
|
|
215
|
-
|
|
216
|
-
### 5. Streamlined Assessment Architecture
|
|
217
|
-
|
|
218
|
-
**Based on Real-World Testing**: Our methodology has been validated through systematic testing using the taskmanager MCP server as a case study (11 tools tested with 8 backend security patterns, detailed in [ASSESSMENT_METHODOLOGY.md](docs/ASSESSMENT_METHODOLOGY.md)).
|
|
219
|
-
|
|
220
|
-
**Eleven Core Assessors** aligned with Anthropic's MCP directory submission requirements:
|
|
221
|
-
|
|
222
|
-
**Original MCP Inspector Assessors:**
|
|
223
|
-
|
|
224
|
-
1. **FunctionalityAssessor** (225 lines)
|
|
225
|
-
- Multi-scenario validation with progressive complexity
|
|
226
|
-
- Coverage tracking and reliability scoring
|
|
227
|
-
- Business logic error detection
|
|
228
|
-
- Performance measurement
|
|
229
|
-
|
|
230
|
-
2. **SecurityAssessor** (443 lines)
|
|
231
|
-
- 13 distinct injection attack patterns (including Calculator Injection) with context-aware reflection detection
|
|
232
|
-
- Direct command injection, calculator injection (eval detection), role override, data exfiltration detection
|
|
233
|
-
- Vulnerability analysis with risk levels (HIGH/MEDIUM/LOW)
|
|
234
|
-
- Zero false positives through intelligent distinction between data reflection and command execution
|
|
235
|
-
|
|
236
|
-
3. **ErrorHandlingAssessor** (692 lines)
|
|
237
|
-
- MCP protocol compliance scoring
|
|
238
|
-
- Error response quality analysis
|
|
239
|
-
- Invalid input resilience testing
|
|
240
|
-
|
|
241
|
-
4. **DocumentationAssessor** (274 lines)
|
|
242
|
-
- README structure and completeness
|
|
243
|
-
- Code example extraction and validation
|
|
244
|
-
- API reference quality assessment
|
|
245
|
-
|
|
246
|
-
5. **UsabilityAssessor** (290 lines)
|
|
247
|
-
- Naming convention consistency
|
|
248
|
-
- Parameter clarity assessment
|
|
249
|
-
- Best practices compliance
|
|
250
|
-
|
|
251
|
-
6. **MCPSpecComplianceAssessor** (560 lines) - Extended
|
|
252
|
-
- JSON-RPC 2.0 compliance validation
|
|
253
|
-
- Protocol message format verification
|
|
254
|
-
- MCP specification adherence
|
|
255
|
-
|
|
256
|
-
**NEW: MCP Directory Compliance Assessors** (added 2025-12):
|
|
257
|
-
|
|
258
|
-
7. **AUPComplianceAssessor** - Policy compliance
|
|
259
|
-
- 14 AUP category violation detection (A-N)
|
|
260
|
-
- High-risk domain identification (Healthcare, Financial, Legal, Children)
|
|
261
|
-
- Tool name/description pattern analysis
|
|
262
|
-
- Source code scanning (enhanced mode)
|
|
263
|
-
|
|
264
|
-
8. **ToolAnnotationAssessor** - Policy #17 compliance
|
|
265
|
-
- readOnlyHint/destructiveHint verification
|
|
266
|
-
- Tool behavior inference from name patterns
|
|
267
|
-
- Annotation misalignment detection
|
|
268
|
-
- Policy #17 compliance reporting
|
|
269
|
-
|
|
270
|
-
9. **ProhibitedLibrariesAssessor** - Policy #28-30 compliance
|
|
271
|
-
- Financial library detection (Stripe, PayPal, Plaid, etc.)
|
|
272
|
-
- Media library detection (Sharp, FFmpeg, OpenCV, PIL)
|
|
273
|
-
- package.json and requirements.txt scanning
|
|
274
|
-
- Source code import analysis
|
|
275
|
-
|
|
276
|
-
10. **ManifestValidationAssessor** - MCPB manifest compliance
|
|
277
|
-
- manifest_version 0.3 validation
|
|
278
|
-
- Required field verification (name, version, mcp_config)
|
|
279
|
-
- Icon presence check
|
|
280
|
-
- ${BUNDLE_ROOT} anti-pattern detection
|
|
281
|
-
|
|
282
|
-
11. **PortabilityAssessor** - Cross-platform compatibility
|
|
283
|
-
- Hardcoded path detection (/Users/, /home/, C:\)
|
|
284
|
-
- Platform-specific code patterns
|
|
285
|
-
- ${\_\_dirname} usage validation
|
|
286
|
-
|
|
287
|
-
**Recent Updates**:
|
|
288
|
-
|
|
289
|
-
- (2025-12-07): Added 5 new MCP Directory compliance assessors with 83 tests
|
|
290
|
-
- (2025-10-05): Removed 2,707 lines of out-of-scope assessment modules to focus on core MCP validation requirements
|
|
291
|
-
|
|
292
|
-
### 6. Advanced Assessment Components
|
|
293
|
-
|
|
294
|
-
We've built a complete assessment architecture with specialized modules:
|
|
295
|
-
|
|
296
|
-
- **AssessmentOrchestrator.ts**: Coordinates multi-phase testing across all assessment dimensions
|
|
297
|
-
- **ResponseValidator.ts**: Advanced response validation with confidence scoring and business logic detection
|
|
298
|
-
- **TestScenarioEngine.ts**: Generates and executes optimized 2-level progressive complexity tests
|
|
299
|
-
- **TestDataGenerator.ts**: Context-aware realistic test data generation with conditional boundary testing
|
|
300
|
-
- **Assessment UI Components**: Rich visualization of test results and recommendations
|
|
301
|
-
|
|
302
|
-
### Documentation
|
|
303
|
-
|
|
304
|
-
Our enhancements include comprehensive documentation:
|
|
305
|
-
|
|
306
|
-
- **ASSESSMENT_METHODOLOGY.md**: Complete methodology with examples and best practices
|
|
307
|
-
- **FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md**: Implementation details and impact analysis
|
|
308
|
-
- **COMPREHENSIVE_TESTING_OPTIMIZATION_PLAN.md**: Detailed optimization strategy (Phases 1-2 complete)
|
|
309
|
-
- **PHASE1_OPTIMIZATION_COMPLETED.md**: Progressive complexity optimization (50% faster)
|
|
310
|
-
- **PHASE2_OPTIMIZATION_COMPLETED.md**: Business logic error detection enhancements
|
|
311
|
-
- **PROJECT_STATUS.md**: Current status, recent changes, and development roadmap
|
|
312
|
-
- **Test Coverage Reports**: Detailed validation of our assessment accuracy
|
|
313
|
-
|
|
314
|
-
## Architecture Overview
|
|
315
|
-
|
|
316
|
-
The MCP Inspector consists of two main components that work together:
|
|
317
|
-
|
|
318
|
-
- **MCP Inspector Client (MCPI)**: A React-based web UI that provides an interactive interface for testing and debugging MCP servers
|
|
319
|
-
- **MCP Proxy (MCPP)**: A Node.js server that acts as a protocol bridge, connecting the web UI to MCP servers via various transport methods (stdio, SSE, streamable-http)
|
|
320
|
-
|
|
321
|
-
Note that the proxy is not a network proxy for intercepting traffic. Instead, it functions as both an MCP client (connecting to your MCP server) and an HTTP server (serving the web UI), enabling browser-based interaction with MCP servers that use different transport protocols.
|
|
322
|
-
|
|
323
|
-
## Assessment Capabilities
|
|
324
|
-
|
|
325
|
-
Our enhanced MCP Inspector includes a comprehensive assessment system that validates MCP servers against Anthropic's directory submission requirements and MCP protocol standards:
|
|
326
|
-
|
|
327
|
-
### Assessment Categories
|
|
328
|
-
|
|
329
|
-
1. **Functionality Testing**
|
|
330
|
-
- Multi-scenario validation with happy path, edge cases, and boundary testing
|
|
331
|
-
- Optimized progressive complexity testing (2 levels: minimal → simple)
|
|
332
|
-
- Business logic validation to distinguish proper error handling from failures
|
|
333
|
-
- Confidence scoring based on test coverage and consistency
|
|
334
|
-
- Evidence-based recommendations with transparent methodology (e.g., "5/5 scenarios verified (happy path, edge cases, boundaries, error handling)")
|
|
335
|
-
|
|
336
|
-
2. **Error Handling**
|
|
337
|
-
- Invalid input resilience testing
|
|
338
|
-
- Comprehensive error message analysis
|
|
339
|
-
- Resource validation vs. unintended failures
|
|
340
|
-
- MCP protocol compliance scoring
|
|
341
|
-
- Quality scoring for descriptive error messages
|
|
342
|
-
|
|
343
|
-
3. **Documentation**
|
|
344
|
-
- Tool description completeness and clarity
|
|
345
|
-
- Parameter documentation validation
|
|
346
|
-
- README structure and examples evaluation
|
|
347
|
-
- API documentation quality assessment
|
|
348
|
-
|
|
349
|
-
4. **Security**
|
|
350
|
-
- 17 distinct injection attack patterns (8 original + 9 advanced patterns)
|
|
351
|
-
- Context-aware reflection detection distinguishes safe data operations from command execution
|
|
352
|
-
- Zero false positives - correctly handles tools that echo/store malicious input as data
|
|
353
|
-
- Input validation and sanitization checks
|
|
354
|
-
- Authentication/authorization testing
|
|
355
|
-
- Sensitive data exposure detection
|
|
356
|
-
- Dual-mode testing: Reviewer mode (3 critical patterns) + Developer mode (all 17 patterns)
|
|
357
|
-
|
|
358
|
-
5. **Usability**
|
|
359
|
-
- Tool naming consistency analysis
|
|
360
|
-
- Description quality assessment
|
|
361
|
-
- Schema completeness validation
|
|
362
|
-
- Parameter clarity evaluation
|
|
363
|
-
|
|
364
|
-
6. **MCP Spec Compliance** (Extended)
|
|
365
|
-
- JSON-RPC 2.0 protocol compliance
|
|
366
|
-
- MCP message format verification
|
|
367
|
-
- Error code standard compliance
|
|
368
|
-
- Protocol specification adherence
|
|
369
|
-
|
|
370
|
-
### Testing Features
|
|
371
|
-
|
|
372
|
-
**Note**: All testing uses comprehensive multi-scenario validation. See the "Our Enhancements" section above for detailed technical descriptions.
|
|
373
|
-
|
|
374
|
-
#### Multi-Scenario Validation
|
|
375
|
-
|
|
376
|
-
The inspector tests each tool with multiple scenarios:
|
|
377
|
-
|
|
378
|
-
- **Happy Path**: Valid inputs with expected success cases
|
|
379
|
-
- **Edge Cases**: Boundary values and unusual but valid inputs
|
|
380
|
-
- **Error Cases**: Invalid inputs to test error handling
|
|
381
|
-
- **Boundary Testing**: Maximum/minimum values and limits
|
|
382
|
-
|
|
383
|
-
#### Progressive Complexity Testing
|
|
384
|
-
|
|
385
|
-
Tools are tested with progressive diagnostic levels to identify where functionality breaks:
|
|
386
|
-
|
|
387
|
-
1. **Minimal**: Only required fields with simplest values - diagnoses basic setup issues
|
|
388
|
-
2. **Simple**: Required fields with realistic simple values - validates core functionality
|
|
389
|
-
|
|
390
|
-
This optimized 2-level approach is **50% faster** than previous 4-level testing while maintaining full coverage through comprehensive multi-scenario validation.
|
|
391
|
-
|
|
392
|
-
#### Business Logic Validation
|
|
393
|
-
|
|
394
|
-
The assessment distinguishes between:
|
|
395
|
-
|
|
396
|
-
- **Proper Validation**: Expected errors for invalid business logic (e.g., "User not found")
|
|
397
|
-
- **Tool Failures**: Unexpected errors indicating implementation issues
|
|
398
|
-
- **Resource Validation**: Proper handling of non-existent resources
|
|
399
|
-
- **Input Validation**: Appropriate rejection of malformed inputs
|
|
400
|
-
|
|
401
|
-
### Assessment Configuration
|
|
402
|
-
|
|
403
|
-
Configure assessment behavior through the UI:
|
|
404
|
-
|
|
405
|
-
| Setting | Description | Default |
|
|
406
|
-
| --------------------------------- | ------------------------------------------------------------------- | --------- |
|
|
407
|
-
| Tool Selection for Error Handling | Multi-select dropdown with checkboxes to choose which tools to test | All tools |
|
|
408
|
-
|
|
409
|
-
**Tool Selection** (as of 2025-10-10):
|
|
410
|
-
|
|
411
|
-
- Visual multi-select dropdown with checkboxes for each tool
|
|
412
|
-
- Search/filter functionality for large tool lists
|
|
413
|
-
- "Select All" / "Deselect All" bulk operations
|
|
414
|
-
- Shows "X of Y tools selected" count
|
|
415
|
-
- Select 0 tools to skip error handling tests entirely (fastest option)
|
|
416
|
-
|
|
417
|
-
**Note**: The old numeric "Error Handling Test Limit" has been replaced with the tool selector. The `maxToolsToTestForErrors` config field is deprecated but still works for backward compatibility.
|
|
418
|
-
|
|
419
|
-
> **Deprecation Notice**: `maxToolsToTestForErrors` will be removed in v2.0.0.
|
|
420
|
-
> Migrate to `selectedToolsForTesting` (undefined = test all, [] = test none).
|
|
421
|
-
|
|
422
|
-
### Viewing Assessment Results
|
|
423
|
-
|
|
424
|
-
The Assessment tab provides:
|
|
425
|
-
|
|
426
|
-
- **Overall Score**: Weighted aggregate score with letter grade (A-F)
|
|
427
|
-
- **Category Breakdown**: Individual scores for each assessment category
|
|
428
|
-
- **Tool Details**: Click any tool name to see detailed test results including:
|
|
429
|
-
- Test scenarios executed
|
|
430
|
-
- Input parameters used
|
|
431
|
-
- Actual responses received
|
|
432
|
-
- Pass/fail status with confidence scores
|
|
433
|
-
- Specific issues identified
|
|
434
|
-
- **Per-Tool JSON Display** (New in 2025-10-10): Each tool in security assessment has its own "Show JSON" button to view only that tool's test results without scrolling through all tools
|
|
435
|
-
- **Recommendations**: Actionable suggestions for improvement
|
|
436
|
-
- **Test Coverage**: Visual indicators of testing completeness
|
|
437
|
-
|
|
438
|
-
### Assessment Result Persistence
|
|
439
|
-
|
|
440
|
-
**New in 2025-10-06**: Assessment results are automatically saved to JSON files for fast CLI-based analysis and troubleshooting.
|
|
441
|
-
|
|
442
|
-
**Automatic Save**:
|
|
443
|
-
|
|
444
|
-
- Every assessment run automatically saves to `/tmp/inspector-assessment-{serverName}.json`
|
|
445
|
-
- Old results are automatically deleted before new runs
|
|
446
|
-
- No manual export needed - completely transparent
|
|
447
|
-
- Console shows: `✅ Assessment auto-saved: /tmp/inspector-assessment-{name}.json`
|
|
448
|
-
|
|
449
|
-
**Quick Analysis Examples**:
|
|
450
|
-
|
|
451
|
-
```bash
|
|
452
|
-
# View full assessment results
|
|
453
|
-
cat /tmp/inspector-assessment-memory-mcp.json | jq
|
|
454
|
-
|
|
455
|
-
# Check only functionality results
|
|
456
|
-
cat /tmp/inspector-assessment-memory-mcp.json | jq '.functionality'
|
|
457
|
-
|
|
458
|
-
# List broken tools
|
|
459
|
-
cat /tmp/inspector-assessment-memory-mcp.json | jq '.functionality.brokenTools'
|
|
460
|
-
|
|
461
|
-
# Get specific tool test results
|
|
462
|
-
cat /tmp/inspector-assessment-memory-mcp.json | jq '.functionality.enhancedResults[] | select(.toolName == "search_nodes")'
|
|
463
|
-
|
|
464
|
-
# Summary of all tools and their status
|
|
465
|
-
cat /tmp/inspector-assessment-memory-mcp.json | jq '.functionality.enhancedResults[] | {tool: .toolName, status: .overallStatus}'
|
|
466
|
-
|
|
467
|
-
# Count security vulnerabilities found
|
|
468
|
-
cat /tmp/inspector-assessment-memory-mcp.json | jq '.security.vulnerabilities | length'
|
|
469
|
-
|
|
470
|
-
# Check error handling coverage
|
|
471
|
-
cat /tmp/inspector-assessment-memory-mcp.json | jq '.errorHandling.metrics.validationCoverage'
|
|
472
|
-
```
|
|
473
|
-
|
|
474
|
-
**Benefits**:
|
|
475
|
-
|
|
476
|
-
- Fast troubleshooting with `jq`, `grep`, or any CLI tool
|
|
477
|
-
- Easy integration with scripts and automation
|
|
478
|
-
- No need to manually export results each time
|
|
479
|
-
- Results persist between inspector sessions for comparison
|
|
480
|
-
|
|
481
|
-
### Test Suite Validation
|
|
482
|
-
|
|
483
|
-
Our assessment capabilities are backed by a comprehensive test suite that validates all assessment functionality:
|
|
484
|
-
|
|
485
|
-
**Test Coverage Summary**:
|
|
486
|
-
|
|
487
|
-
- **665 passing tests** across all project modules (100% pass rate)
|
|
488
|
-
- **291 assessment module tests** specifically created for validation of our enhancements
|
|
489
|
-
|
|
490
|
-
#### Assessment Module Test Breakdown
|
|
491
|
-
|
|
492
|
-
The assessment functionality is validated by **291 specialized tests** across 19 test files:
|
|
493
|
-
|
|
494
|
-
| Test File | Tests | Purpose |
|
|
495
|
-
| ------------------------------------- | ------- | -------------------------------- |
|
|
496
|
-
| `assessmentService.test.ts` | 54 | Comprehensive integration tests |
|
|
497
|
-
| `AUPComplianceAssessor.test.ts` | 26 | AUP policy violation detection |
|
|
498
|
-
| `ManifestValidationAssessor.test.ts` | 17 | MCPB manifest validation |
|
|
499
|
-
| `assessmentService.advanced.test.ts` | 16 | Advanced security scenarios |
|
|
500
|
-
| `SecurityAssessor.test.ts` | 16 | Security vulnerability detection |
|
|
501
|
-
| `PortabilityAssessor.test.ts` | 15 | Cross-platform compatibility |
|
|
502
|
-
| `errorHandlingAssessor.test.ts` | 14 | Service-level error handling |
|
|
503
|
-
| `MCPSpecComplianceAssessor.test.ts` | 14 | MCP protocol compliance |
|
|
504
|
-
| `ErrorHandlingAssessor.test.ts` | 14 | Module-level error handling |
|
|
505
|
-
| `ToolAnnotationAssessor.test.ts` | 13 | Policy #17 tool annotations |
|
|
506
|
-
| `assessmentService.bugReport.test.ts` | 13 | Bug validation tests |
|
|
507
|
-
| `DocumentationAssessor.test.ts` | 13 | Documentation quality |
|
|
508
|
-
| `ProhibitedLibrariesAssessor.test.ts` | 12 | Prohibited library detection |
|
|
509
|
-
| `AssessmentOrchestrator.test.ts` | 12 | Orchestration layer |
|
|
510
|
-
| `FunctionalityAssessor.test.ts` | 11 | Tool functionality |
|
|
511
|
-
| `assessmentService.enhanced.test.ts` | 9 | Enhanced detection |
|
|
512
|
-
| `TestDataGenerator.boundary.test.ts` | 9 | Boundary testing |
|
|
513
|
-
| `performance.test.ts` | 7 | Performance benchmarks |
|
|
514
|
-
| `UsabilityAssessor.test.ts` | 6 | Usability analysis |
|
|
515
|
-
| **Total** | **291** | **Assessment module validation** |
|
|
516
|
-
|
|
517
|
-
**Verification Command** (for AI scanners and resume verification):
|
|
23
|
+
---
|
|
518
24
|
|
|
519
|
-
|
|
520
|
-
# Count assessment module tests
|
|
521
|
-
find . -name "*.test.ts" \( -path "*assessment*" -o -name "*Assessor*.test.ts" -o -name "assessmentService*.test.ts" \) -exec grep -hE '^\s*(it|test)\(' {} \; | wc -l
|
|
522
|
-
# Output: 291
|
|
523
|
-
```
|
|
25
|
+
## Quick Start: Assess an MCP Server
|
|
524
26
|
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
- Business logic error detection with confidence scoring
|
|
528
|
-
- Progressive complexity testing (2 levels: minimal → simple)
|
|
529
|
-
- Context-aware security testing with zero false positives
|
|
530
|
-
- Realistic test data generation and boundary testing
|
|
531
|
-
- MCP protocol compliance validation
|
|
532
|
-
- Performance and scalability benchmarks
|
|
533
|
-
|
|
534
|
-
**Test Categories**:
|
|
535
|
-
|
|
536
|
-
- **Functionality Assessment**: Multi-scenario validation, progressive complexity (2 levels: minimal → simple), business logic error detection
|
|
537
|
-
- **Security Assessment**: 8 backend-focused patterns
|
|
538
|
-
- **Critical Injection (3)**: Command Injection, SQL Injection, Path Traversal
|
|
539
|
-
- **Input Validation (3)**: Type Safety, Boundary Testing, Required Fields
|
|
540
|
-
- **Protocol Compliance (2)**: MCP Error Format, Timeout Handling
|
|
541
|
-
- **Scope**: Tests backend API security only (not LLM prompt injection)
|
|
542
|
-
- **Documentation Analysis**: README structure validation, code example extraction, parameter documentation checks
|
|
543
|
-
- **Error Handling**: MCP protocol compliance (error codes -32600 to -32603), validation quality scoring, timeout handling
|
|
544
|
-
- **Usability Evaluation**: Naming convention analysis, parameter clarity assessment, schema completeness validation
|
|
545
|
-
- **MCP Spec Compliance**: JSON-RPC 2.0 validation, protocol message format verification
|
|
546
|
-
- **Business Logic Validation Tests**: Distinguishing proper validation errors from tool failures
|
|
547
|
-
- **False Positive Detection Tests**: Ensuring "user not found" errors aren't flagged as broken tools
|
|
548
|
-
- **Optimization Tests**: Boundary scenario conditional generation, progressive complexity efficiency
|
|
549
|
-
- **Test Files**: Located in `client/src/services/__tests__/` and `client/src/services/assessment/__tests__/`
|
|
550
|
-
- **Recent Improvements**:
|
|
551
|
-
- Achieved 100% test pass rate (582 passing, 0 failing) - 2025-10-11
|
|
552
|
-
- Updated all tests for focused backend testing (8 security patterns × 3 payloads) - 2025-10-12
|
|
553
|
-
- Fixed all failing tests after upstream sync - 2025-10-04
|
|
554
|
-
- Added boundary testing optimization validation - 2025-10-05
|
|
555
|
-
|
|
556
|
-
**Running the Test Suite**:
|
|
27
|
+
Run a full assessment on any MCP server:
|
|
557
28
|
|
|
558
29
|
```bash
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
**Test Quality**:
|
|
568
|
-
|
|
569
|
-
- All tests use realistic test data (not placeholder values)
|
|
570
|
-
- Tests validate both positive and negative cases
|
|
571
|
-
- Progressive complexity levels (2 levels) tested systematically
|
|
572
|
-
- Security tests cover all 8 injection attack patterns
|
|
573
|
-
- Error handling tests verify MCP standard error codes
|
|
574
|
-
- Business logic error detection validated with confidence scoring
|
|
575
|
-
- Optimization logic validated with dedicated test suites
|
|
576
|
-
|
|
577
|
-
### Assessment API
|
|
578
|
-
|
|
579
|
-
Programmatically run assessments using the CLI:
|
|
30
|
+
# Create a config file
|
|
31
|
+
cat > /tmp/config.json << 'EOF'
|
|
32
|
+
{
|
|
33
|
+
"transport": "http",
|
|
34
|
+
"url": "http://localhost:8000/mcp"
|
|
35
|
+
}
|
|
36
|
+
EOF
|
|
580
37
|
|
|
581
|
-
```bash
|
|
582
38
|
# Run full assessment
|
|
583
|
-
mcp-
|
|
584
|
-
# or with npx
|
|
585
|
-
npx @bryan-thompson/inspector-assessment-cli node build/index.js --assess
|
|
586
|
-
|
|
587
|
-
# Run specific category
|
|
588
|
-
mcp-inspector-assess-cli node build/index.js --assess functionality
|
|
39
|
+
mcp-assess-full --server my-server --config /tmp/config.json
|
|
589
40
|
|
|
590
|
-
#
|
|
591
|
-
mcp-inspector-assess-cli node build/index.js --assess --output assessment-report.json
|
|
41
|
+
# Results saved to /tmp/inspector-full-assessment-my-server.json
|
|
592
42
|
```
|
|
593
43
|
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
The full assessment CLI (`npm run assess:full`) provides comprehensive 12-module assessment with policy compliance mapping:
|
|
44
|
+
For STDIO servers (local commands):
|
|
597
45
|
|
|
598
46
|
```bash
|
|
599
|
-
|
|
600
|
-
npm run assess:full -- --server <name> --config <path>
|
|
601
|
-
|
|
602
|
-
# With source code analysis (enables External API Scanner, AUP, Portability checks)
|
|
603
|
-
npm run assess:full -- --server <name> --config <path> --source <path-to-source>
|
|
604
|
-
|
|
605
|
-
# Generate markdown report with policy compliance
|
|
606
|
-
npm run assess:full -- --server <name> --config <path> --format markdown --include-policy
|
|
607
|
-
|
|
608
|
-
# Quick pre-flight validation only
|
|
609
|
-
npm run assess:full -- --server <name> --config <path> --preflight
|
|
610
|
-
```
|
|
611
|
-
|
|
612
|
-
**CLI Options:**
|
|
613
|
-
|
|
614
|
-
| Option | Description |
|
|
615
|
-
| ----------------------------- | ------------------------------------------------------------------------------------ |
|
|
616
|
-
| `--server, -s <name>` | Server name (required) |
|
|
617
|
-
| `--config, -c <path>` | Path to server config JSON |
|
|
618
|
-
| `--output, -o <path>` | Output path (default: `/tmp/inspector-full-assessment-<server>.<ext>`) |
|
|
619
|
-
| `--source <path>` | Source code path for deep analysis (enables AUP, portability, external API scanning) |
|
|
620
|
-
| `--format, -f <type>` | Output format: `json` (default) or `markdown` |
|
|
621
|
-
| `--include-policy` | Include 30-requirement policy compliance mapping in report |
|
|
622
|
-
| `--preflight` | Quick validation only (tools exist, manifest valid, server responds) |
|
|
623
|
-
| `--pattern-config, -p <path>` | Path to custom annotation pattern JSON |
|
|
624
|
-
| `--claude-enabled` | Enable Claude Code integration for intelligent analysis |
|
|
625
|
-
| `--json` | Output only JSON path (no console summary) |
|
|
626
|
-
| `--verbose, -v` | Enable verbose logging |
|
|
627
|
-
|
|
628
|
-
**Config File Format (HTTP transport):**
|
|
629
|
-
|
|
630
|
-
```json
|
|
47
|
+
cat > /tmp/config.json << 'EOF'
|
|
631
48
|
{
|
|
632
|
-
"
|
|
633
|
-
"
|
|
634
|
-
}
|
|
635
|
-
```
|
|
636
|
-
|
|
637
|
-
**Config File Format (STDIO transport):**
|
|
638
|
-
|
|
639
|
-
```json
|
|
640
|
-
{
|
|
641
|
-
"transport": "stdio",
|
|
642
|
-
"command": "node",
|
|
643
|
-
"args": ["build/index.js"],
|
|
49
|
+
"command": "python3",
|
|
50
|
+
"args": ["server.py"],
|
|
644
51
|
"env": {}
|
|
645
52
|
}
|
|
646
|
-
|
|
647
|
-
|
|
648
|
-
**Pre-flight Mode:**
|
|
649
|
-
|
|
650
|
-
The `--preflight` flag runs quick validation without full assessment:
|
|
651
|
-
|
|
652
|
-
```bash
|
|
653
|
-
npm run assess:full -- --server my-server --config config.json --preflight
|
|
654
|
-
```
|
|
655
|
-
|
|
656
|
-
Returns JSON with:
|
|
657
|
-
|
|
658
|
-
- `passed`: boolean - overall validation result
|
|
659
|
-
- `toolCount`: number - tools discovered
|
|
660
|
-
- `manifestValid`: boolean - if manifest.json is valid (when --source provided)
|
|
661
|
-
- `serverResponsive`: boolean - if first tool call succeeds
|
|
662
|
-
- `errors`: string[] - any validation errors
|
|
663
|
-
|
|
664
|
-
**External API Scanner:**
|
|
665
|
-
|
|
666
|
-
When `--source` is provided, the External API Scanner automatically scans source code for:
|
|
667
|
-
|
|
668
|
-
- External API URLs (GitHub, Slack, AWS, OpenAI, Anthropic, etc.)
|
|
669
|
-
- Affiliation warnings (e.g., server named "github-mcp" by unaffiliated developer)
|
|
670
|
-
- Privacy policy considerations for detected services
|
|
671
|
-
|
|
672
|
-
**Policy Compliance Report:**
|
|
673
|
-
|
|
674
|
-
When `--include-policy` is used, the report includes mapping to Anthropic's 30 Software Directory Policy requirements across 5 categories:
|
|
675
|
-
|
|
676
|
-
- Safety & Security (6 requirements)
|
|
677
|
-
- Compatibility (6 requirements)
|
|
678
|
-
- Functionality (7 requirements)
|
|
679
|
-
- Developer Requirements (8 requirements)
|
|
680
|
-
- Unsupported Use Cases (3 requirements)
|
|
681
|
-
|
|
682
|
-
## Running the Inspector
|
|
683
|
-
|
|
684
|
-
### Requirements
|
|
685
|
-
|
|
686
|
-
- Node.js: ^22.7.5
|
|
53
|
+
EOF
|
|
687
54
|
|
|
688
|
-
|
|
689
|
-
|
|
690
|
-
To get up and running right away with the UI, just execute the following:
|
|
691
|
-
|
|
692
|
-
```bash
|
|
693
|
-
bunx @bryan-thompson/inspector-assessment
|
|
694
|
-
# or with npx
|
|
695
|
-
npx @bryan-thompson/inspector-assessment
|
|
55
|
+
mcp-assess-full --server my-server --config /tmp/config.json
|
|
696
56
|
```
|
|
697
57
|
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
### Docker Container
|
|
58
|
+
---
|
|
701
59
|
|
|
702
|
-
|
|
60
|
+
## CLI Commands
|
|
703
61
|
|
|
704
|
-
|
|
705
|
-
docker run --rm \
|
|
706
|
-
-p 127.0.0.1:6274:6274 \
|
|
707
|
-
-p 127.0.0.1:6277:6277 \
|
|
708
|
-
-e HOST=0.0.0.0 \
|
|
709
|
-
-e MCP_AUTO_OPEN_ENABLED=false \
|
|
710
|
-
ghcr.io/modelcontextprotocol/inspector:latest
|
|
711
|
-
```
|
|
62
|
+
The inspector provides three CLI commands for different workflows:
|
|
712
63
|
|
|
713
|
-
|
|
64
|
+
| Command | Purpose | Use Case |
|
|
65
|
+
| ---------------------- | ----------------------------- | ---------------------------- |
|
|
66
|
+
| `mcp-assess-full` | Complete 16-module assessment | Full validation, CI/CD gates |
|
|
67
|
+
| `mcp-assess-security` | Security-only testing | Quick vulnerability scan |
|
|
68
|
+
| `mcp-inspector-assess` | Interactive web UI | Debugging, exploration |
|
|
714
69
|
|
|
715
|
-
|
|
70
|
+
### Common Options
|
|
716
71
|
|
|
717
72
|
```bash
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
npx @bryan-thompson/inspector-assessment node build/index.js
|
|
721
|
-
```
|
|
73
|
+
# Full assessment with all modules
|
|
74
|
+
mcp-assess-full --server <name> --config <path>
|
|
722
75
|
|
|
723
|
-
|
|
76
|
+
# Security-only (faster)
|
|
77
|
+
mcp-assess-security --server <name> --config <path>
|
|
724
78
|
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
bunx @bryan-thompson/inspector-assessment node build/index.js arg1 arg2
|
|
79
|
+
# Skip slow modules for CI/CD
|
|
80
|
+
mcp-assess-full --server <name> --skip-modules temporal,security
|
|
728
81
|
|
|
729
|
-
#
|
|
730
|
-
|
|
82
|
+
# Run only specific modules
|
|
83
|
+
mcp-assess-full --server <name> --only-modules functionality,toolAnnotations
|
|
731
84
|
|
|
732
|
-
#
|
|
733
|
-
|
|
85
|
+
# Generate markdown report
|
|
86
|
+
mcp-assess-full --server <name> --format markdown --output report.md
|
|
734
87
|
|
|
735
|
-
#
|
|
736
|
-
|
|
88
|
+
# Pre-flight validation (quick check)
|
|
89
|
+
mcp-assess-full --server <name> --preflight
|
|
737
90
|
```
|
|
738
91
|
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
```bash
|
|
742
|
-
CLIENT_PORT=8080 SERVER_PORT=9000 bunx @bryan-thompson/inspector-assessment node build/index.js
|
|
743
|
-
```
|
|
744
|
-
|
|
745
|
-
For more details on ways to use the inspector, see the [Inspector section of the MCP docs site](https://modelcontextprotocol.io/docs/tools/inspector). For help with debugging, see the [Debugging guide](https://modelcontextprotocol.io/docs/tools/debugging).
|
|
746
|
-
|
|
747
|
-
### Servers File Export
|
|
748
|
-
|
|
749
|
-
The MCP Inspector provides convenient buttons to export server launch configurations for use in clients such as Cursor, Claude Code, or the Inspector's CLI. The file is usually called `mcp.json`.
|
|
750
|
-
|
|
751
|
-
- **Server Entry** - Copies a single server configuration entry to your clipboard. This can be added to your `mcp.json` file inside the `mcpServers` object with your preferred server name.
|
|
752
|
-
|
|
753
|
-
**STDIO transport example:**
|
|
754
|
-
|
|
755
|
-
```json
|
|
756
|
-
{
|
|
757
|
-
"command": "node",
|
|
758
|
-
"args": ["build/index.js", "--debug"],
|
|
759
|
-
"env": {
|
|
760
|
-
"API_KEY": "your-api-key",
|
|
761
|
-
"DEBUG": "true"
|
|
762
|
-
}
|
|
763
|
-
}
|
|
764
|
-
```
|
|
765
|
-
|
|
766
|
-
**SSE transport example:**
|
|
767
|
-
|
|
768
|
-
```json
|
|
769
|
-
{
|
|
770
|
-
"type": "sse",
|
|
771
|
-
"url": "http://localhost:3000/events",
|
|
772
|
-
"note": "For SSE connections, add this URL directly in Client"
|
|
773
|
-
}
|
|
774
|
-
```
|
|
775
|
-
|
|
776
|
-
**Streamable HTTP transport example:**
|
|
777
|
-
|
|
778
|
-
```json
|
|
779
|
-
{
|
|
780
|
-
"type": "streamable-http",
|
|
781
|
-
"url": "http://localhost:3000/mcp",
|
|
782
|
-
"note": "For Streamable HTTP connections, add this URL directly in your MCP Client"
|
|
783
|
-
}
|
|
784
|
-
```
|
|
92
|
+
For complete CLI documentation, see [CLI Assessment Guide](docs/CLI_ASSESSMENT_GUIDE.md).
|
|
785
93
|
|
|
786
|
-
|
|
787
|
-
|
|
788
|
-
**STDIO transport example:**
|
|
789
|
-
|
|
790
|
-
```json
|
|
791
|
-
{
|
|
792
|
-
"mcpServers": {
|
|
793
|
-
"default-server": {
|
|
794
|
-
"command": "node",
|
|
795
|
-
"args": ["build/index.js", "--debug"],
|
|
796
|
-
"env": {
|
|
797
|
-
"API_KEY": "your-api-key",
|
|
798
|
-
"DEBUG": "true"
|
|
799
|
-
}
|
|
800
|
-
}
|
|
801
|
-
}
|
|
802
|
-
}
|
|
803
|
-
```
|
|
804
|
-
|
|
805
|
-
**SSE transport example:**
|
|
806
|
-
|
|
807
|
-
```json
|
|
808
|
-
{
|
|
809
|
-
"mcpServers": {
|
|
810
|
-
"default-server": {
|
|
811
|
-
"type": "sse",
|
|
812
|
-
"url": "http://localhost:3000/events",
|
|
813
|
-
"note": "For SSE connections, add this URL directly in Client"
|
|
814
|
-
}
|
|
815
|
-
}
|
|
816
|
-
}
|
|
817
|
-
```
|
|
818
|
-
|
|
819
|
-
**Streamable HTTP transport example:**
|
|
820
|
-
|
|
821
|
-
```json
|
|
822
|
-
{
|
|
823
|
-
"mcpServers": {
|
|
824
|
-
"default-server": {
|
|
825
|
-
"type": "streamable-http",
|
|
826
|
-
"url": "http://localhost:3000/mcp",
|
|
827
|
-
"note": "For Streamable HTTP connections, add this URL directly in your MCP Client"
|
|
828
|
-
}
|
|
829
|
-
}
|
|
830
|
-
}
|
|
831
|
-
```
|
|
832
|
-
|
|
833
|
-
These buttons appear in the Inspector UI after you've configured your server settings, making it easy to save and reuse your configurations.
|
|
834
|
-
|
|
835
|
-
For SSE and Streamable HTTP transport connections, the Inspector provides similar functionality for both buttons. The "Server Entry" button copies the configuration that can be added to your existing configuration file, while the "Servers File" button creates a complete configuration file containing the URL for direct use in clients.
|
|
836
|
-
|
|
837
|
-
You can paste the Server Entry into your existing `mcp.json` file under your chosen server name, or use the complete Servers File payload to create a new configuration file.
|
|
838
|
-
|
|
839
|
-
### Authentication
|
|
840
|
-
|
|
841
|
-
The inspector supports bearer token authentication for SSE connections. Enter your token in the UI when connecting to an MCP server, and it will be sent in the Authorization header. You can override the header name using the input field in the sidebar.
|
|
842
|
-
|
|
843
|
-
### Security Considerations
|
|
844
|
-
|
|
845
|
-
The MCP Inspector includes a proxy server that can run and communicate with local MCP processes. The proxy server should not be exposed to untrusted networks as it has permissions to spawn local processes and can connect to any specified MCP server.
|
|
846
|
-
|
|
847
|
-
#### Authentication
|
|
848
|
-
|
|
849
|
-
The MCP Inspector proxy server requires authentication by default. When starting the server, a random session token is generated and printed to the console:
|
|
94
|
+
---
|
|
850
95
|
|
|
851
|
-
|
|
852
|
-
🔑 Session token: 3a1c267fad21f7150b7d624c160b7f09b0b8c4f623c7107bbf13378f051538d4
|
|
96
|
+
## Assessment Modules (16 Total)
|
|
853
97
|
|
|
854
|
-
|
|
855
|
-
http://localhost:6274/?MCP_PROXY_AUTH_TOKEN=3a1c267fad21f7150b7d624c160b7f09b0b8c4f623c7107bbf13378f051538d4
|
|
856
|
-
```
|
|
98
|
+
### Core Modules (5)
|
|
857
99
|
|
|
858
|
-
|
|
100
|
+
| Module | Purpose | Key Features |
|
|
101
|
+
| ------------------ | -------------------------- | --------------------------------------------------- |
|
|
102
|
+
| **Functionality** | Tool execution validation | Multi-scenario testing, business logic detection |
|
|
103
|
+
| **Security** | Vulnerability detection | Comprehensive attack patterns, zero false positives |
|
|
104
|
+
| **Documentation** | README/description quality | Completeness scoring, example validation |
|
|
105
|
+
| **Error Handling** | MCP protocol compliance | Error code validation, response quality |
|
|
106
|
+
| **Usability** | Developer experience | Naming conventions, schema completeness |
|
|
859
107
|
|
|
860
|
-
|
|
108
|
+
### Extended Modules (6) - MCP Directory Compliance
|
|
861
109
|
|
|
862
|
-
|
|
110
|
+
| Module | Purpose | Policy Alignment |
|
|
111
|
+
| ------------------------ | ---------------------------- | --------------------------------------- |
|
|
112
|
+
| **MCP Spec Compliance** | Protocol adherence | JSON-RPC 2.0, MCP message formats |
|
|
113
|
+
| **AUP Compliance** | Policy violation detection | 14 AUP categories (A-N) |
|
|
114
|
+
| **Tool Annotations** | readOnlyHint/destructiveHint | Policy #17 compliance |
|
|
115
|
+
| **Prohibited Libraries** | Dependency security | Blocked packages (Stripe, FFmpeg, etc.) |
|
|
116
|
+
| **Manifest Validation** | MCPB manifest.json | manifest_version 0.3 spec |
|
|
117
|
+
| **Portability** | Cross-platform compatibility | Hardcoded paths, platform-specific code |
|
|
863
118
|
|
|
864
|
-
|
|
865
|
-
2. Find "Proxy Session Token" and enter the token displayed in the proxy console
|
|
866
|
-
3. Click "Save" to apply the configuration
|
|
119
|
+
### Advanced Modules (5)
|
|
867
120
|
|
|
868
|
-
|
|
121
|
+
| Module | Purpose | Features |
|
|
122
|
+
| ------------------------ | -------------------------- | --------------------------------- |
|
|
123
|
+
| **External API Scanner** | External service detection | API URLs, affiliation warnings |
|
|
124
|
+
| **Temporal** | Rug pull detection | Behavior changes over invocations |
|
|
125
|
+
| **Resources** | Resource capability | Discovery, read success, errors |
|
|
126
|
+
| **Prompts** | Prompt capability | Execution, multimodal support |
|
|
127
|
+
| **Cross-Capability** | Chained vulnerabilities | Multi-tool attack patterns |
|
|
869
128
|
|
|
870
|
-
|
|
871
|
-
|
|
872
|
-
```bash
|
|
873
|
-
DANGEROUSLY_OMIT_AUTH=true npm start
|
|
874
|
-
```
|
|
129
|
+
For detailed module documentation, see [Assessment Catalog](docs/ASSESSMENT_CATALOG.md).
|
|
875
130
|
|
|
876
131
|
---
|
|
877
132
|
|
|
878
|
-
|
|
133
|
+
## For MCP Directory Reviewers
|
|
879
134
|
|
|
880
|
-
|
|
135
|
+
If you're reviewing MCP servers for the Anthropic MCP Directory, see our **[Reviewer Quick Start Guide](docs/REVIEWER_QUICK_START.md)** for:
|
|
881
136
|
|
|
882
|
-
|
|
137
|
+
- **60-second fast screening** workflow for approve/reject decisions
|
|
138
|
+
- **5-minute detailed review** process for borderline cases
|
|
139
|
+
- **Common pitfalls** explanation (false positives in security, informational vs scored tests)
|
|
140
|
+
- **Decision matrix** with clear approval criteria
|
|
883
141
|
|
|
884
142
|
---
|
|
885
143
|
|
|
886
|
-
|
|
144
|
+
## Security Testing: Pure Behavior Detection
|
|
887
145
|
|
|
888
|
-
|
|
889
|
-
MCP_PROXY_AUTH_TOKEN=$(openssl rand -hex 32) npm start
|
|
890
|
-
```
|
|
146
|
+
The inspector uses **pure behavior-based detection** for security assessment, analyzing tool responses to identify actual code execution vs safe data handling.
|
|
891
147
|
|
|
892
|
-
|
|
893
|
-
|
|
894
|
-
By default, both the MCP Inspector proxy server and client bind only to `localhost` to prevent network access. This ensures they are not accessible from other devices on the network. If you need to bind to all interfaces for development purposes, you can override this with the `HOST` environment variable:
|
|
148
|
+
### How It Works
|
|
895
149
|
|
|
896
150
|
```bash
|
|
897
|
-
|
|
151
|
+
# Run security assessment
|
|
152
|
+
mcp-assess-security --server my-server --config config.json
|
|
898
153
|
```
|
|
899
154
|
|
|
900
|
-
**
|
|
155
|
+
**Detection Strategy:**
|
|
901
156
|
|
|
902
|
-
|
|
157
|
+
1. **Reflection Detection**: Identifies when tools safely echo malicious input as data
|
|
158
|
+
- `"Stored query: ../../../etc/passwd"` → SAFE (reflection)
|
|
159
|
+
- `"Query results for: ..."` → SAFE (search results)
|
|
903
160
|
|
|
904
|
-
|
|
161
|
+
2. **Execution Evidence**: Detects actual code execution
|
|
162
|
+
- Response contains `"root:x:0:0"` → VULNERABLE (file accessed)
|
|
163
|
+
- Response contains `"total 42 drwx"` → VULNERABLE (directory listed)
|
|
905
164
|
|
|
906
|
-
|
|
907
|
-
|
|
908
|
-
|
|
165
|
+
3. **Category Classification**: Distinguishes safe tool types
|
|
166
|
+
- Search/retrieval tools return data, not code execution
|
|
167
|
+
- CRUD operations create resources, not execute code
|
|
909
168
|
|
|
910
|
-
###
|
|
169
|
+
### Supported Attack Patterns
|
|
911
170
|
|
|
912
|
-
|
|
171
|
+
- Command Injection, SQL Injection, Path Traversal, XXE, NoSQL Injection
|
|
172
|
+
- Calculator Injection, Code Execution (Python/JS)
|
|
173
|
+
- Data Exfiltration, Token Theft, Permission Scope
|
|
174
|
+
- Unicode Bypass, Nested Injection, Package Squatting
|
|
175
|
+
- DoS/Resource Exhaustion, Insecure Deserialization
|
|
176
|
+
- Configuration Drift, Tool Shadowing
|
|
913
177
|
|
|
914
|
-
|
|
915
|
-
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
|
|
916
|
-
| `MCP_SERVER_REQUEST_TIMEOUT` | Client-side timeout (ms) - Inspector will cancel the request if no response is received within this time. Note: servers may have their own timeouts | 300000 |
|
|
917
|
-
| `MCP_REQUEST_TIMEOUT_RESET_ON_PROGRESS` | Reset timeout on progress notifications | true |
|
|
918
|
-
| `MCP_REQUEST_MAX_TOTAL_TIMEOUT` | Maximum total timeout for requests sent to the MCP server (ms) (Use with progress notifications) | 60000 |
|
|
919
|
-
| `MCP_PROXY_FULL_ADDRESS` | Set this if you are running the MCP Inspector Proxy on a non-default address. Example: http://10.1.1.22:5577 | "" |
|
|
920
|
-
| `MCP_AUTO_OPEN_ENABLED` | Enable automatic browser opening when inspector starts (works with authentication enabled). Only as environment var, not configurable in browser. | true |
|
|
178
|
+
See [Security Patterns Catalog](docs/SECURITY_PATTERNS_CATALOG.md) for complete pattern documentation.
|
|
921
179
|
|
|
922
|
-
|
|
180
|
+
---
|
|
923
181
|
|
|
924
|
-
|
|
182
|
+
## Testbed Validation
|
|
925
183
|
|
|
926
|
-
The inspector
|
|
184
|
+
The inspector is validated against purpose-built testbed servers with ground-truth labeled tools:
|
|
927
185
|
|
|
928
186
|
```bash
|
|
929
|
-
|
|
930
|
-
|
|
931
|
-
|
|
932
|
-
Example server configuration file:
|
|
933
|
-
|
|
934
|
-
```json
|
|
935
|
-
{
|
|
936
|
-
"mcpServers": {
|
|
937
|
-
"everything": {
|
|
938
|
-
"command": "npx",
|
|
939
|
-
"args": ["@modelcontextprotocol/server-everything"],
|
|
940
|
-
"env": {
|
|
941
|
-
"hello": "Hello MCP!"
|
|
942
|
-
}
|
|
943
|
-
},
|
|
944
|
-
"my-server": {
|
|
945
|
-
"command": "node",
|
|
946
|
-
"args": ["build/index.js", "arg1", "arg2"],
|
|
947
|
-
"env": {
|
|
948
|
-
"key": "value",
|
|
949
|
-
"key2": "value2"
|
|
950
|
-
}
|
|
951
|
-
}
|
|
952
|
-
}
|
|
953
|
-
}
|
|
954
|
-
```
|
|
955
|
-
|
|
956
|
-
#### Transport Types in Config Files
|
|
187
|
+
# Test against vulnerable-mcp testbed (10 vulnerable + 6 safe tools)
|
|
188
|
+
npm run assess -- --server vulnerable-mcp --config /tmp/vulnerable-mcp-config.json
|
|
189
|
+
# Results: 200 vulnerabilities detected, 0 false positives (100% precision)
|
|
957
190
|
|
|
958
|
-
|
|
959
|
-
|
|
960
|
-
|
|
961
|
-
|
|
962
|
-
```json
|
|
963
|
-
{
|
|
964
|
-
"mcpServers": {
|
|
965
|
-
"my-stdio-server": {
|
|
966
|
-
"type": "stdio",
|
|
967
|
-
"command": "npx",
|
|
968
|
-
"args": ["@modelcontextprotocol/server-everything"]
|
|
969
|
-
}
|
|
970
|
-
}
|
|
971
|
-
}
|
|
191
|
+
# Test against hardened-mcp testbed (same tool names, safe implementations)
|
|
192
|
+
npm run assess -- --server hardened-mcp --config /tmp/hardened-mcp-config.json
|
|
193
|
+
# Results: 0 vulnerabilities (proves behavior-based detection, not name-based)
|
|
972
194
|
```
|
|
973
195
|
|
|
974
|
-
**
|
|
975
|
-
|
|
976
|
-
```json
|
|
977
|
-
{
|
|
978
|
-
"mcpServers": {
|
|
979
|
-
"my-sse-server": {
|
|
980
|
-
"type": "sse",
|
|
981
|
-
"url": "http://localhost:3000/sse"
|
|
982
|
-
}
|
|
983
|
-
}
|
|
984
|
-
}
|
|
985
|
-
```
|
|
196
|
+
**Key Insight**: Both servers have tools named `vulnerable_calculator_tool`, `vulnerable_system_exec_tool`, etc. The inspector detects 200 vulnerabilities on one server and 0 on the other - proving pure behavior-based detection, not name-based heuristics.
|
|
986
197
|
|
|
987
|
-
|
|
198
|
+
See [Testbed Setup Guide](docs/TESTBED_SETUP_GUIDE.md) for detailed validation results.
|
|
988
199
|
|
|
989
|
-
|
|
990
|
-
{
|
|
991
|
-
"mcpServers": {
|
|
992
|
-
"my-http-server": {
|
|
993
|
-
"type": "streamable-http",
|
|
994
|
-
"url": "http://localhost:3000/mcp"
|
|
995
|
-
}
|
|
996
|
-
}
|
|
997
|
-
}
|
|
998
|
-
```
|
|
200
|
+
---
|
|
999
201
|
|
|
1000
|
-
|
|
202
|
+
## Assessment Output
|
|
1001
203
|
|
|
1002
|
-
|
|
204
|
+
### JSON Results
|
|
1003
205
|
|
|
1004
|
-
|
|
206
|
+
Every assessment saves results to JSON:
|
|
1005
207
|
|
|
1006
208
|
```bash
|
|
1007
|
-
#
|
|
1008
|
-
|
|
1009
|
-
```
|
|
1010
|
-
|
|
1011
|
-
2. **A server named "default-server"** - automatically selected:
|
|
1012
|
-
|
|
1013
|
-
```json
|
|
1014
|
-
{
|
|
1015
|
-
"mcpServers": {
|
|
1016
|
-
"default-server": {
|
|
1017
|
-
"command": "npx",
|
|
1018
|
-
"args": ["@modelcontextprotocol/server-everything"]
|
|
1019
|
-
},
|
|
1020
|
-
"other-server": {
|
|
1021
|
-
"command": "node",
|
|
1022
|
-
"args": ["other.js"]
|
|
1023
|
-
}
|
|
1024
|
-
}
|
|
1025
|
-
}
|
|
1026
|
-
```
|
|
1027
|
-
|
|
1028
|
-
> **Tip:** You can easily generate this configuration format using the **Server Entry** and **Servers File** buttons in the Inspector UI, as described in the Servers File Export section above.
|
|
1029
|
-
|
|
1030
|
-
You can also set the initial `transport` type, `serverUrl`, `serverCommand`, and `serverArgs` via query params, for example:
|
|
1031
|
-
|
|
1032
|
-
```
|
|
1033
|
-
http://localhost:6274/?transport=sse&serverUrl=http://localhost:8787/sse
|
|
1034
|
-
http://localhost:6274/?transport=streamable-http&serverUrl=http://localhost:8787/mcp
|
|
1035
|
-
http://localhost:6274/?transport=stdio&serverCommand=npx&serverArgs=arg1%20arg2
|
|
1036
|
-
```
|
|
209
|
+
# Default location
|
|
210
|
+
/tmp/inspector-full-assessment-<server-name>.json
|
|
1037
211
|
|
|
1038
|
-
|
|
1039
|
-
|
|
1040
|
-
```
|
|
1041
|
-
http://localhost:6274/?MCP_SERVER_REQUEST_TIMEOUT=60000&MCP_REQUEST_TIMEOUT_RESET_ON_PROGRESS=false&MCP_PROXY_FULL_ADDRESS=http://10.1.1.22:5577
|
|
212
|
+
# Custom output
|
|
213
|
+
mcp-assess-full --server my-server --output ./results.json
|
|
1042
214
|
```
|
|
1043
215
|
|
|
1044
|
-
|
|
1045
|
-
|
|
1046
|
-
### From this repository
|
|
1047
|
-
|
|
1048
|
-
If you're working on the inspector itself:
|
|
1049
|
-
|
|
1050
|
-
Development mode:
|
|
216
|
+
**Quick Analysis:**
|
|
1051
217
|
|
|
1052
218
|
```bash
|
|
1053
|
-
|
|
1054
|
-
|
|
1055
|
-
# To co-develop with the typescript-sdk package (assuming it's cloned in ../typescript-sdk; set MCP_SDK otherwise):
|
|
1056
|
-
npm run dev:sdk "cd sdk && npm run examples:simple-server:w"
|
|
1057
|
-
# then open http://localhost:3000/mcp as SHTTP in the inspector.
|
|
1058
|
-
# To go back to the deployed SDK version:
|
|
1059
|
-
# npm run unlink:sdk && npm i
|
|
1060
|
-
```
|
|
219
|
+
# View overall status
|
|
220
|
+
cat /tmp/inspector-full-assessment-my-server.json | jq '.overallStatus'
|
|
1061
221
|
|
|
1062
|
-
|
|
1063
|
-
|
|
1064
|
-
>
|
|
1065
|
-
> ```bash
|
|
1066
|
-
> npm run dev:windows
|
|
1067
|
-
> ```
|
|
222
|
+
# List security vulnerabilities
|
|
223
|
+
cat /tmp/inspector-full-assessment-my-server.json | jq '.security.vulnerabilities'
|
|
1068
224
|
|
|
1069
|
-
|
|
225
|
+
# Check broken tools
|
|
226
|
+
cat /tmp/inspector-full-assessment-my-server.json | jq '.functionality.brokenTools'
|
|
1070
227
|
|
|
1071
|
-
|
|
1072
|
-
|
|
1073
|
-
npm start
|
|
228
|
+
# Get module scores
|
|
229
|
+
cat /tmp/inspector-full-assessment-my-server.json | jq '.moduleSummary'
|
|
1074
230
|
```
|
|
1075
231
|
|
|
1076
|
-
###
|
|
1077
|
-
|
|
1078
|
-
CLI mode enables programmatic interaction with MCP servers from the command line, ideal for scripting, automation, and integration with coding assistants. This creates an efficient feedback loop for MCP server development.
|
|
232
|
+
### Exit Codes
|
|
1079
233
|
|
|
1080
234
|
```bash
|
|
1081
|
-
mcp-
|
|
235
|
+
mcp-assess-full --server my-server
|
|
236
|
+
echo $?
|
|
237
|
+
# 0 = PASS (all modules passed)
|
|
238
|
+
# 1 = FAIL (vulnerabilities or failures found)
|
|
1082
239
|
```
|
|
1083
240
|
|
|
1084
|
-
|
|
1085
|
-
|
|
1086
|
-
```bash
|
|
1087
|
-
# Basic usage
|
|
1088
|
-
mcp-inspector-assess-cli node build/index.js
|
|
1089
|
-
|
|
1090
|
-
# With config file
|
|
1091
|
-
mcp-inspector-assess-cli --config path/to/config.json --server myserver
|
|
1092
|
-
|
|
1093
|
-
# List available tools
|
|
1094
|
-
mcp-inspector-assess-cli node build/index.js --method tools/list
|
|
1095
|
-
|
|
1096
|
-
# Call a specific tool
|
|
1097
|
-
mcp-inspector-assess-cli node build/index.js --method tools/call --tool-name mytool --tool-arg key=value --tool-arg another=value2
|
|
1098
|
-
|
|
1099
|
-
# Call a tool with JSON arguments
|
|
1100
|
-
mcp-inspector-assess-cli node build/index.js --method tools/call --tool-name mytool --tool-arg 'options={"format": "json", "max_tokens": 100}'
|
|
1101
|
-
|
|
1102
|
-
# List available resources
|
|
1103
|
-
mcp-inspector-assess-cli node build/index.js --method resources/list
|
|
1104
|
-
|
|
1105
|
-
# List available prompts
|
|
1106
|
-
mcp-inspector-assess-cli node build/index.js --method prompts/list
|
|
1107
|
-
|
|
1108
|
-
# Connect to a remote MCP server (default is SSE transport)
|
|
1109
|
-
mcp-inspector-assess-cli https://my-mcp-server.example.com
|
|
1110
|
-
|
|
1111
|
-
# Connect to a remote MCP server (with Streamable HTTP transport)
|
|
1112
|
-
mcp-inspector-assess-cli https://my-mcp-server.example.com --transport http --method tools/list
|
|
1113
|
-
|
|
1114
|
-
# Connect to a remote MCP server (with custom headers)
|
|
1115
|
-
mcp-inspector-assess-cli https://my-mcp-server.example.com --transport http --method tools/list --header "X-API-Key: your-api-key"
|
|
1116
|
-
|
|
1117
|
-
# Call a tool on a remote server
|
|
1118
|
-
mcp-inspector-assess-cli https://my-mcp-server.example.com --method tools/call --tool-name remotetool --tool-arg param=value
|
|
1119
|
-
|
|
1120
|
-
# List resources from a remote server
|
|
1121
|
-
mcp-inspector-assess-cli https://my-mcp-server.example.com --method resources/list
|
|
1122
|
-
```
|
|
241
|
+
---
|
|
1123
242
|
|
|
1124
|
-
|
|
243
|
+
## Quality Metrics
|
|
1125
244
|
|
|
1126
|
-
|
|
245
|
+
- **Test Coverage**: ~1550 tests passing across 65 test suites
|
|
246
|
+
- **Assessment Module Tests**: 291+ tests validating assessment enhancements
|
|
247
|
+
- **Code Quality**: Production TypeScript types, proper error handling
|
|
248
|
+
- **Upstream Sync**: Up-to-date with v0.18.0
|
|
1127
249
|
|
|
1128
|
-
**
|
|
250
|
+
**Run tests:**
|
|
1129
251
|
|
|
1130
252
|
```bash
|
|
1131
|
-
#
|
|
1132
|
-
npm
|
|
253
|
+
npm test # All ~1550 tests
|
|
254
|
+
npm test -- assessment # Assessment module tests
|
|
255
|
+
npm test -- SecurityAssessor # Security tests
|
|
1133
256
|
```
|
|
1134
257
|
|
|
1135
|
-
|
|
1136
|
-
|
|
1137
|
-
1. **Reflection Detection**: Identifies when tools safely echo malicious input as data
|
|
1138
|
-
- Pattern: "Stored query: ../../../etc/passwd" → SAFE (reflection)
|
|
1139
|
-
- Pattern: "Query results for: ..." → SAFE (search results)
|
|
1140
|
-
|
|
1141
|
-
2. **Execution Evidence**: Detects actual code execution
|
|
1142
|
-
- Pattern: Response contains "root:x:0:0" → VULNERABLE (file accessed)
|
|
1143
|
-
- Pattern: Response contains "total 42 drwx" → VULNERABLE (directory listed)
|
|
258
|
+
---
|
|
1144
259
|
|
|
1145
|
-
|
|
1146
|
-
- Search/retrieval tools return data, not code execution
|
|
1147
|
-
- CRUD operations create resources, not execute code
|
|
1148
|
-
- Safe storage tools treat input as pure data
|
|
260
|
+
## Documentation
|
|
1149
261
|
|
|
1150
|
-
|
|
262
|
+
### Quick Start
|
|
1151
263
|
|
|
1152
|
-
|
|
264
|
+
| Document | Purpose |
|
|
265
|
+
| ------------------------------------------------------ | ------------------------------------------- |
|
|
266
|
+
| [Reviewer Quick Start](docs/REVIEWER_QUICK_START.md) | 60-second screening for directory reviewers |
|
|
267
|
+
| [CLI Assessment Guide](docs/CLI_ASSESSMENT_GUIDE.md) | Complete CLI modes and options |
|
|
268
|
+
| [Architecture & Value](docs/ARCHITECTURE_AND_VALUE.md) | What this provides and why |
|
|
1153
269
|
|
|
1154
|
-
|
|
1155
|
-
# Test against broken-mcp testbed (10 vulnerable + 6 safe tools)
|
|
1156
|
-
npm run assess -- --server broken-mcp --config testbed.json
|
|
270
|
+
### API & Integration
|
|
1157
271
|
|
|
1158
|
-
|
|
1159
|
-
|
|
272
|
+
| Document | Purpose |
|
|
273
|
+
| -------------------------------------------------------- | ---------------------------- |
|
|
274
|
+
| [Programmatic API Guide](docs/PROGRAMMATIC_API_GUIDE.md) | AssessmentOrchestrator usage |
|
|
275
|
+
| [API Reference](docs/API_REFERENCE.md) | Complete API documentation |
|
|
276
|
+
| [Integration Guide](docs/INTEGRATION_GUIDE.md) | CI/CD, multi-server patterns |
|
|
1160
277
|
|
|
1161
|
-
|
|
278
|
+
### Assessment Details
|
|
1162
279
|
|
|
1163
|
-
|
|
280
|
+
| Document | Purpose |
|
|
281
|
+
| -------------------------------------------------------------- | ----------------------------- |
|
|
282
|
+
| [Assessment Catalog](docs/ASSESSMENT_CATALOG.md) | All 16 modules reference |
|
|
283
|
+
| [Security Patterns Catalog](docs/SECURITY_PATTERNS_CATALOG.md) | Comprehensive attack patterns |
|
|
284
|
+
| [Testbed Setup Guide](docs/TESTBED_SETUP_GUIDE.md) | A/B validation |
|
|
1164
285
|
|
|
1165
|
-
|
|
286
|
+
For complete documentation, see [docs/README.md](docs/README.md).
|
|
1166
287
|
|
|
1167
|
-
|
|
1168
|
-
|
|
1169
|
-
```bash
|
|
1170
|
-
# Before changes: Record baseline
|
|
1171
|
-
npm run assess -- --server broken-mcp --output /tmp/baseline.json
|
|
1172
|
-
|
|
1173
|
-
# After changes: Verify no regressions
|
|
1174
|
-
npm run assess -- --server broken-mcp --output /tmp/after.json
|
|
288
|
+
---
|
|
1175
289
|
|
|
1176
|
-
|
|
1177
|
-
cat /tmp/after.json | jq '[.security.promptInjectionTests[] | select(.toolName | startswith("safe_")) | select(.vulnerable == true)] | length'
|
|
1178
|
-
# Output: 0
|
|
1179
|
-
```
|
|
290
|
+
## Evidence & Validation
|
|
1180
291
|
|
|
1181
|
-
|
|
292
|
+
All performance claims are backed by implementation analysis.
|
|
1182
293
|
|
|
1183
|
-
|
|
294
|
+
| Claim | Evidence |
|
|
295
|
+
| --------------------------------- | --------------------------------------------------------------------------------- |
|
|
296
|
+
| Progressive complexity (2 levels) | [TestScenarioEngine.ts](client/src/services/assessment/TestScenarioEngine.ts) |
|
|
297
|
+
| Comprehensive security patterns | [securityPatterns.ts](client/src/lib/securityPatterns.ts) |
|
|
298
|
+
| Zero false positives | [SecurityAssessor.ts](client/src/services/assessment/modules/SecurityAssessor.ts) |
|
|
1184
299
|
|
|
1185
|
-
|
|
1186
|
-
| ------------------------ | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
1187
|
-
| **Server Development** | Visual interface for interactive testing and debugging during development | Scriptable commands for quick testing and continuous integration; creates feedback loops with AI coding assistants like Cursor for rapid development |
|
|
1188
|
-
| **Resource Exploration** | Interactive browser with hierarchical navigation and JSON visualization | Programmatic listing and reading for automation and scripting |
|
|
1189
|
-
| **Tool Testing** | Form-based parameter input with real-time response visualization | Command-line tool execution with JSON output for scripting |
|
|
1190
|
-
| **Prompt Engineering** | Interactive sampling with streaming responses and visual comparison | Batch processing of prompts with machine-readable output |
|
|
1191
|
-
| **Debugging** | Request history, visualized errors, and real-time notifications | Direct JSON output for log analysis and integration with other tools |
|
|
1192
|
-
| **Automation** | N/A | Ideal for CI/CD pipelines, batch processing, and integration with coding assistants |
|
|
1193
|
-
| **Learning MCP** | Rich visual interface helps new users understand server capabilities | Simplified commands for focused learning of specific endpoints |
|
|
300
|
+
---
|
|
1194
301
|
|
|
1195
|
-
##
|
|
302
|
+
## Contributing
|
|
1196
303
|
|
|
1197
|
-
|
|
304
|
+
We welcome contributions! See [PROJECT_STATUS.md](PROJECT_STATUS.md) for current development status.
|
|
1198
305
|
|
|
1199
|
-
|
|
1200
|
-
- **Preserve explicit default values** - If a field schema contains an explicit default (e.g., `default: null`), and the current value matches that default, include it in the request. This is a meaningful value the tool expects
|
|
1201
|
-
- **Always include required fields** - Preserve required field values even when empty, allowing the MCP server to validate and return appropriate error messages
|
|
1202
|
-
- **Defer deep validation to the server** - Implement basic field presence checking in the Inspector client, but rely on the MCP server for parameter validation according to its schema
|
|
306
|
+
**Areas of interest:**
|
|
1203
307
|
|
|
1204
|
-
|
|
308
|
+
- Additional security patterns
|
|
309
|
+
- Performance optimizations
|
|
310
|
+
- CI/CD integration examples
|
|
311
|
+
- New assessment modules
|
|
1205
312
|
|
|
1206
|
-
|
|
313
|
+
**Repository**: https://github.com/triepod-ai/inspector-assessment
|
|
1207
314
|
|
|
1208
|
-
|
|
315
|
+
---
|
|
1209
316
|
|
|
1210
|
-
|
|
317
|
+
## Links
|
|
1211
318
|
|
|
1212
|
-
|
|
319
|
+
- **npm Package**: https://www.npmjs.com/package/@bryan-thompson/inspector-assessment
|
|
320
|
+
- **GitHub Repository**: https://github.com/triepod-ai/inspector-assessment
|
|
321
|
+
- **Issues**: https://github.com/triepod-ai/inspector-assessment/issues
|
|
322
|
+
- **MCP Documentation**: https://modelcontextprotocol.io
|
|
323
|
+
- **Changelog**: [CHANGELOG.md](CHANGELOG.md)
|
|
1213
324
|
|
|
1214
|
-
|
|
1215
|
-
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
|
|
1216
|
-
| Progressive complexity testing (2 levels) | Implementation in [TestScenarioEngine.ts](client/src/services/assessment/TestScenarioEngine.ts) | Measured |
|
|
1217
|
-
| 50% faster comprehensive testing | Analysis in [PHASE1_OPTIMIZATION_COMPLETED.md](docs/PHASE1_OPTIMIZATION_COMPLETED.md) and [COMPREHENSIVE_TESTING_ANALYSIS.md](docs/COMPREHENSIVE_TESTING_ANALYSIS.md) | Measured |
|
|
1218
|
-
| 8 backend security patterns | Implementation in [securityPatterns.ts](client/src/lib/securityPatterns.ts) - focused on API security, not LLM behaviors | Measured |
|
|
1219
|
-
| Zero false positives in security testing | Context-aware reflection detection in [SecurityAssessor.ts](client/src/services/assessment/modules/SecurityAssessor.ts) | Validated |
|
|
1220
|
-
| Context-aware test data generation | Implementation in [TestDataGenerator.ts](client/src/services/assessment/TestDataGenerator.ts) | Measured |
|
|
1221
|
-
| MCP error code recognition | Implementation in [ResponseValidator.ts](client/src/services/assessment/ResponseValidator.ts) | Measured |
|
|
1222
|
-
| 80% reduction in false positives | Analysis in [FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md](docs/FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md#key-problems-addressed) | Estimated |
|
|
1223
|
-
| Business logic error detection | Implementation in [ResponseValidator.ts](client/src/services/assessment/ResponseValidator.ts) and [PHASE2_OPTIMIZATION_COMPLETED.md](docs/PHASE2_OPTIMIZATION_COMPLETED.md) | Measured |
|
|
1224
|
-
| Conditional boundary testing optimization | Implementation in [TestDataGenerator.ts](client/src/services/assessment/TestDataGenerator.ts) and [PHASE2_OPTIMIZATION_COMPLETED.md](docs/PHASE2_OPTIMIZATION_COMPLETED.md) | Measured |
|
|
1225
|
-
| Taskmanager case study results | Methodology validation in [ASSESSMENT_METHODOLOGY.md](docs/ASSESSMENT_METHODOLOGY.md) | Case Study |
|
|
325
|
+
---
|
|
1226
326
|
|
|
1227
|
-
|
|
327
|
+
## License
|
|
1228
328
|
|
|
1229
|
-
|
|
1230
|
-
- **Architecture & Value**: [ARCHITECTURE_AND_VALUE.md](docs/ARCHITECTURE_AND_VALUE.md) - Why behavioral testing matters, mcp-auditor integration
|
|
1231
|
-
- **Implementation Details**: [FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md](docs/FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md)
|
|
1232
|
-
- **Assessment Methodology**: [ASSESSMENT_METHODOLOGY.md](docs/ASSESSMENT_METHODOLOGY.md)
|
|
1233
|
-
- **Testing Comparison**: [TESTING_COMPARISON_EXAMPLE.md](docs/TESTING_COMPARISON_EXAMPLE.md)
|
|
1234
|
-
- **Error Handling Validation**: [ERROR_HANDLING_VALIDATION_SUMMARY.md](ERROR_HANDLING_VALIDATION_SUMMARY.md)
|
|
1235
|
-
- **Optimization Documentation**:
|
|
1236
|
-
- [COMPREHENSIVE_TESTING_ANALYSIS.md](docs/COMPREHENSIVE_TESTING_ANALYSIS.md) - Performance analysis and optimization opportunities
|
|
1237
|
-
- [COMPREHENSIVE_TESTING_OPTIMIZATION_PLAN.md](docs/COMPREHENSIVE_TESTING_OPTIMIZATION_PLAN.md) - 4-phase optimization roadmap
|
|
1238
|
-
- [PHASE1_OPTIMIZATION_COMPLETED.md](docs/PHASE1_OPTIMIZATION_COMPLETED.md) - Progressive complexity optimization (50% faster)
|
|
1239
|
-
- [PHASE2_OPTIMIZATION_COMPLETED.md](docs/PHASE2_OPTIMIZATION_COMPLETED.md) - Business logic error detection and boundary testing
|
|
329
|
+
This project is licensed under the MIT License—see the [LICENSE](LICENSE) file for details.
|
|
1240
330
|
|
|
1241
|
-
|
|
331
|
+
---
|
|
1242
332
|
|
|
1243
|
-
|
|
333
|
+
<a id="about-this-fork"></a>
|
|
1244
334
|
|
|
1245
|
-
|
|
1246
|
-
2. Running the test suites in `client/src/services/__tests__/`
|
|
1247
|
-
3. Reviewing the methodology documentation in `docs/`
|
|
1248
|
-
4. Testing against your own MCP servers using the assessment features
|
|
335
|
+
## Appendix: Fork History & Acknowledgments
|
|
1249
336
|
|
|
1250
|
-
|
|
337
|
+
This is an enhanced fork of [Anthropic's MCP Inspector](https://github.com/modelcontextprotocol/inspector) with significantly expanded assessment capabilities.
|
|
1251
338
|
|
|
1252
|
-
|
|
339
|
+
| Repository | URL |
|
|
340
|
+
| ------------- | -------------------------------------------------- |
|
|
341
|
+
| **Original** | https://github.com/modelcontextprotocol/inspector |
|
|
342
|
+
| **This Fork** | https://github.com/triepod-ai/inspector-assessment |
|
|
1253
343
|
|
|
1254
|
-
If you
|
|
344
|
+
**Note**: If you want the official Anthropic inspector without assessment features, use:
|
|
1255
345
|
|
|
1256
|
-
```
|
|
1257
|
-
|
|
1258
|
-
https://github.com/triepod-ai/inspector-assessment
|
|
1259
|
-
Enhancements: Advanced assessment methodology, progressive complexity testing,
|
|
1260
|
-
business logic error detection, and comprehensive security validation.
|
|
1261
|
-
Based on Anthropic's MCP Inspector: https://github.com/modelcontextprotocol/inspector
|
|
346
|
+
```bash
|
|
347
|
+
npx @modelcontextprotocol/inspector
|
|
1262
348
|
```
|
|
1263
349
|
|
|
1264
|
-
###
|
|
1265
|
-
|
|
1266
|
-
- **Project Status & Recent Changes**: [PROJECT_STATUS.md](PROJECT_STATUS.md)
|
|
1267
|
-
- **Comprehensive Assessment Methodology**: [docs/ASSESSMENT_METHODOLOGY.md](docs/ASSESSMENT_METHODOLOGY.md)
|
|
1268
|
-
- **Functionality Test Enhancements**: [docs/FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md](docs/FUNCTIONALITY_TEST_ENHANCEMENTS_IMPLEMENTED.md)
|
|
1269
|
-
- **Original MCP Inspector Documentation**: https://modelcontextprotocol.io/docs/tools/inspector
|
|
350
|
+
### What We Added
|
|
1270
351
|
|
|
1271
|
-
|
|
352
|
+
We built a comprehensive assessment framework on top of the original inspector, transforming it from a debugging tool into a full validation suite. Key additions:
|
|
1272
353
|
|
|
1273
|
-
|
|
354
|
+
- **16 Assessment Modules** covering functionality, security, compliance
|
|
355
|
+
- **Pure Behavior-Based Detection** analyzing responses, not tool names
|
|
356
|
+
- **Zero False Positives** through context-aware reflection detection
|
|
357
|
+
- **CLI-First Workflow** with three specialized commands
|
|
1274
358
|
|
|
1275
|
-
|
|
359
|
+
### Base Inspector Features
|
|
1276
360
|
|
|
1277
|
-
|
|
1278
|
-
- More sophisticated business logic detection
|
|
1279
|
-
- Performance profiling enhancements
|
|
1280
|
-
- Integration with CI/CD pipelines
|
|
1281
|
-
- Additional assessment visualizations
|
|
361
|
+
For documentation on the underlying inspector UI and operational features (Docker, authentication, configuration, transports), see:
|
|
1282
362
|
|
|
1283
|
-
|
|
363
|
+
- [Base Inspector Guide](docs/BASE_INSPECTOR_GUIDE.md)
|
|
364
|
+
- [Fork History](docs/FORK_HISTORY.md)
|
|
365
|
+
- [Upstream Sync Workflow](docs/UPSTREAM_SYNC_WORKFLOW.md)
|
|
1284
366
|
|
|
1285
367
|
### Acknowledgments
|
|
1286
368
|
|
|
1287
369
|
This project builds upon the excellent foundation provided by Anthropic's MCP Inspector team. We're grateful for their work on the original inspector and the MCP protocol specification.
|
|
1288
|
-
|
|
1289
|
-
## Links
|
|
1290
|
-
|
|
1291
|
-
- **npm Package**: https://www.npmjs.com/package/@bryan-thompson/inspector-assessment
|
|
1292
|
-
- **GitHub Repository**: https://github.com/triepod-ai/inspector-assessment
|
|
1293
|
-
- **Original MCP Inspector**: https://github.com/modelcontextprotocol/inspector
|
|
1294
|
-
- **Issues & Bug Reports**: https://github.com/triepod-ai/inspector-assessment/issues
|
|
1295
|
-
- **MCP Documentation**: https://modelcontextprotocol.io
|
|
1296
|
-
- **Publishing Guide**: [PUBLISHING_GUIDE.md](PUBLISHING_GUIDE.md)
|
|
1297
|
-
- **Changelog**: [CHANGELOG.md](CHANGELOG.md)
|
|
1298
|
-
|
|
1299
|
-
## License
|
|
1300
|
-
|
|
1301
|
-
This project is licensed under the MIT License—see the [LICENSE](LICENSE) file for details.
|