@slbdn/mcp-tester 1.4.1 → 1.5.3

package/docs/advanced.md

@@ -0,0 +1,424 @@
+# Advanced Usage
+
+Advanced patterns and features of MCP Tester.
+
+## Concurrent Operations
+
+Run multiple tool calls in parallel:
+
+```typescript
+const results = await Promise.all([
+  client.callTool({ name: 'tool1', arguments: {} }),
+  client.callTool({ name: 'tool2', arguments: {} }),
+  client.callTool({ name: 'tool3', arguments: {} }),
+]);
+
+results.forEach((result, index) => {
+  console.log(`Tool ${index + 1}:`, result.content[0]?.text);
+});
+```
+
+## Timeout Handling
+
+Customize timeouts at two levels:
+
+**Global timeout** — applies to all requests:
+
+```typescript
+const client = new MCPClient({ timeout: 60000 });
+```
+
+**Per-call timeout** — overrides the global default:
+
+```typescript
+const result = await client.callTool({
+  name: 'slow-operation',
+  arguments: { duration: 5000 },
+  timeout: 10000,
+});
+```
+
+## Retry Logic
+
+Configure automatic retries with exponential backoff:
+
+```typescript
+const client = new MCPClient({
+  retries: 3,
+  retryDelay: 1000, // base delay in ms
+});
+```
+
+Per-call override:
+
+```typescript
+await client.callTool({
+  name: 'flaky-tool',
+  arguments: {},
+  retries: 5,
+});
+```
+
+Retry behavior:
+- Exponential backoff: `delay * backoffMultiplier^(attempt - 1)`
+- Default multiplier: 2
+- Max delay cap: 10,000ms
+- Retries only on thrown errors (not on successful responses with error content)
+
+## Environment Variables
+
+Pass environment variables to the MCP server process:
+
+```typescript
+await client.start({
+  command: 'node',
+  args: ['./server.js'],
+  env: {
+    NODE_ENV: 'test',
+    API_ENDPOINT: process.env.API_ENDPOINT || 'http://localhost:3000',
+    LOG_LEVEL: 'debug',
+  },
+});
+```
+
+`undefined` values are automatically filtered out, so conditional env vars work cleanly:
+
+```typescript
+env: {
+  NODE_ENV: 'production',
+  API_KEY: process.env.API_KEY, // won't be passed if undefined
+}
+```
+
+## Notification Handling
+
+Monitor server-initiated notifications:
+
+```typescript
+client.setNotificationHandlers({
+  onLoggingMessage: (level, data) => {
+    console.log(`[${level}] ${data}`);
+  },
+  onResourceListChanged: async () => {
+    console.log('Resources changed, refreshing...');
+    const resources = await client.listResources();
+    console.log(`Now have ${resources.length} resources`);
+  },
+});
+```
+
+## Elicitation Handling
+
+Respond to server requests for user input:
+
+```typescript
+await client.setElicitationHandler(async (request) => {
+  console.log('Elicitation request:', request.params.mode);
+
+  if (request.params.mode === 'form') {
+    return {
+      action: 'accept',
+      content: {
+        username: 'test-user',
+        email: 'test@example.com',
+      },
+    };
+  }
+
+  return { action: 'decline' };
+});
+```
+
+Handler return values:
+- `action: 'accept'` — provide `content` with the collected input
+- `action: 'decline'` — user declined the request
+- `action: 'cancel'` — user cancelled the operation
+
+## Logging
+
+Configure logging at construction or dynamically:
+
+```typescript
+const client = new MCPClient({
+  logLevel: 'debug', // 'debug' | 'info' | 'warn' | 'error' | 'none'
+});
+
+// Change at runtime
+client.setLogLevel('warn');
+```
+
+Enable protocol-level logging to see raw JSON messages:
+
+```typescript
+const client = new MCPClient({
+  enableProtocolLogging: true,
+  logLevel: 'debug',
+});
+```
+
+All log output goes to stderr so it doesn't interfere with stdout.
+
+## Mock Server for Unit Testing
+
+Use the in-memory mock server for tests that don't need a real server process:
+
+```typescript
+import { MockMCPServer } from './fixtures/mock-server.js';
+
+describe('My Feature', () => {
+  let mockServer: MockMCPServer;
+
+  beforeEach(() => {
+    mockServer = new MockMCPServer();
+  });
+
+  it('should handle tool call', async () => {
+    const result = await mockServer.handleToolCall('echo', { message: 'test' });
+    expect(result.content[0].text).toBe('Echo: test');
+  });
+});
+```
+
+### Built-in Tools
+
+| Tool | Description |
+|------|-------------|
+| `echo` | Echo back the input message |
+| `add` | Add two numbers |
+| `delay` | Delay for specified milliseconds |
+| `error_tool` | Always throws an error |
+| `counter` | Stateful counter: increment, get, reset |
+| `items` | Stateful item list: add, list, remove, clear |
+| `transform` | Transform text: upper, lower, reverse, length |
+
+### Built-in Resources: `text://example`, `config://settings`
+### Built-in Prompts: `greet`, `summarize`
+
+### Testing Retry Logic
+
+```typescript
+const server = new MockMCPServer({ failureRate: 0.5 });
+// 50% of tool calls will throw randomly
+// Great for testing retry logic
+```
+
+### Testing with Delays
+
+```typescript
+const server = new MockMCPServer({ defaultDelay: 100 });
+// All tool calls take ≥100ms — test timeout handling
+```
+
+### Input Validation Testing
+
+```typescript
+const server = new MockMCPServer({ validateSchemas: true });
+await expect(
+  server.handleToolCall('echo', {})
+).rejects.toThrow('Missing required field');
+```
+
+### Stateful Testing
+
+```typescript
+await server.handleToolCall('counter', { action: 'increment', by: 5 });
+await server.handleToolCall('counter', { action: 'increment', by: 3 });
+const result = await server.handleToolCall('counter', { action: 'get' });
+// result.content[0].text === 'Counter: 8'
+
+server.resetState(); // clears counters, items, call history
+```
+
+### Custom Handlers
+
+```typescript
+server.registerToolHandler('echo', (args) => [
+  { type: 'text', text: `Override: ${args.message}` },
+]);
+server.registerResourceHandler('text://example', (uri) => [
+  { uri, mimeType: 'text/plain', text: 'Custom content' },
+]);
+server.registerPromptHandler('greet', (args) => [
+  { role: 'assistant', content: { type: 'text', text: `Hi ${args.name}!` } },
+]);
+```
+
+### Call History Assertions
+
+```typescript
+await server.handleToolCall('echo', { message: 'test' });
+server.getCallCount('echo');    // 1
+server.getCallHistory()[0].args; // { message: 'test' }
+```
+
+See [Testing Guide](./testing.md#mock-server) for the full mock server API reference.
+
+## TypeScript Type Generation
+
+Generate typed `.d.ts` declarations from any MCP server's tool schemas:
+
+```bash
+npx mcp-tester generate-types node ./server.js -o server.d.ts
+```
+
+Or use the API:
+
+```typescript
+import { generateTypes } from '@slbdn/mcp-tester';
+import { writeFileSync } from 'fs';
+
+const types = await generateTypes({
+  command: 'node',
+  args: ['./server.js'],
+});
+
+writeFileSync('server.d.ts', types);
+```
+
+### Generated Output
+
+```typescript
+/** Echo back the input */
+export interface EchoArgs {
+  /** Message to echo */
+  message: string;
+}
+
+/** Add two numbers */
+export interface AddArgs {
+  a: number;
+  b: number;
+}
+
+/** Union of all tool names */
+export type ToolName = 'echo' | 'add' | 'delay' | 'error_tool';
+
+/** Look up the argument type for a tool by name */
+export interface ToolArgsMap {
+  'echo': EchoArgs;
+  'add': AddArgs;
+}
+
+/** Discriminated union for typed callTool() */
+export type ToolCall =
+  | { name: 'echo'; arguments: EchoArgs }
+  | { name: 'add'; arguments: AddArgs };
+```
+
+### Using Generated Types
+
+```typescript
+import type { ToolArgsMap, ToolName } from './server.d.ts';
+
+// Type-safe arguments
+const result = await client.callTool({
+  name: 'add',
+  arguments: { a: 1, b: 2 } as ToolArgsMap['add'],
+});
+```
+
+### Supported JSON Schema Features
+
+| Feature | TypeScript Output |
+|---------|------------------|
+| `{ type: 'string' }` | `string` |
+| `{ type: 'number' }` | `number` |
+| `{ type: 'boolean' }` | `boolean` |
+| `{ enum: ['a', 'b'] }` | `'a' \| 'b'` |
+| `{ const: 'fixed' }` | `'fixed'` |
+| `{ type: 'array', items: ... }` | `T[]` or `Array<T>` |
+| `{ type: 'object', properties: ... }` | `{ prop: Type }` |
+| `{ oneOf: [...] }` | Union type |
+| `{ anyOf: [...] }` | Union type |
+| `{ allOf: [...] }` | Intersection type |
+| `{ $ref: '#/definitions/X' }` | `X` |
+| Optional fields (not in `required`) | `field?: Type` |
+| `description` | JSDoc comments |
+
+## Server Health Checks
+
+Detect zombie processes and monitor server health during long-running test suites.
+
+### One-Time Check
+
+```typescript
+const health = await client.isHealthy();
+if (!health.healthy) {
+  console.error(`Server unhealthy: ${health.message}`);
+  console.error(`PID ${health.pid}, latency: ${health.latencyMs}ms`);
+}
+```
+
+### Periodic Monitoring
+
+```typescript
+client.startHealthMonitor({
+  interval: 3000,
+  onUnhealthy: (status) => {
+    console.error(`Server died: ${status.message}`);
+    // Optionally restart or fail the test
+  },
+  onRecovery: (status) => {
+    console.log(`Server recovered after crash`);
+  },
+});
+
+// Clean up
+client.stopHealthMonitor(); // explicit
+await client.stop();          // also stops the monitor
+```
+
+### Zombie Detection
+
+Uses `process.kill(pid, 0)` to check if the server process is still alive. Works on macOS, Linux, and Windows. Detects crashed, killed, or orphaned processes that `isConnected()` cannot detect (the client object still exists but the process is gone).
+
+### HealthStatus Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `healthy` | `boolean` | Is the server responsive |
+| `checkedAt` | `number` | Unix timestamp of the check |
+| `latencyMs` | `number` | Round-trip time, or -1 if failed |
+| `pid` | `number \| null` | Server process ID |
+| `message` | `string` | Human-readable status |
+
+## Performance Benchmarks
+
+Run the benchmark suite to measure MCP Tester's performance characteristics:
+
+```bash
+npm run benchmark
+```
+
+This runs 19 benchmarks against the mock server (50 runs each, 3 warmup), measuring:
+
+| Category | Benchmarks |
+|----------|------------|
+| **Connection** | Reconnect, disconnect |
+| **Tool calls** | Echo, add, delay (10ms simulated) |
+| **Sequential** | 5 and 10 tools in a row |
+| **Parallel** | 5, 10, and 20 tools concurrently |
+| **Payloads** | 10KB, 100KB, 1MB echo payloads |
+| **Metadata** | List tools, resources, prompts; get prompt with args |
+| **Lifecycle** | Full start → list → call → stop patterns |
+
+### Reference Numbers (Node.js 24, Apple Silicon)
+
+| Operation | Mean Latency | Throughput |
+|----------|-------------|------------|
+| Single tool call (echo) | **0.19 ms** | ~5,130 ops/s |
+| Add tool (numbers) | **0.19 ms** | ~5,360 ops/s |
+| 10 parallel tool calls | **0.55 ms total** | ~18,200 ops/s |
+| 20 parallel tool calls | **0.86 ms total** | ~23,300 ops/s |
+| List tools / resources / prompts | **0.15 ms** | ~6,700 ops/s |
+| 1MB payload echo | **7.79 ms** | ~1,080 Mbps |
+| Server reconnect | **165 ms** | ~6 ops/s |
+| Full lifecycle (start+call+stop) | **167 ms** | ~6 ops/s |
+
+### Key Takeaways
+
+- **Parallelization wins** — 10 parallel calls take the same time as 1 (~0.5ms total), giving ~18x throughput vs sequential
+- **Connection overhead dominates single-call cost** — once connected, tool calls are sub-millisecond
+- **Payload size has minimal impact** — going from 10KB to 1MB adds only ~7ms of overhead
+- **Metadata listings are cheap** — listing tools, resources, or prompts costs ~0.15ms
+- **The stdio transport is fast** — most latency comes from process spawn, not the protocol itself

package/docs/ai-generation.md

@@ -0,0 +1,211 @@
+# AI-Assisted Test Generation
+
+`mcp-tester` can derive test cases for your MCP server's tools in three ways:
+
+| Mode | Flag | What it does | Needs network? |
+|------|------|--------------|:---:|
+| `static` | *(default)* | One happy-path call per tool | No |
+| `edge` | `--edge-cases` | Deterministic boundary/invalid cases from JSON Schema rules | No |
+| `ai` | `--ai-generate` | Edge cases **plus** AI-suggested cases, validated against the real schemas | Yes (or a local model) |
+
+**AI never writes test code.** The model only proposes *structured test-case
+data* (tool, arguments, expectation, rationale). A deterministic renderer emits
+the final TypeScript, and every AI-proposed case is validated against the
+tool's real input schema before it reaches your test file — hallucinated
+arguments are dropped, not compiled.
+
+## Quick start
+
+### Deterministic edge cases (offline)
+
+```bash
+npx mcp-tester generate node ./server.js --edge-cases -o tests/generated.test.ts
+```
+
+The rules engine inspects each tool's `inputSchema` and emits cases per rule:
+
+| Rule | Case | Expectation |
+|------|------|-------------|
+| `rule:required` | omit each required property | error |
+| `rule:enum` | value outside the enum | error |
+| `rule:min-length` | string one shorter than `minLength` | error |
+| `rule:max-length` | string one longer than `maxLength` | observe |
+| `rule:bounds` | number one below `minimum` / above `maximum` | error |
+| `rule:type` | wrong JSON type per property | error |
+| `rule:min-items` | array shorter than `minItems` | error |
+| `rule:additional-properties` | unknown key when `additionalProperties: false` | error |
+
+Expectations:
+
+- **`error`** — the case asserts the call fails (`expect(isToolError(result)).toBe(true)`)
+- **`observe`** — the case calls the tool and logs what happened (no hard
+  assertion; for behavior that is legitimately server-defined)
+- **`success`** — the case asserts the call succeeds
+
+### AI-suggested cases
+
+```bash
+export MCP_TESTER_AI_API_KEY=sk-...
+npx mcp-tester generate node ./server.js --ai-generate -o tests/generated.test.ts
+```
+
+`--ai-generate` includes everything `--edge-cases` produces, then asks the
+model for additional interesting cases (unusual-but-valid inputs, realistic
+payloads, combinations the rules can't reason about). Each suggestion is
+checked against the tool's schema:
+
+```text
+mcp-tester: AI suggested cases: 5, kept after schema validation: 4 (dropped 1)
+```
+
+Works with any OpenAI-compatible endpoint:
+
+| Provider | Base URL | Key needed |
+|----------|----------|:---:|
+| OpenAI | `https://api.openai.com/v1` (default) | yes |
+| Ollama | `http://localhost:11434/v1` | no |
+| LM Studio | `http://localhost:1234/v1` | no |
+| vLLM | `http://host:8000/v1` | optional |
+
+```bash
+# Ollama — no API key required for localhost
+npx mcp-tester generate node ./server.js \
+  --ai-generate --ai-base-url http://localhost:11434/v1 --ai-model llama3
+```
+
+### Flags
+
+| Flag | Meaning |
+|------|---------|
+| `--edge-cases` | Add deterministic schema-derived cases (offline) |
+| `--ai-generate` | Edge cases + AI suggestions (implies `--edge-cases`) |
+| `--ai-model <model>` | Model name (default `gpt-4o-mini` or `MCP_TESTER_AI_MODEL`) |
+| `--ai-base-url <url>` | OpenAI-compatible base URL (default `MCP_TESTER_AI_BASE_URL`) |
+| `--require-ai` | Fail instead of falling back when AI is unavailable |
+| `--verify` | Dry-run every derived case against the live server first |
+
+Passing both `--edge-cases` and `--ai-generate` is an error — `--ai-generate`
+already includes the edge cases.
+
+## Verifying derived cases against the real server
+
+Derived expectations are *predictions*. `--verify` dry-runs every case against
+the live server before writing the file:
+
+```bash
+npx mcp-tester generate node ./server.js --edge-cases --verify -o tests/generated.test.ts
+```
+
+- Cases whose prediction **holds** stay as normal assertions.
+- Cases where the server **disagrees** (e.g. the rules predicted an error but
+  the server returned success) are emitted as `it.skip` blocks with a comment:
+
+```typescript
+// rule:required — "message" is required; the server should reject its absence
+// verify: expected error but server returned success — adjust manually
+it.skip('rejects call without required "message"', async () => { ... });
+```
+
+That skip is a signal, not a failure: either the server is too permissive (fix
+the server — many MCP servers forget to validate `missing required property`)
+or the expectation was wrong (edit the test). Either way you've learned
+something concrete about your server.
+
+## Environment variables
+
+| Variable | Purpose | Default |
+|----------|---------|---------|
+| `MCP_TESTER_AI_API_KEY` | API key for hosted endpoints | *(none)* |
+| `MCP_TESTER_AI_BASE_URL` | OpenAI-compatible endpoint | `https://api.openai.com/v1` |
+| `MCP_TESTER_AI_MODEL` | Model name | `gpt-4o-mini` |
+
+A base URL pointing at `localhost`/`127.0.0.1`/`[::1]` is considered
+configured **without** an API key (local model servers don't need auth).
+
+Without configuration, `--ai-generate` prints a warning and falls back to the
+deterministic edge cases. With `--require-ai`, it fails instead.
+
+## Library API
+
+```typescript
+import {
+  generateTests,
+  generateTestsFromClient,
+  OpenAICompatProvider,
+  suggestEdgeCasesForTools,
+} from '@slbdn/mcp-tester';
+
+// Deterministic cases only
+await generateTests({ command: 'node', args: ['./server.js'], mode: 'edge' });
+
+// AI-assisted with an explicit provider (no env vars needed)
+await generateTests({
+  command: 'node',
+  args: ['./server.js'],
+  mode: 'ai',
+  provider: new OpenAICompatProvider({
+    baseUrl: 'http://localhost:11434/v1',
+    model: 'llama3',
+  }),
+});
+
+// Rules only, no file emission — inspect the cases yourself
+const cases = suggestEdgeCasesForTools(tools);
+// → [{ tool, title, args, expectation, rationale, source }, ...]
+```
+
+A `GeneratedCase` is plain data — safe to log, diff, and version-control:
+
+```typescript
+interface GeneratedCase {
+  tool: string;                                // tool name
+  title: string;                               // human-readable test title
+  args: Record<string, unknown>;               // arguments to send
+  expectation: 'success' | 'error' | 'observe';
+  rationale: string;                           // why this case exists
+  source: `rule:${string}` | `ai:${string}`;   // provenance, e.g. rule:required
+}
+```
+
+Custom providers implement one method:
+
+```typescript
+interface AIProvider {
+  name: string;
+  suggestCases(analysis: ServerAnalysis): Promise<GeneratedCase[]>;
+}
+```
+
+`OpenAICompatProvider` accepts `fetchImpl` for testing and custom HTTP
+behavior, caches responses in `.mcp-tester-cache/` (set `cacheDir: null` to
+disable), truncates long tool descriptions, and batches at most 40 tools per
+request.
+
+### Provider options reference
+
+| Option | Default | Effect |
+|---|---|---|
+| `apiKey` | *(unset)* | Bearer token. Optional for local gateways (Ollama, LM Studio). |
+| `baseUrl` | `https://api.openai.com/v1` | Chat-completions endpoint. |
+| `model` | `gpt-4o-mini` | Model name. |
+| `timeout` | `60000` | Per-request timeout in ms. |
+| `maxTokens` | `4096` | Cap on completion tokens. |
+| `cacheDir` | `.mcp-tester-cache` | Where to write cache files. `null` disables caching. |
+| `cacheMaxAgeMs` | *(unset)* | When set, cache files older than this are ignored and re-fetched. Useful when iterating on a model. |
+| `strictJson` | `true` | Sends `response_format: { type: 'json_object' }`. Set to `false` for gateways that reject it (Ollama, older vLLM). The provider auto-falls-back on HTTP 400 mentioning `response_format` regardless. |
+| `fetchImpl` | `globalThis.fetch` | Injectable fetch for tests and hermetic CI. |
+
+Cache key includes the model name, the `serverLabel`, and the tool set
+(sorted by name) — so changing any of them invalidates the entry.
+The cache file is JSON: `{ "model", "serverLabel", "cases": [...] }`.
+
+## Safety and cost notes
+
+- **No code from the model.** AI output is parsed as JSON case data and
+  validated; it can never become executable test logic.
+- **Prompt-injection resistant.** Tool descriptions are untrusted input; at
+  worst they influence *which cases are proposed* — still schema-validated.
+- **Caching.** Identical analyses hit the on-disk cache, so re-running
+  generation is free and deterministic.
+- **Deterministic fallback.** No key, no quota, no network — you still get the
+  full rules engine.