@slbdn/mcp-tester 1.4.1 → 1.5.3

package/docs/cicd.md

@@ -0,0 +1,289 @@
+# CI/CD Integration
+
+Using MCP Tester in continuous integration pipelines.
+
+## GitHub Actions
+
+Create `.github/workflows/test.yml`:
+
+```yaml
+name: Test MCP Server
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main, develop]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        node-version: [20, 22]
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v5
+
+      - name: Setup Node.js ${{ matrix.node-version }}
+        uses: actions/setup-node@v5
+        with:
+          node-version: ${{ matrix.node-version }}
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build TypeScript
+        run: npm run build
+
+      - name: Run tests
+        run: npm test
+
+  coverage:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v5
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v5
+        with:
+          node-version: '20'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build TypeScript
+        run: npm run build
+
+      - name: Run tests with coverage
+        run: npm run test:coverage
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v5
+        with:
+          files: ./coverage/lcov.info
+          flags: unittests
+          name: codecov-umbrella
+          fail_ci_if_error: false
+          verbose: true
+
+      - name: Generate coverage summary
+        id: coverage
+        run: |
+          SUMMARY=$(cat coverage/coverage-summary.json 2>/dev/null || echo '{}')
+          STATEMENTS=$(echo "$SUMMARY" | jq -r '.total.statements.pct // "N/A"')
+          BRANCHES=$(echo "$SUMMARY" | jq -r '.total.branches.pct // "N/A"')
+          FUNCTIONS=$(echo "$SUMMARY" | jq -r '.total.functions.pct // "N/A"')
+          LINES=$(echo "$SUMMARY" | jq -r '.total.lines.pct // "N/A"')
+
+          echo "statements=$STATEMENTS" >> $GITHUB_OUTPUT
+          echo "branches=$BRANCHES" >> $GITHUB_OUTPUT
+          echo "functions=$FUNCTIONS" >> $GITHUB_OUTPUT
+          echo "lines=$LINES" >> $GITHUB_OUTPUT
+
+          {
+            echo "## 📊 Test Coverage Report"
+            echo ""
+            echo "| Metric | Coverage |"
+            echo "|--------|----------|"
+            echo "| Statements | ${STATEMENTS}% |"
+            echo "| Branches | ${BRANCHES}% |"
+            echo "| Functions | ${FUNCTIONS}% |"
+            echo "| Lines | ${LINES}% |"
+            echo ""
+            echo "Thresholds: Statements ≥67% · Branches ≥61% · Functions ≥60% · Lines ≥67%"
+          } > coverage-comment.md
+
+      - name: Comment PR with coverage
+        if: github.event_name == 'pull_request'
+        uses: marocchino/sticky-pull-request-comment@v2
+        with:
+          header: coverage
+          path: coverage-comment.md
+
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v5
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v5
+        with:
+          node-version: '20'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Run linter
+        run: npm run lint
+
+      - name: Check formatting
+        run: npm run format:check
+
+  security:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v5
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v5
+        with:
+          node-version: '20'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Run security audit
+        run: npm audit --audit-level=moderate
+```
+
+### How It Works
+
+| Job | Purpose | Node Version |
+|-----|---------|---------------|
+| `test` | Run full test suite, upload HTML test report | Matrix: 20, 22 |
+| `coverage` | Generate coverage report, upload to Codecov, comment on PRs | 20 |
+| `lint` | ESLint + Prettier checks | 20 |
+| `security` | npm audit for vulnerabilities | 20 |
+
+### PR Coverage Comments
+
+Every pull request receives an automated coverage comment like:
+
+> ## 📊 Test Coverage Report
+>
+> | Metric | Coverage |
+> |--------|----------|
+> | Statements | 68% |
+> | Branches | 61% |
+> | Functions | 60% |
+> | Lines | 68% |
+>
+> Thresholds: Statements ≥67% · Branches ≥61% · Functions ≥60% · Lines ≥67%
+
+The comment updates on each push (using `sticky-pull-request-comment` with a `header` key).
+
+### HTML Test Reports
+
+Every CI run uploads an HTML test report as a downloadable artifact:
+
+- Generated by `jest-html-reporters` on every `npm test` run
+- Saved to `reports/test-report.html`
+- Uploaded per Node version (`test-report-node-20`, `test-report-node-22`)
+- Retained for 14 days
+- Includes: collapsible test tree, timing per test, failure details with stack traces
+
+To view locally:
+
+```bash
+npm test                        # generates reports/test-report.html
+open reports/test-report.html   # macOS
+```
+
+### Coverage Thresholds
+
+Jest enforces minimum coverage thresholds in `jest.config.js`:
+
+| File | Stmts | Branch | Func | Lines |
+|------|-------|--------|------|-------|
+| Global | 67% | 61% | 60% | 67% |
+| `assert.ts` | 96% | 87% | 100% | 96% |
+| `matchers.ts` | 68% | 28% | 42% | 64% |
+| `MCPClient.ts` | 68% | 54% | 75% | 68% |
+| `logger.ts` | 84% | 82% | 94% | 83% |
+| `masking.ts` | 100% | 88% | 100% | 100% |
+| `errors.ts` | 100% | 100% | 100% | 100% |
+| `env.ts` | 80% | 100% | 60% | 87% |
+| `mock-server.ts` | 96% | 64% | 94% | 96% |
+
+If coverage drops below these thresholds, the `test:coverage` command will exit with an error — failing the CI build.
+
+## CircleCI
+
+Create `.circleci/config.yml`:
+
+```yaml
+version: 2.1
+jobs:
+  test:
+    docker:
+      - image: cimg/node:20
+    steps:
+      - checkout
+      - run: npm ci
+      - run: npm run build
+      - run: npm test
+
+  coverage:
+    docker:
+      - image: cimg/node:20
+    steps:
+      - checkout
+      - run: npm ci
+      - run: npm run build
+      - run: npm run test:coverage
+      - store_artifacts:
+          path: coverage
+          destination: coverage
+
+workflows:
+  version: 2
+  test_and_coverage:
+    jobs:
+      - test
+      - coverage
+```
+
+## Jenkins
+
+Create `Jenkinsfile`:
+
+```groovy
+pipeline {
+  agent any
+  stages {
+    stage('Test') {
+      steps {
+        sh 'npm ci'
+        sh 'npm run build'
+        sh 'npm test'
+      }
+    }
+    stage('Coverage') {
+      steps {
+        sh 'npm run test:coverage'
+        publishHTML(target: [
+          reportDir: 'coverage/lcov-report',
+          reportFiles: 'index.html',
+          reportName: 'Coverage'
+        ])
+      }
+    }
+  }
+}
+```
+
+## Using the CLI in CI
+
+```bash
+# Quick smoke test
+npx @slbdn/mcp-tester test node ./server.js
+
+# List tools and verify
+npx @slbdn/mcp-tester list-tools node ./server.js --json
+
+# Call a tool and check result
+npx @slbdn/mcp-tester call-tool health-check node ./server.js --json
+```

package/docs/cli.md

@@ -0,0 +1,225 @@
+# CLI Tool
+
+MCP Tester includes a CLI for quick server testing without writing code.
+
+Install globally or use via npx:
+
+```bash
+npx @slbdn/mcp-tester test node ./server.js
+```
+
+## Transports
+
+The CLI supports three transport types:
+
+### Stdio (default)
+
+Pass the command and arguments as positional args:
+
+```bash
+mcp-tester test node ./server.js
+mcp-tester list-tools node ./server.js --json
+```
+
+### HTTP (Streamable HTTP)
+
+Pass the URL as the first argument (auto-detected), or use `--transport http --url`:
+
+```bash
+# Auto-detect: URL as first arg
+mcp-tester test http://localhost:3000/mcp
+mcp-tester list-tools http://localhost:3000/mcp --json
+
+# Explicit
+mcp-tester test --transport http --url http://localhost:3000/mcp
+
+# With headers
+mcp-tester test --transport http --url https://api.example.com/mcp --headers '{"Authorization":"Bearer token"}'
+```
+
+### SSE (legacy)
+
+```bash
+mcp-tester test --transport sse --url http://localhost:3000/sse
+```
+
+## Commands
+
+### `test` — Test Server Connection
+
+Connects to the server and lists all capabilities.
+
+```bash
+mcp-tester test node ./server.js
+```
+
+### `list-tools` (alias: `lt`)
+
+List all available tools.
+
+```bash
+mcp-tester list-tools node ./server.js
+mcp-tester lt node ./server.js --json
+```
+
+### `call-tool` (alias: `ct`)
+
+Call a tool with optional JSON parameters.
+
+```bash
+mcp-tester call-tool echo node ./server.js --params '{"message":"Hello"}'
+mcp-tester ct echo node ./server.js --params '{"message":"World"}' --json
+```
+
+### `read-resource` (alias: `rr`)
+
+Read a resource by URI.
+
+```bash
+mcp-tester read-resource text://example node ./server.js
+mcp-tester rr text://example node ./server.js --json
+```
+
+### `get-prompt` (alias: `gp`)
+
+Get a prompt with optional JSON arguments.
+
+```bash
+mcp-tester get-prompt greet node ./server.js --args '{"name":"Alice"}'
+mcp-tester gp greet node ./server.js --args '{"name":"Bob"}' --json
+```
+
+## Global Options
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--timeout <ms>` | `-t` | 30000 | Request timeout |
+| `--verbose` | `-v` | off | Enable debug output |
+| `--log-level <level>` | `-l` | info | Log level (debug/info/warn/error) |
+| `--json` | | off | Output as JSON (commands that support it) |
+
+### `generate` (alias: `gen`) — Generate Test Files
+
+Connect to a server, inspect all capabilities, and generate a ready-to-run test file.
+
+```bash
+# Generate Jest test file
+mcp-tester generate node ./server.js -o server.test.ts
+
+# Generate Vitest test file
+mcp-tester gen node ./server.js --framework vitest -o server.test.ts
+
+# Skip specific sections
+mcp-tester generate node ./server.js --no-resources --no-prompts -o tools-only.test.ts
+
+# Deterministic edge cases (offline)
+mcp-tester generate node ./server.js --edge-cases -o edge.test.ts
+
+# AI-suggested cases (OpenAI-compatible; Ollama needs no key on localhost)
+mcp-tester generate node ./server.js --ai-generate --ai-model llama3 \
+  --ai-base-url http://localhost:11434/v1 -o ai.test.ts
+
+# Dry-run derived cases against the live server
+mcp-tester generate node ./server.js --edge-cases --verify -o edge.test.ts
+
+# Print to stdout (no file)
+mcp-tester generate node ./server.js
+```
+
+**Options:**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `-o, --output <file>` | stdout | Output file path |
+| `--framework <fw>` | `jest` | Test framework (`jest` or `vitest`) |
+| `--description <desc>` | auto | Test suite description |
+| `--no-tools` | | Skip tool tests |
+| `--no-resources` | | Skip resource tests |
+| `--no-prompts` | | Skip prompt tests |
+| `--no-matchers` | | Skip custom matchers import |
+| `--edge-cases` | off | Add schema-derived boundary/invalid test cases (offline) |
+| `--ai-generate` | off | Edge cases + AI-suggested cases (env: `MCP_TESTER_AI_API_KEY`) |
+| `--ai-model <model>` | `gpt-4o-mini` | AI model name |
+| `--ai-base-url <url>` | `https://api.openai.com/v1` | OpenAI-compatible API base URL |
+| `--require-ai` | off | Fail instead of falling back when AI is unavailable |
+| `--verify` | off | Dry-run derived cases against the live server and annotate mismatches |
+
+See [AI-Assisted Test Generation](./ai-generation.md) for the full workflow.
+
+### `generate-types` (alias: `gen-types`) — Generate TypeScript Types
+
+Connect to a server, read tool schemas, and generate typed `.d.ts` declarations.
+
+```bash
+# Generate and save to file
+mcp-tester generate-types node ./server.js -o server.d.ts
+
+# Print to stdout
+mcp-tester generate-types node ./server.js
+
+# Skip resource/prompt types
+mcp-tester generate-types node ./server.js --no-resources --no-prompts -o types.d.ts
+```
+
+**Options:**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `-o, --output <file>` | stdout | Output file path |
+| `--module-name <name>` | `@slbdn/mcp-tester` | Module name for import hints |
+| `--no-resources` | | Skip resource URI types |
+| `--no-prompts` | | Skip prompt argument types |
+
+**Generated types:**
+
+| Type | Description |
+|------|-------------|
+| `{ToolName}Args` | Typed interface per tool (e.g., `EchoArgs`, `AddArgs`) |
+| `ToolName` | Union of all tool names (e.g., `'echo' \| 'add'`) |
+| `ToolArgsMap` | Lookup map: `ToolArgsMap['add']` → `AddArgs` |
+| `ToolCall` | Discriminated union: `{ name: 'add'; arguments: AddArgs }` |
+| `ResourceUri` | Union of all resource URIs |
+| `{PromptName}Args` | Typed interface per prompt |
+| `PromptName` | Union of all prompt names |
+| `PromptArgsMap` | Lookup map for prompt arguments |
+| `ServerCapabilities` | Overview interface with typed arrays |
+
+**Usage with generated types:**
+
+```typescript
+import type { ToolArgsMap, ToolName } from './server.d.ts';
+
+const result = await client.callTool({
+  name: 'add',
+  arguments: { a: 1, b: 2 } as ToolArgsMap['add'],
+});
+```
+
+## Global Options
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--timeout <ms>` | `-t` | 30000 | Request timeout |
+| `--verbose` | `-v` | off | Enable debug output |
+| `--log-level <level>` | `-l` | info | Log level (debug/info/warn/error) |
+| `--json` | | off | Output as JSON (commands that support it) |
+
+## Output Format
+
+Default output is human-readable with emojis. Use `--json` for machine-readable output:
+
+```bash
+mcp-tester lt node ./server.js --json
+```
+
+```json
+{
+  "tools": [
+    {
+      "name": "echo",
+      "description": "Echo back the input",
+      "inputSchema": { ... }
+    }
+  ]
+}
+```