@slbdn/mcp-tester 1.4.3 → 1.5.3

package/CHANGELOG.md

@@ -7,6 +7,60 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.5.3] - 2026-10-08
+
+### Fixed
+- **`npm run test:coverage` failed its branch-coverage gate for `src/generate-cases.ts`** (`81.87% < 85%`) even though every test passed. Added tests covering the previously untested `additionalProperties`-as-schema constraint checks (`minLength`/`maxLength`/`maximum`/`exclusiveMinimum`/`exclusiveMaximum`/`minItems`/`maxItems`/`uniqueItems`), constraint-only violating-value inference, union/`null`/untyped-property handling, and rule-engine edges (numeric enums, unmatched `required` keys, required object properties, `maxItems` without an `items` schema). Branch coverage for the file is now 97.65%; overall branch coverage is 87.76%.
+
+## [1.5.2] - 2026-10-08
+
+### Fixed
+- **`npm ci` failed on Node 20 / npm 10, blocking the 1.5.1 publish.** The 1.5.1 lockfile was left inconsistent by `npm audit fix`: it dropped the top-level `@emnapi/core` and `@emnapi/runtime` entries that satisfy `@napi-rs/wasm-runtime`'s peer dependencies. npm 11 (which wrote the lock) tolerated this, but npm 10's stricter `npm ci` sync check rejected it with `EUSAGE` (`Missing: @emnapi/core@1.11.3 from lock file`). The entries are restored; `npm ci` and `npm audit --audit-level=high` both pass on npm 10.
+  - 1.5.1 was never published to npm; this release supersedes it and carries the same security updates described below.
+
+### Changed
+- CI and the release verification job now run on Node.js 20 and 22 instead of 20 and 21. Node 21 is end-of-life and is outside the `engines` range of Jest 30 and several dev dependencies (`glob@13`, `minimatch@10`, `lru-cache@11`, `eslint-visitor-keys@5`, …), which produced `EBADENGINE` warnings during install. The starter-template CI matrices were updated to match.
+
+## [1.5.1] - 2026-10-08
+
+### Security
+- Cleared the `npm audit` findings that blocked the 1.5.0 publish (the `release.yml` job's hard `npm audit --audit-level=high` gate). A semver-safe `npm audit fix` bumped:
+  - `@modelcontextprotocol/sdk` 1.29.0 → 1.32.1 (high — the OAuth client could send credentials to an authorization server chosen by the MCP server, [GHSA-6qxp-vccf-f47h](https://github.com/advisories/GHSA-6qxp-vccf-f47h)).
+  - `proxy-addr` 2.0.7 → 2.0.8 (critical — IP spoofing via an IPv4-mapped IPv6 trust subnet, [GHSA-jqcg-44mw-7w3h](https://github.com/advisories/GHSA-jqcg-44mw-7w3h)), transitive via `express` through the SDK's server integration.
+  - `ts-jest` → 29.4.14 and `babel-plugin-istanbul` → 8.0.2 transitively.
+  - The remaining advisories are dev-only moderate findings (`sprintf-js` via `ts-jest`) and sit below the release gate's `--audit-level=high` threshold; clearing them requires a breaking `ts-jest` downgrade (`npm audit fix --force`), so they are intentionally left in place.
+
+## [1.5.0] - 2026-10-08
+
+### Added
+- **AI-assisted & schema-driven test-case generation** (TODO #39) — `generate` now goes beyond happy-path calls:
+  - `--edge-cases` — deterministic boundary/invalid cases from each tool's JSON Schema (offline, no AI): missing required args, invalid enums, wrong types, `minLength`/`maxLength`, `minimum`/`maximum` bounds, `minItems`, `additionalProperties: false`. Expectations are `error`, `observe`, or `success`, and every case is tagged with a provenance comment (`// rule:required — ...`).
+  - `--ai-generate` — augments the edge cases with LLM suggestions via any OpenAI-compatible endpoint (OpenAI, Ollama, LM Studio, vLLM). The model only proposes **structured case data** (`GeneratedCase`); every suggestion is validated against the real tool schemas and rendered deterministically — hallucinated arguments are dropped with a summary line, and the model never emits code — model-supplied rationales are sanitized before being embedded in a comment, so a crafted rationale cannot break out and become executable test code. Configure via `MCP_TESTER_AI_API_KEY` / `MCP_TESTER_AI_BASE_URL` / `MCP_TESTER_AI_MODEL` or the new `--ai-model` / `--ai-base-url` / `--require-ai` flags. Responses are cached in `.mcp-tester-cache/`.
+  - `--verify` — dry-runs every derived case against the live server before writing the file; predictions that don't hold become `it.skip` blocks with an explanatory comment instead of broken tests.
+  - New modules: `generate-cases.ts` (rules engine, `GeneratedCase`, `suggestEdgeCases*`, `validateArgsAgainstSchema`, `mergeAndValidateCases`) and `ai/provider.ts` (`OpenAICompatProvider` with injectable `fetchImpl`, `createProviderFromEnv`). All exported from the package barrel.
+  - Docs: [`docs/ai-generation.md`](docs/ai-generation.md), README section, CLI reference.
+- **Starter templates** under `templates/` with a new `mcp-tester create <template> <dest>` CLI command:
+  - `minimal-jest` — one test file, one mock server, zero ceremony.
+  - `standard-jest` — per-capability test files, HTML reporter, GitHub Actions matrix.
+  - `full-stack` — real TS server, code generation scripts (`npm run gen:tests` / `gen:types`), typed tool calls, parallel stress tests.
+- `create list` shows available templates. `__NAME__` placeholders in `package.json` are substituted with the destination directory name. `--no-install` skips `npm install`; `--git` initializes a repo.
+- Documentation: [`docs/starter-templates.md`](docs/starter-templates.md) and a new README section.
+
+### Fixed
+- **Jest matcher types were broken for consumers since the Jest 30 upgrade (1.4.3)** — `expect(tools).toHaveTool(...)` produced TS2339 in consumer projects (the global `jest.Matchers` augmentation no longer reaches Jest 30's `expect()` return type). The matchers are now additionally declared against the `expect` package's `Matchers` interface via `MCPMatchers`, so autocomplete/type-checking works in both ts-jest and plain `tsc` setups. Because TypeScript does not apply module augmentations shipped inside `node_modules`, the package now ships a copy-ready `jest.d.ts` — copy it into your project to enable the types (see README).
+- Templates now typecheck cleanly under `strict` `tsc --noEmit` (they were only ever compiled by ts-jest's non-strict inline config): `content[0].text`-style union accesses replaced with typed `contentText()` / `promptText()` / `resourceText()` helpers, and full-stack's server handler no longer assumes `arguments` is defined.
+- `generateTests` (stdio variant) now respects the `includeTools` / `includeResources` / `includePrompts` flags when fetching from the server, matching the behaviour of `generateTestsFromClient`. Previously it always called `listResources` and `listPrompts` even when the output flags said to skip them — causing generation to fail against servers that don't declare those capabilities.
+- `examples/mock-server.js`: the `delay` tool now rejects invalid `ms` values (`NaN`, negative, > 60000) instead of passing them to `setTimeout` — caught by the new `--verify` mode.
+- The published package now ships the top-level `docs/*.md` guides, so the README's documentation links resolve on npmjs.com (they previously 404'd — the generated `docs/api/` HTML is still excluded).
+- The generated `it(...)` titles and call arguments now escape the exact quote delimiter they are interpolated into, so tool/resource/prompt names containing apostrophes, double quotes, or backslashes produce valid test files. Observe-oracle cases emit a proper single-quoted tool literal instead of a literal `${c.tool}` expression (which previously caused a `ReferenceError`).
+- AI mode no longer drops the deterministic one-past-`maxLength` `observe` case during schema validation, so `--edge-cases` and `--ai-generate` produce the same rule cases. The merge diagnostic now counts only AI rejections.
+- The `additionalProperties`-as-schema rule now emits a value that actually violates the declared type (it previously injected a number that satisfied `{ type: 'number' }`), and the `pattern` rule skips patterns that match every probe value (e.g. `.*`) instead of asserting a false `error`.
+- `parseJsonLoose` falls back past an unparseable code fence to a valid JSON payload elsewhere in the response, and its object scanner skips past a well-formed object instead of rescanning every nested `{`.
+- `validateValueAgainstSchema` now checks `enum`/`pattern`/numeric/array constraints on `additionalProperties` values, not just type and string length.
+
+### Security
+- **Generated test code cannot be turned into an injection vector** — model-supplied case rationales are collapsed onto a single line comment before being emitted, so a crafted multi-line rationale can no longer close the comment and inject executable statements into the generated file. The copy-ready `jest.d.ts` shipped to consumers also no longer contains a `*/` inside its doc comment (which made the file a syntax error).
+
 ## [1.4.3] - 2026-10-05
 
 ### Security

package/README.md

@@ -248,6 +248,34 @@ The generator connects to the server, inspects all tools/resources/prompts, and
 - Prompt tests with sample arguments
 - Proper `beforeEach`/`afterEach` cleanup
 
+#### Edge Cases & AI-Assisted Generation
+
+Go beyond happy-path calls with schema-derived boundary tests — optionally
+augmented by an LLM (OpenAI, Ollama, LM Studio, or any OpenAI-compatible
+endpoint). The AI only proposes **structured case data**; every case is
+validated against the real schemas and rendered deterministically — the model
+never writes code:
+
+```bash
+# Deterministic edge cases (offline): missing required args, invalid enums,
+# wrong types, boundary values, minItems, additionalProperties
+npx @slbdn/mcp-tester generate node ./server.js --edge-cases -o edge.test.ts
+
+# Add AI-suggested cases (needs MCP_TESTER_AI_API_KEY, or a local base URL)
+MCP_TESTER_AI_API_KEY=sk-... \
+npx @slbdn/mcp-tester generate node ./server.js --ai-generate -o ai.test.ts
+
+# Ollama — no API key needed for localhost
+npx @slbdn/mcp-tester generate node ./server.js \
+  --ai-generate --ai-base-url http://localhost:11434/v1 --ai-model llama3
+
+# Dry-run derived cases against the live server: predictions that fail
+# become it.skip blocks with an explanation instead of broken tests
+npx @slbdn/mcp-tester generate node ./server.js --edge-cases --verify -o edge.test.ts
+```
+
+📖 Full guide: [AI-Assisted Test Generation](./docs/ai-generation.md)
+
 ### Generate TypeScript Types
 
 Connect to any MCP server and generate typed `.d.ts` declarations from tool schemas:
@@ -285,6 +313,18 @@ const result = await client.callTool({
 
 Supports: primitives, enums, nested objects, arrays, `oneOf`/`anyOf`/`allOf`, `$ref`, `const`.
 
+## Starter Templates
+
+Three ready-to-use scaffolds for new projects. Pick the size that matches your need:
+
+```bash
+npx @slbdn/mcp-tester create minimal-jest my-server-tests   # Tiny: one test file
+npx @slbdn/mcp-tester create standard-jest my-server-tests  # Real-world: CI + HTML report
+npx @slbdn/mcp-tester create full-stack my-server-tests     # Library: code generation + typed tools
+```
+
+Each scaffolds a runnable project with a mock server, then runs `npm install`. Use `--no-install` to skip, `--git` to init a repo. See [docs/starter-templates.md](docs/starter-templates.md) for details.
+
 ## Speed Up Your Tests with Parallel Execution
 
 MCP Tester is built for concurrency. Fire off multiple tool calls at once — no waiting in line:
@@ -370,6 +410,10 @@ import {
 
 beforeAll(() => setupJestMatchers());
 
+// For type-safe matcher autocompletion, copy the augmentation into your
+// project. See `node_modules/@slbdn/mcp-tester/jest.d.ts` for the source
+// and full instructions.
+
 // Collection matchers
 expect(tools).toHaveTool('echo');
 expect(resources).toHaveResource('config://settings');
@@ -476,6 +520,7 @@ await runTest("Echo returns input", async () => {
 | [Examples](./docs/examples.md) | Practical code examples |
 | [Advanced Usage](./docs/advanced.md) | Timeouts, retries, concurrency, notifications, logging |
 | [CLI Reference](./docs/cli.md) | CLI commands, options, and output formats |
+| [AI-Assisted Test Generation](./docs/ai-generation.md) | Edge cases, LLM integration, `--verify` |
 | [CI/CD Integration](./docs/cicd.md) | GitHub Actions, CircleCI, Jenkins |
 | [Troubleshooting](./docs/troubleshooting.md) | Common issues and solutions |
 | [Releases](./docs/releases.md) | Release process and commands |
@@ -510,6 +555,10 @@ mcp-tester/
 │   ├── mock-server.js           # Basic standalone MCP server
 │   ├── simple-server.ts         # Minimal TypeScript MCP server
 │   └── stateful-server.ts        # Stateful MCP server (counter + todos)
+├── templates/                    # Starter templates (mcp-tester create <template>)
+│   ├── minimal-jest/             # One test file, one mock server
+│   ├── standard-jest/            # Per-capability tests, HTML report, CI matrix
+│   └── full-stack/               # TS server + code generation + typed tools
 └── docs/                        # Documentation
 ```
 

package/dist/ai/provider.d.ts

@@ -0,0 +1,145 @@
+/**
+ * AI test-case provider — pluggable, optional augmentation of the
+ * deterministic edge-case engine.
+ *
+ * The provider speaks the OpenAI-compatible chat-completions protocol
+ * (`POST {baseUrl}/chat/completions`), which covers OpenAI, Ollama, LM
+ * Studio, vLLM, Groq, Together, and most local gateways. Implemented with
+ * plain `fetch` — no SDK dependency.
+ *
+ * Security model: the LLM returns *data* (JSON test cases), never code.
+ * Every case is validated against the tool's real schema by
+ * `mergeAndValidateCases` before it can reach a generated file, so prompt
+ * injection via server-provided tool descriptions cannot inject code.
+ *
+ * @module ai/provider
+ */
+import type { Tool } from '@modelcontextprotocol/sdk/types.js';
+import { type GeneratedCase } from '../generate-cases.js';
+/** Server snapshot handed to a provider. Descriptions are length-capped. */
+export interface ServerAnalysis {
+    /** Human-readable server label for the prompt. */
+    serverLabel?: string;
+    /** Tools to reason about. */
+    tools: Array<{
+        name: string;
+        description?: string;
+        inputSchema: Record<string, unknown>;
+    }>;
+}
+/** Anything that can propose test cases for a server analysis. */
+export interface AIProvider {
+    /** Provider/model identity, used for case provenance (`ai:<name>`). */
+    readonly name: string;
+    /** Propose cases; implementers should tolerate partial failures. */
+    suggestCases(analysis: ServerAnalysis): Promise<GeneratedCase[]>;
+}
+/** Options for {@link OpenAICompatProvider}. */
+export interface OpenAICompatOptions {
+    /** API base URL. @defaultValue 'https://api.openai.com/v1' */
+    baseUrl?: string;
+    /** Chat model name. @defaultValue 'gpt-4o-mini' */
+    model?: string;
+    /** Bearer token; omit for local servers that don't need one. */
+    apiKey?: string;
+    /** Request timeout in ms. @defaultValue 60000 */
+    timeout?: number;
+    /** Cap on completion tokens per request. @defaultValue 4096 */
+    maxTokens?: number;
+    /** Cache directory; `null` disables caching. @defaultValue '.mcp-tester-cache' */
+    cacheDir?: string | null;
+    /**
+     * Injectable fetch for tests. @defaultValue globalThis.fetch
+     */
+    fetchImpl?: typeof fetch;
+    /**
+     * Send OpenAI's `response_format: { type: 'json_object' }` flag. Supported
+     * by OpenAI, Together, Groq, and most modern gateways; some local servers
+     * (Ollama, older vLLM) reject it. Set to `false` if your gateway returns
+     * HTTP 400 mentioning `response_format`. When `true` (default), a 400 that
+     * explicitly blames `response_format` triggers one automatic retry without
+     * the flag.
+     * @defaultValue true
+     */
+    strictJson?: boolean;
+    /**
+     * Maximum age in ms for cached AI responses. When set, cache files older
+     * than this are ignored and re-fetched. Unset (default) means cache never
+     * expires; entries are only invalidated when the model or tool set changes.
+     * @defaultValue undefined
+     */
+    cacheMaxAgeMs?: number;
+}
+export declare const DEFAULT_AI_BASE_URL = "https://api.openai.com/v1";
+export declare const DEFAULT_AI_MODEL = "gpt-4o-mini";
+/**
+ * Extract a JSON object from model content, tolerating markdown fences and
+ * prose around the payload. Uses brace-aware scanning so a `}` inside a
+ * string value or in a nested object doesn't truncate the payload.
+ *
+ * @returns The parsed object, or `null` when no parseable JSON object is
+ *   found in the input.
+ */
+export declare function parseJsonLoose(content: string): {
+    cases?: unknown;
+} | null;
+/**
+ * OpenAI-compatible chat-completions provider.
+ *
+ * Works with OpenAI, Ollama (`http://localhost:11434/v1`), LM Studio,
+ * vLLM, Groq, and any compatible gateway.
+ */
+export declare class OpenAICompatProvider implements AIProvider {
+    readonly name: string;
+    private readonly baseUrl;
+    private readonly model;
+    private readonly apiKey?;
+    private readonly timeout;
+    private readonly maxTokens;
+    private readonly cacheDir;
+    private readonly cacheMaxAgeMs;
+    private readonly fetchImpl;
+    private readonly strictJson;
+    /** Tracks whether this provider has already observed a `response_format` rejection. */
+    private responseFormatDisabled;
+    constructor(options?: OpenAICompatOptions);
+    suggestCases(analysis: ServerAnalysis): Promise<GeneratedCase[]>;
+    /** Single chat-completions round trip. Throws with response context. */
+    private chatCompletion;
+    /**
+     * One HTTP request to the chat-completions endpoint. Categorises the
+     * response so the caller can decide whether to retry without
+     * `response_format` or just throw.
+     */
+    private rawChatCompletion;
+    private cacheKeyFor;
+    private readCache;
+    private writeCache;
+}
+/**
+ * Build a provider from `MCP_TESTER_AI_*` environment variables.
+ *
+ * - `MCP_TESTER_AI_API_KEY` — required for hosted APIs; optional for local.
+ * - `MCP_TESTER_AI_BASE_URL` — @defaultValue 'https://api.openai.com/v1'
+ * - `MCP_TESTER_AI_MODEL` — @defaultValue 'gpt-4o-mini'
+ *
+ * @returns A provider, or `null` when no API key is configured. A base URL
+ *   pointing at localhost counts as configured even without a key (Ollama
+ *   and friends don't need auth).
+ */
+export declare function createProviderFromEnv(env?: NodeJS.ProcessEnv): OpenAICompatProvider | null;
+/**
+ * Convenience: ask the AI provider to suggest test cases, then validate them
+ * against the tools' real input schemas. Does NOT merge with deterministic
+ * rules — callers wanting the union should call {@link suggestEdgeCases} (or
+ * `suggestEdgeCasesForTools`) themselves and concatenate.
+ *
+ * @returns Validated cases, rejected entries with reasons, and the provider
+ *   name (for diagnostic logging).
+ */
+export declare function suggestCasesWithAI(analysis: ServerAnalysis, provider: AIProvider, schemas: Map<string, Tool>): Promise<{
+    cases: GeneratedCase[];
+    rejected: string[];
+    providerName: string;
+}>;
+//# sourceMappingURL=provider.d.ts.map

package/dist/ai/provider.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"provider.d.ts","sourceRoot":"","sources":["../../src/ai/provider.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;GAeG;AAKH,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,oCAAoC,CAAC;AAC/D,OAAO,EACL,KAAK,aAAa,EAInB,MAAM,sBAAsB,CAAC;AAE9B,4EAA4E;AAC5E,MAAM,WAAW,cAAc;IAC7B,kDAAkD;IAClD,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,6BAA6B;IAC7B,KAAK,EAAE,KAAK,CAAC;QACX,IAAI,EAAE,MAAM,CAAC;QACb,WAAW,CAAC,EAAE,MAAM,CAAC;QACrB,WAAW,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;KACtC,CAAC,CAAC;CACJ;AAED,kEAAkE;AAClE,MAAM,WAAW,UAAU;IACzB,uEAAuE;IACvE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,oEAAoE;IACpE,YAAY,CAAC,QAAQ,EAAE,cAAc,GAAG,OAAO,CAAC,aAAa,EAAE,CAAC,CAAC;CAClE;AAED,gDAAgD;AAChD,MAAM,WAAW,mBAAmB;IAClC,8DAA8D;IAC9D,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,mDAAmD;IACnD,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,gEAAgE;IAChE,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,iDAAiD;IACjD,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,+DAA+D;IAC/D,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,kFAAkF;IAClF,QAAQ,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACzB;;OAEG;IACH,SAAS,CAAC,EAAE,OAAO,KAAK,CAAC;IACzB;;;;;;;;OAQG;IACH,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;CACxB;AAED,eAAO,MAAM,mBAAmB,8BAA8B,CAAC;AAC/D,eAAO,MAAM,gBAAgB,gBAAgB,CAAC;AAyC9C;;;;;;;GAOG;AACH,wBAAgB,cAAc,CAAC,OAAO,EAAE,MAAM,GAAG;IAAE,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,GAAG,IAAI,CAe1E;AA+FD;;;;;GAKG;AACH,qBAAa,oBAAqB,YAAW,UAAU;IACrD,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IAEtB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAS;IAC/B,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAgB;IACzC,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAqB;IACnD,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAe;IACzC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAU;IACrC,uFAAuF;IACvF,OAAO,CAAC,sBAAsB,CAAS;gBAE3B,OAAO,GAAE,mBAAwB;IAavC,YAAY,CAAC,QAAQ,EAAE,cAAc,GAAG,OAAO,CAAC,aAAa,EAAE,CAAC;IAsCtE,wEAAwE;YAC1D,cAAc;IAsC5B;;;;OAIG;YACW,iBAAiB;IA8D/B,OAAO,CAAC,WAAW;IAoBnB,OAAO,CAAC,SAAS;IAyBjB,OAAO,CAAC,UAAU;CAanB;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,qBAAqB,CACnC,GAAG,GAAE,MAAM,CAAC,UAAwB,GACnC,oBAAoB,GAAG,IAAI,CAa7B;AAED;;;;;;;;GAQG;AACH,wBAAsB,kBAAkB,CACtC,QAAQ,EAAE,cAAc,EACxB,QAAQ,EAAE,UAAU,EACpB,OAAO,EAAE,GAAG,CAAC,MAAM,EAAE,IAAI,CAAC,GACzB,OAAO,CAAC;IAAE,KAAK,EAAE,aAAa,EAAE,CAAC;IAAC,QAAQ,EAAE,MAAM,EAAE,CAAC;IAAC,YAAY,EAAE,MAAM,CAAA;CAAE,CAAC,CAQ/E"}