@k08200/mcp-probe 1.0.0 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. package/README.md +227 -15
  2. package/dist/assertions.d.ts +11 -0
  3. package/dist/assertions.d.ts.map +1 -0
  4. package/dist/assertions.js +156 -0
  5. package/dist/assertions.js.map +1 -0
  6. package/dist/checker.d.ts.map +1 -1
  7. package/dist/checker.js +46 -24
  8. package/dist/checker.js.map +1 -1
  9. package/dist/cli.js +102 -1
  10. package/dist/cli.js.map +1 -1
  11. package/dist/doctor.d.ts +15 -0
  12. package/dist/doctor.d.ts.map +1 -0
  13. package/dist/doctor.js +145 -0
  14. package/dist/doctor.js.map +1 -0
  15. package/dist/init.d.ts +22 -0
  16. package/dist/init.d.ts.map +1 -0
  17. package/dist/init.js +141 -0
  18. package/dist/init.js.map +1 -0
  19. package/dist/issues.d.ts +5 -0
  20. package/dist/issues.d.ts.map +1 -0
  21. package/dist/issues.js +126 -0
  22. package/dist/issues.js.map +1 -0
  23. package/dist/protocols/mcp-client.d.ts.map +1 -1
  24. package/dist/protocols/mcp-client.js +39 -19
  25. package/dist/protocols/mcp-client.js.map +1 -1
  26. package/dist/redact.d.ts +3 -0
  27. package/dist/redact.d.ts.map +1 -0
  28. package/dist/redact.js +34 -0
  29. package/dist/redact.js.map +1 -0
  30. package/dist/reporters/github.d.ts.map +1 -1
  31. package/dist/reporters/github.js +15 -9
  32. package/dist/reporters/github.js.map +1 -1
  33. package/dist/reporters/json-reporter.d.ts.map +1 -1
  34. package/dist/reporters/json-reporter.js +2 -1
  35. package/dist/reporters/json-reporter.js.map +1 -1
  36. package/dist/reporters/terminal.d.ts.map +1 -1
  37. package/dist/reporters/terminal.js +18 -5
  38. package/dist/reporters/terminal.js.map +1 -1
  39. package/dist/types.d.ts +25 -3
  40. package/dist/types.d.ts.map +1 -1
  41. package/examples/contract-failure.tools.json +24 -0
  42. package/examples/datadog.tools.json +4 -1
  43. package/examples/github-actions/fleet.yml +1 -1
  44. package/examples/github-actions/remote-server.yml +1 -1
  45. package/examples/github-actions/single-server.yml +1 -1
  46. package/examples/recipes/datadog.tools.json +7 -2
  47. package/examples/recipes/gmail.tools.json +6 -2
  48. package/examples/recipes/supabase.tools.json +18 -2
  49. package/examples/self-check.tools.json +20 -0
  50. package/package.json +16 -5
  51. package/schemas/mcp-probe.config.schema.json +74 -0
  52. package/schemas/mcp-probe.sidecar.schema.json +68 -0
package/README.md CHANGED
@@ -5,12 +5,23 @@
5
5
  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
6
6
  [![Node.js](https://img.shields.io/node/v/@k08200/mcp-probe)](package.json)
7
7
 
8
- **Quality checker for MCP servers.** Validates protocol handshake, discovery, optional tool-call dry-runs, and response latency in one command.
8
+ **CI readiness gate for MCP servers.** Validates protocol handshake, discovery, optional tool-call dry-runs, stderr noise, and response latency in one command.
9
9
 
10
- The `npm audit` for the [MCP](https://modelcontextprotocol.io) ecosystem — because [awesome-mcp-servers](https://github.com/punkpeye/awesome-mcp-servers) lists 200+ servers and there was no way to know if they actually worked.
10
+ The `npm audit` for the [MCP](https://modelcontextprotocol.io) ecosystem — because an MCP server can start, pass `tools/list`, and still fail every real tool call when auth handoff, browser OAuth, or downstream permissions are broken.
11
+
12
+ Read the v1 launch post: [mcp-probe v1.0.0: A CI readiness gate for MCP servers](https://dev.to/k08200/mcp-probe-v100-a-ci-readiness-gate-for-mcp-servers-4ch0)
11
13
 
12
14
  ## Quick Start for CI
13
15
 
16
+ Scaffold the config, sidecar, and GitHub Actions workflow:
17
+
18
+ ```bash
19
+ npx @k08200/mcp-probe@latest init \
20
+ --target @your-org/your-mcp-server \
21
+ --discover \
22
+ --github-actions
23
+ ```
24
+
14
25
  Add this workflow to any project that depends on MCP servers:
15
26
 
16
27
  ```yaml
@@ -27,7 +38,7 @@ jobs:
27
38
  timeout-minutes: 5
28
39
 
29
40
  steps:
30
- - uses: actions/checkout@v4
41
+ - uses: actions/checkout@v6
31
42
 
32
43
  - name: Validate MCP server
33
44
  run: |
@@ -42,6 +53,27 @@ For teams running several MCP servers, use a config file:
42
53
  npx @k08200/mcp-probe --config mcp-probe.config.json --github-summary
43
54
  ```
44
55
 
56
+ For production CI, add sidecar inputs so dry-runs call real read-only paths instead of schema-minimum placeholders:
57
+
58
+ ```json
59
+ {
60
+ "tools": {
61
+ "logs_query": {
62
+ "input": {
63
+ "query": "service:web status:error",
64
+ "timeframe": "1h"
65
+ },
66
+ "expect": {
67
+ "status": "pass",
68
+ "not_error_code": [401, 403],
69
+ "requiredFields": ["source", "freshness"],
70
+ "maxRows": 100
71
+ }
72
+ }
73
+ }
74
+ }
75
+ ```
76
+
45
77
  ```bash
46
78
  npx @k08200/mcp-probe @modelcontextprotocol/server-memory
47
79
  ```
@@ -77,6 +109,8 @@ mcp-probe @modelcontextprotocol/server-memory
77
109
 
78
110
  ## Install
79
111
 
112
+ Requires Node.js 20.19 or newer.
113
+
80
114
  ```bash
81
115
  # No install needed
82
116
  npx @k08200/mcp-probe <target>
@@ -91,6 +125,31 @@ npm install -g @k08200/mcp-probe
91
125
  # Check an npm package
92
126
  mcp-probe @modelcontextprotocol/server-memory
93
127
 
128
+ # Scaffold config + .mcp-probe.json + optional GitHub Actions workflow
129
+ mcp-probe init --target @modelcontextprotocol/server-memory --github-actions
130
+
131
+ # Discover tool names first and scaffold sidecar entries automatically
132
+ mcp-probe init --target @modelcontextprotocol/server-memory --discover --github-actions
133
+
134
+ # Check whether this project is ready to run mcp-probe in CI
135
+ mcp-probe doctor
136
+
137
+ # JSON output for scripting or internal CI preflight checks
138
+ mcp-probe doctor --config-file mcp-probe.config.json --output json
139
+
140
+ # Scaffold a remote server config with auth from an env var
141
+ mcp-probe init \
142
+ --target https://mcp.example.com/mcp \
143
+ --transport http \
144
+ --header-env MCP_TOKEN \
145
+ --github-actions
146
+
147
+ # Choose custom scaffold paths
148
+ mcp-probe init \
149
+ --target @your-org/your-mcp-server \
150
+ --config-file ci/mcp-probe.config.json \
151
+ --sidecar-file ci/mcp-tools.json
152
+
94
153
  # Check a server that requires arguments (e.g. directories to serve)
95
154
  mcp-probe @modelcontextprotocol/server-filesystem /tmp /Users/me/projects
96
155
 
@@ -143,8 +202,65 @@ mcp-probe @scope/server --tools-file .mcp-probe.json
143
202
  | **Prompts discovery** | Runs `prompts/list` when the server advertises prompts. |
144
203
  | **Tool call dry-run** | Optional `tools/call` checks via `--probe-tools` or `--tools-file`. |
145
204
 
205
+ ## Issue codes and remediation hints
206
+
207
+ When a check warns or fails, mcp-probe attaches stable issue metadata:
208
+
209
+ ```json
210
+ {
211
+ "name": "Tool call dry-run",
212
+ "status": "warn",
213
+ "message": "1 auth/permission errors (1 sidecar, 0 auto)",
214
+ "issue": {
215
+ "code": "TOOL_CALL_AUTH",
216
+ "hint": "At least one tool call hit auth or permission handling. This often means CI needs tokens or the server needs non-browser auth."
217
+ }
218
+ }
219
+ ```
220
+
221
+ These hints appear in terminal output, JSON output, GitHub Actions summaries, and workflow annotations so PR failures point at the likely fix instead of only showing raw MCP errors.
222
+
223
+ Common issue codes:
224
+
225
+ | Code | Meaning |
226
+ |------|---------|
227
+ | `TARGET_NOT_FOUND` | The npm package, local file, or executable could not be started. |
228
+ | `HANDSHAKE_TIMEOUT` | The server did not complete MCP `initialize` before the timeout. |
229
+ | `HANDSHAKE_AUTH` | Initialization failed with an auth-like error. |
230
+ | `NO_TOOLS` | The server responded but did not expose tools. |
231
+ | `TOOL_SCHEMA_INVALID` | A discovered tool has an invalid schema. |
232
+ | `TOOL_CALL_AUTH` | A real tool call reached auth or permission handling. |
233
+ | `CONTRACT_ASSERTION_FAILED` | A tool call completed but failed one or more sidecar assertions. |
234
+ | `AUTO_DRY_RUN_INPUT` | Auto-generated schema-minimum input failed; add sidecar inputs. |
235
+ | `TOOL_CALL_FAILED` | A sidecar tool call returned a non-auth error. |
236
+
146
237
  ## Batch CI gate
147
238
 
239
+ If you are starting from scratch, generate the files:
240
+
241
+ ```bash
242
+ mcp-probe init --target @your-org/your-mcp-server --discover --github-actions
243
+ ```
244
+
245
+ This creates:
246
+
247
+ | File | Purpose |
248
+ |------|---------|
249
+ | `mcp-probe.config.json` | Batch config with one server and `probeTools: true`. |
250
+ | `.mcp-probe.json` | Sidecar template for real tool-call sample inputs. |
251
+ | `.github/workflows/mcp-probe.yml` | GitHub Actions readiness gate. |
252
+
253
+ Existing files are skipped unless you pass `--force`.
254
+
255
+ Generated config and sidecar files include JSON Schema references:
256
+
257
+ | Schema | File |
258
+ |--------|------|
259
+ | [`mcp-probe.config.schema.json`](schemas/mcp-probe.config.schema.json) | `mcp-probe.config.json` |
260
+ | [`mcp-probe.sidecar.schema.json`](schemas/mcp-probe.sidecar.schema.json) | `.mcp-probe.json` |
261
+
262
+ When `--discover` is enabled, mcp-probe connects to the target server, runs discovery, and pre-populates `.mcp-probe.json` with the discovered tool names and schema-minimum sample inputs. Review those values before using them as a production CI gate.
263
+
148
264
  Use `--config` when a project depends on several MCP servers and you want one CI command to validate all of them:
149
265
 
150
266
  ```json
@@ -196,6 +312,29 @@ Config fields:
196
312
  | `servers[].probeTools` | Enables dry-run tool calls for that server. |
197
313
  | `servers[].toolsFile` | Sidecar input file for meaningful `tools/call` samples. Relative paths resolve from the config file directory. |
198
314
 
315
+ ## Project doctor
316
+
317
+ Use `mcp-probe doctor` before wiring mcp-probe into CI or after changing config files:
318
+
319
+ ```bash
320
+ mcp-probe doctor
321
+ ```
322
+
323
+ It checks:
324
+
325
+ | Check | Description |
326
+ |-------|-------------|
327
+ | **Node.js version** | Confirms the current runtime satisfies mcp-probe's required Node.js version. |
328
+ | **Config file** | Validates that `mcp-probe.config.json` exists and can be parsed. |
329
+ | **Sidecar files** | Validates each configured `toolsFile`, resolving relative paths from the config file directory. |
330
+ | **GitHub Actions workflow** | Warns when no workflow file mentions `mcp-probe`. |
331
+
332
+ For automation, use JSON output:
333
+
334
+ ```bash
335
+ mcp-probe doctor --config-file ci/mcp-probe.config.json --output json
336
+ ```
337
+
199
338
  ## Stderr classification
200
339
 
201
340
  Many MCP servers write harmless warnings to stderr during startup: optional config notices, update checks, deprecation warnings, and similar noise. If the server later fails to initialize, raw stderr can make those warnings look like the root cause.
@@ -260,6 +399,55 @@ mcp-probe @your-org/datadog-mcp --tools-file ./ci/mcp-tools.json
260
399
 
261
400
  Sidecar inputs are used first; generated minimal inputs are fallback only. Auth and permission failures such as 401/403 are surfaced as warnings so CI can distinguish "OAuth handoff needed" from transport or runtime failure.
262
401
 
402
+ ## Tool call contract assertions
403
+
404
+ For production MCP servers, especially database-backed servers, a successful `tools/call` is still not enough. Agents depend on a contract: read-only roles, scoped data, stable error codes, safe limits, and no leaked internals.
405
+
406
+ Add assertions to `.mcp-probe.json` to validate that contract:
407
+
408
+ ```json
409
+ {
410
+ "tools": {
411
+ "execute_sql": {
412
+ "input": {
413
+ "project_id": "YOUR_PROJECT_ID",
414
+ "query": "select 1 as health_check"
415
+ },
416
+ "expect": {
417
+ "status": "pass",
418
+ "requiredFields": ["rowCount", "limit", "source", "freshness"],
419
+ "maxRows": 100
420
+ }
421
+ },
422
+ "execute_sql_write_denied": {
423
+ "input": {
424
+ "project_id": "YOUR_PROJECT_ID",
425
+ "query": "delete from users where id = 1"
426
+ },
427
+ "expect": {
428
+ "status": "fail",
429
+ "errorCode": "WRITE_NOT_ALLOWED",
430
+ "notContains": ["DATABASE_URL", "password", "stack"]
431
+ }
432
+ }
433
+ }
434
+ }
435
+ ```
436
+
437
+ Supported assertions:
438
+
439
+ | Assertion | Purpose |
440
+ |-----------|---------|
441
+ | `status` | Expected call status: `pass`, `fail`, or `warn`. Use `fail` for denied-write probes. |
442
+ | `requiredFields` | Field names that must appear anywhere in the tool result payload. |
443
+ | `maxRows` | Maximum allowed row count, using `rowCount`, `rowsReturned`, or common row arrays. |
444
+ | `errorCode` | Stable error code expected in an error response. |
445
+ | `contains` | Text snippets that must appear in the result or error payload. |
446
+ | `notContains` | Text snippets that must not appear; useful for stack traces, secrets, and raw internals. |
447
+ | `not_error_code` | HTTP/status codes that should be warnings instead of failures, usually auth handoff codes. |
448
+
449
+ If an assertion fails, mcp-probe returns `CONTRACT_ASSERTION_FAILED` and includes per-assertion details in JSON and GitHub Actions summaries.
450
+
263
451
  ## Status badges
264
452
 
265
453
  Use `--badge-file` to write a [shields.io endpoint](https://shields.io/badges/endpoint-badge) JSON file:
@@ -311,7 +499,7 @@ jobs:
311
499
  timeout-minutes: 5
312
500
 
313
501
  steps:
314
- - uses: actions/checkout@v4
502
+ - uses: actions/checkout@v6
315
503
 
316
504
  - name: Validate MCP server
317
505
  run: |
@@ -340,7 +528,7 @@ jobs:
340
528
  timeout-minutes: 10
341
529
 
342
530
  steps:
343
- - uses: actions/checkout@v4
531
+ - uses: actions/checkout@v6
344
532
 
345
533
  - name: Validate MCP fleet
346
534
  run: |
@@ -362,6 +550,8 @@ Copy-ready examples live in [`examples/github-actions`](examples/github-actions)
362
550
 
363
551
  mcp-probe also dogfoods itself in CI with [`examples/self-check.config.json`](examples/self-check.config.json), which validates batch mode, sidecar inputs, GitHub summaries, and badge output against a local fixture MCP server.
364
552
 
553
+ It also includes [`examples/contract-failure.tools.json`](examples/contract-failure.tools.json), an intentionally broken sidecar used by CI to prove contract failures surface as `CONTRACT_ASSERTION_FAILED`. That fixture checks the negative path: missing metadata, row-limit violations, and denied writes that must fail safely.
554
+
365
555
  ## Recipes
366
556
 
367
557
  Production MCP checks work best with sidecar inputs that exercise real call paths instead of generated empty values. Copy-ready starting points live in [`examples/recipes`](examples/recipes):
@@ -377,25 +567,43 @@ Tool names vary by MCP server implementation. Run your server once with `--outpu
377
567
  ## JSON output
378
568
 
379
569
  ```bash
380
- mcp-probe @modelcontextprotocol/server-memory --probe-tools --output json
570
+ mcp-probe @your-org/datadog-mcp --tools-file .mcp-probe.json --output json
381
571
  ```
382
572
 
383
573
  ```json
384
574
  {
385
- "target": "@modelcontextprotocol/server-memory",
575
+ "target": "@your-org/datadog-mcp",
386
576
  "timestamp": "2026-05-17T12:00:00.000Z",
387
- "overallStatus": "pass",
577
+ "overallStatus": "warn",
388
578
  "checks": [
389
- { "name": "Target resolution", "status": "pass", "message": "npx --yes @modelcontextprotocol/server-memory" },
390
- { "name": "MCP protocol handshake", "status": "pass", "message": "memory-server v0.6.3", "latencyMs": 1392 },
391
- { "name": "Tools discovery", "status": "pass", "message": "Found 9 tools", "latencyMs": 33 },
579
+ { "name": "Target resolution", "status": "pass", "message": "npx --yes @your-org/datadog-mcp" },
580
+ { "name": "MCP protocol handshake", "status": "pass", "message": "datadog-mcp v1.0.0", "latencyMs": 1392 },
581
+ { "name": "Tools discovery", "status": "pass", "message": "Found 12 tools", "latencyMs": 33 },
392
582
  { "name": "Tool schema validation", "status": "pass", "message": "All tool schemas are valid" },
393
- { "name": "Tool call dry-run", "status": "pass", "message": "9 passed (2 sidecar, 7 auto)" }
583
+ {
584
+ "name": "Tool call dry-run",
585
+ "status": "warn",
586
+ "message": "1 auth/permission errors (1 sidecar, 0 auto)",
587
+ "issue": {
588
+ "code": "TOOL_CALL_AUTH",
589
+ "hint": "At least one tool call hit auth or permission handling. This often means CI needs tokens or the server needs non-browser auth."
590
+ }
591
+ }
394
592
  ],
395
- "serverInfo": { "name": "memory-server", "version": "0.6.3", "capabilities": ["tools"] },
396
- "tools": [{ "name": "create_entities", "description": "Create multiple new entities in the knowledge graph" }],
593
+ "serverInfo": { "name": "datadog-mcp", "version": "1.0.0", "capabilities": ["tools"] },
594
+ "tools": [{ "name": "logs_query", "description": "Query Datadog logs" }],
397
595
  "toolCallResults": [
398
- { "tool": "read_graph", "status": "pass", "latencyMs": 41, "source": "auto" }
596
+ {
597
+ "tool": "logs_query",
598
+ "status": "warn",
599
+ "latencyMs": 41,
600
+ "source": "sidecar",
601
+ "error": "401 Unauthorized",
602
+ "issue": {
603
+ "code": "TOOL_CALL_AUTH",
604
+ "hint": "The server registered this tool, but the call path hit auth or permission handling. Check OAuth/browser handoff, service tokens, and CI secrets."
605
+ }
606
+ }
399
607
  ],
400
608
  "totalLatencyMs": 1455
401
609
  }
@@ -423,6 +631,10 @@ mcp-probe @modelcontextprotocol/server-memory --probe-tools --output json
423
631
 
424
632
  Issues and PRs are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).
425
633
 
634
+ ## Changelog
635
+
636
+ See [CHANGELOG.md](CHANGELOG.md).
637
+
426
638
  ## License
427
639
 
428
640
  [MIT](LICENSE)
@@ -0,0 +1,11 @@
1
+ import type { AssertionResult, CheckStatus, ToolExpectations } from './types.js';
2
+ type EvaluationInput = {
3
+ result?: unknown;
4
+ error?: string;
5
+ actualStatus: CheckStatus;
6
+ expect?: ToolExpectations;
7
+ };
8
+ export declare function evaluateToolAssertions(input: EvaluationInput): AssertionResult[];
9
+ export declare function assertionFailureMessage(assertions: AssertionResult[]): string | undefined;
10
+ export {};
11
+ //# sourceMappingURL=assertions.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"assertions.d.ts","sourceRoot":"","sources":["../src/assertions.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,eAAe,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAEjF,KAAK,eAAe,GAAG;IACrB,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,YAAY,EAAE,WAAW,CAAC;IAC1B,MAAM,CAAC,EAAE,gBAAgB,CAAC;CAC3B,CAAC;AA6GF,wBAAgB,sBAAsB,CAAC,KAAK,EAAE,eAAe,GAAG,eAAe,EAAE,CAsDhF;AAED,wBAAgB,uBAAuB,CAAC,UAAU,EAAE,eAAe,EAAE,GAAG,MAAM,GAAG,SAAS,CAIzF"}
@@ -0,0 +1,156 @@
1
+ function stableStringify(value) {
2
+ if (typeof value === 'string')
3
+ return value;
4
+ try {
5
+ return JSON.stringify(value);
6
+ }
7
+ catch {
8
+ return String(value);
9
+ }
10
+ }
11
+ function parseMaybeJson(text) {
12
+ try {
13
+ return JSON.parse(text);
14
+ }
15
+ catch {
16
+ return text;
17
+ }
18
+ }
19
+ function extractPayload(result, error) {
20
+ const values = [];
21
+ const textParts = [];
22
+ if (error) {
23
+ values.push(error);
24
+ textParts.push(error);
25
+ }
26
+ if (result !== undefined) {
27
+ values.push(result);
28
+ textParts.push(stableStringify(result));
29
+ }
30
+ if (result && typeof result === 'object') {
31
+ const content = result.content;
32
+ if (Array.isArray(content)) {
33
+ for (const part of content) {
34
+ if (!part || typeof part !== 'object')
35
+ continue;
36
+ const record = part;
37
+ const text = typeof record.text === 'string' ? record.text : undefined;
38
+ if (text !== undefined) {
39
+ values.push(parseMaybeJson(text));
40
+ textParts.push(text);
41
+ }
42
+ for (const key of ['json', 'data', 'resource']) {
43
+ if (record[key] !== undefined) {
44
+ values.push(record[key]);
45
+ textParts.push(stableStringify(record[key]));
46
+ }
47
+ }
48
+ }
49
+ }
50
+ }
51
+ return { values, text: textParts.join('\n') };
52
+ }
53
+ function objectValues(value) {
54
+ if (Array.isArray(value))
55
+ return value.flatMap(objectValues);
56
+ if (!value || typeof value !== 'object')
57
+ return [];
58
+ const record = value;
59
+ return [record, ...Object.values(record).flatMap(objectValues)];
60
+ }
61
+ function hasField(value, field) {
62
+ if (!value || typeof value !== 'object')
63
+ return false;
64
+ if (Array.isArray(value))
65
+ return value.some((entry) => hasField(entry, field));
66
+ const record = value;
67
+ if (Object.prototype.hasOwnProperty.call(record, field))
68
+ return true;
69
+ return Object.values(record).some((entry) => hasField(entry, field));
70
+ }
71
+ function findNumberField(value, field) {
72
+ for (const candidate of objectValues(value)) {
73
+ const record = candidate;
74
+ const found = record[field];
75
+ if (typeof found === 'number')
76
+ return found;
77
+ if (typeof found === 'string' && found.trim() !== '' && Number.isFinite(Number(found))) {
78
+ return Number(found);
79
+ }
80
+ }
81
+ return undefined;
82
+ }
83
+ function findRowsLength(value) {
84
+ for (const candidate of objectValues(value)) {
85
+ const record = candidate;
86
+ const rows = record.rows ?? record.data ?? record.items ?? record.records;
87
+ if (Array.isArray(rows))
88
+ return rows.length;
89
+ }
90
+ return undefined;
91
+ }
92
+ function includesText(haystack, needle) {
93
+ return haystack.toLowerCase().includes(needle.toLowerCase());
94
+ }
95
+ function pass(name, message) {
96
+ return { name, status: 'pass', message };
97
+ }
98
+ function fail(name, message) {
99
+ return { name, status: 'fail', message };
100
+ }
101
+ export function evaluateToolAssertions(input) {
102
+ const { expect } = input;
103
+ if (!expect)
104
+ return [];
105
+ const payload = extractPayload(input.result, input.error);
106
+ const assertions = [];
107
+ if (expect.status) {
108
+ assertions.push(input.actualStatus === expect.status
109
+ ? pass('status', `Tool status matched expected ${expect.status}`)
110
+ : fail('status', `Expected tool status ${expect.status}, got ${input.actualStatus}`));
111
+ }
112
+ for (const field of expect.requiredFields ?? []) {
113
+ const found = payload.values.some((value) => hasField(value, field));
114
+ assertions.push(found
115
+ ? pass(`requiredFields.${field}`, `Found required field "${field}"`)
116
+ : fail(`requiredFields.${field}`, `Missing required field "${field}"`));
117
+ }
118
+ if (expect.maxRows !== undefined) {
119
+ const rowCount = payload.values
120
+ .map((value) => findNumberField(value, 'rowCount') ?? findNumberField(value, 'rowsReturned') ?? findRowsLength(value))
121
+ .find((value) => value !== undefined);
122
+ if (rowCount === undefined) {
123
+ assertions.push(fail('maxRows', 'Could not determine row count from result metadata'));
124
+ }
125
+ else {
126
+ assertions.push(rowCount <= expect.maxRows
127
+ ? pass('maxRows', `Row count ${rowCount} is within maxRows ${expect.maxRows}`)
128
+ : fail('maxRows', `Row count ${rowCount} exceeds maxRows ${expect.maxRows}`));
129
+ }
130
+ }
131
+ if (expect.errorCode) {
132
+ const found = payload.values.some((value) => hasField(value, 'code') && includesText(stableStringify(value), expect.errorCode))
133
+ || includesText(payload.text, expect.errorCode);
134
+ assertions.push(found
135
+ ? pass('errorCode', `Found expected error code ${expect.errorCode}`)
136
+ : fail('errorCode', `Missing expected error code ${expect.errorCode}`));
137
+ }
138
+ for (const expectedText of expect.contains ?? []) {
139
+ assertions.push(includesText(payload.text, expectedText)
140
+ ? pass(`contains.${expectedText}`, `Output contains "${expectedText}"`)
141
+ : fail(`contains.${expectedText}`, `Output does not contain "${expectedText}"`));
142
+ }
143
+ for (const forbiddenText of expect.notContains ?? []) {
144
+ assertions.push(!includesText(payload.text, forbiddenText)
145
+ ? pass(`notContains.${forbiddenText}`, `Output does not contain "${forbiddenText}"`)
146
+ : fail(`notContains.${forbiddenText}`, `Output leaked forbidden text "${forbiddenText}"`));
147
+ }
148
+ return assertions;
149
+ }
150
+ export function assertionFailureMessage(assertions) {
151
+ const failures = assertions.filter((assertion) => assertion.status === 'fail');
152
+ if (failures.length === 0)
153
+ return undefined;
154
+ return failures.map((assertion) => assertion.message).join('; ');
155
+ }
156
+ //# sourceMappingURL=assertions.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"assertions.js","sourceRoot":"","sources":["../src/assertions.ts"],"names":[],"mappings":"AAcA,SAAS,eAAe,CAAC,KAAc;IACrC,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,KAAK,CAAC;IAC5C,IAAI,CAAC;QACH,OAAO,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC;IAC/B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC;IACvB,CAAC;AACH,CAAC;AAED,SAAS,cAAc,CAAC,IAAY;IAClC,IAAI,CAAC;QACH,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAC1B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;IACd,CAAC;AACH,CAAC;AAED,SAAS,cAAc,CAAC,MAAe,EAAE,KAAc;IACrD,MAAM,MAAM,GAAc,EAAE,CAAC;IAC7B,MAAM,SAAS,GAAa,EAAE,CAAC;IAC/B,IAAI,KAAK,EAAE,CAAC;QACV,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACnB,SAAS,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IACxB,CAAC;IAED,IAAI,MAAM,KAAK,SAAS,EAAE,CAAC;QACzB,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACpB,SAAS,CAAC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,CAAC,CAAC;IAC1C,CAAC;IAED,IAAI,MAAM,IAAI,OAAO,MAAM,KAAK,QAAQ,EAAE,CAAC;QACzC,MAAM,OAAO,GAAI,MAAgC,CAAC,OAAO,CAAC;QAC1D,IAAI,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;YAC3B,KAAK,MAAM,IAAI,IAAI,OAAO,EAAE,CAAC;gBAC3B,IAAI,CAAC,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ;oBAAE,SAAS;gBAChD,MAAM,MAAM,GAAG,IAA+B,CAAC;gBAC/C,MAAM,IAAI,GAAG,OAAO,MAAM,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;gBACvE,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;oBACvB,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,CAAC,CAAC;oBAClC,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;gBACvB,CAAC;gBACD,KAAK,MAAM,GAAG,IAAI,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,CAAC,EAAE,CAAC;oBAC/C,IAAI,MAAM,CAAC,GAAG,CAAC,KAAK,SAAS,EAAE,CAAC;wBAC9B,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;wBACzB,SAAS,CAAC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;oBAC/C,CAAC;gBACH,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;IAED,OAAO,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;AAChD,CAAC;AAED,SAAS,YAAY,CAAC,KAAc;IAClC,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,OAAO,KAAK,CAAC,OAAO,CAAC,YAAY,CAAC,CAAC;IAC7D,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,EAAE,CAAC;IACnD,MAAM,MAAM,GAAG,KAAgC,CAAC;IAChD,OAAO,CAAC,MAAM,EAAE,GAAG,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,OAAO,CAAC,YAAY,CAAC,CAAC,CAAC;AAClE,CAAC;AAED,SAAS,QAAQ,CAAC,KAAc,EAAE,KAAa;IAC7C,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,KAAK,CAAC;IACtD,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,OAAO,KAAK,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC;IAC/E,MAAM,MAAM,GAAG,KAAgC,CAAC;IAChD,IAAI,MAAM,CAAC,SAAS,CAAC,cAAc,CAAC,IAAI,CAAC,MAAM,EAAE,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC;IACrE,OAAO,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC;AACvE,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,KAAa;IACpD,KAAK,MAAM,SAAS,IAAI,YAAY,CAAC,KAAK,CAAC,EAAE,CAAC;QAC5C,MAAM,MAAM,GAAG,SAAoC,CAAC;QACpD,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;QAC5B,IAAI,OAAO,KAAK,KAAK,QAAQ;YAAE,OAAO,KAAK,CAAC;QAC5C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC;YACvF,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC;QACvB,CAAC;IACH,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED,SAAS,cAAc,CAAC,KAAc;IACpC,KAAK,MAAM,SAAS,IAAI,YAAY,CAAC,KAAK,CAAC,EAAE,CAAC;QAC5C,MAAM,MAAM,GAAG,SAAoC,CAAC;QACpD,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,MAAM,CAAC,IAAI,IAAI,MAAM,CAAC,KAAK,IAAI,MAAM,CAAC,OAAO,CAAC;QAC1E,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;YAAE,OAAO,IAAI,CAAC,MAAM,CAAC;IAC9C,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED,SAAS,YAAY,CAAC,QAAgB,EAAE,MAAc;IACpD,OAAO,QAAQ,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC;AAC/D,CAAC;AAED,SAAS,IAAI,CAAC,IAAY,EAAE,OAAe;IACzC,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC;AAC3C,CAAC;AAED,SAAS,IAAI,CAAC,IAAY,EAAE,OAAe;IACzC,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC;AAC3C,CAAC;AAED,MAAM,UAAU,sBAAsB,CAAC,KAAsB;IAC3D,MAAM,EAAE,MAAM,EAAE,GAAG,KAAK,CAAC;IACzB,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IAEvB,MAAM,OAAO,GAAG,cAAc,CAAC,KAAK,CAAC,MAAM,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC;IAC1D,MAAM,UAAU,GAAsB,EAAE,CAAC;IAEzC,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;QAClB,UAAU,CAAC,IAAI,CAAC,KAAK,CAAC,YAAY,KAAK,MAAM,CAAC,MAAM;YAClD,CAAC,CAAC,IAAI,CAAC,QAAQ,EAAE,gCAAgC,MAAM,CAAC,MAAM,EAAE,CAAC;YACjE,CAAC,CAAC,IAAI,CAAC,QAAQ,EAAE,wBAAwB,MAAM,CAAC,MAAM,SAAS,KAAK,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC;IAC1F,CAAC;IAED,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,cAAc,IAAI,EAAE,EAAE,CAAC;QAChD,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC;QACrE,UAAU,CAAC,IAAI,CAAC,KAAK;YACnB,CAAC,CAAC,IAAI,CAAC,kBAAkB,KAAK,EAAE,EAAE,yBAAyB,KAAK,GAAG,CAAC;YACpE,CAAC,CAAC,IAAI,CAAC,kBAAkB,KAAK,EAAE,EAAE,2BAA2B,KAAK,GAAG,CAAC,CAAC,CAAC;IAC5E,CAAC;IAED,IAAI,MAAM,CAAC,OAAO,KAAK,SAAS,EAAE,CAAC;QACjC,MAAM,QAAQ,GAAG,OAAO,CAAC,MAAM;aAC5B,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,eAAe,CAAC,KAAK,EAAE,UAAU,CAAC,IAAI,eAAe,CAAC,KAAK,EAAE,cAAc,CAAC,IAAI,cAAc,CAAC,KAAK,CAAC,CAAC;aACrH,IAAI,CAAC,CAAC,KAAK,EAAmB,EAAE,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC;QACzD,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAC3B,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,oDAAoD,CAAC,CAAC,CAAC;QACzF,CAAC;aAAM,CAAC;YACN,UAAU,CAAC,IAAI,CAAC,QAAQ,IAAI,MAAM,CAAC,OAAO;gBACxC,CAAC,CAAC,IAAI,CAAC,SAAS,EAAE,aAAa,QAAQ,sBAAsB,MAAM,CAAC,OAAO,EAAE,CAAC;gBAC9E,CAAC,CAAC,IAAI,CAAC,SAAS,EAAE,aAAa,QAAQ,oBAAoB,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC;QAClF,CAAC;IACH,CAAC;IAED,IAAI,MAAM,CAAC,SAAS,EAAE,CAAC;QACrB,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC,IAAI,YAAY,CAAC,eAAe,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC,SAAU,CAAC,CAAC;eAC3H,YAAY,CAAC,OAAO,CAAC,IAAI,EAAE,MAAM,CAAC,SAAS,CAAC,CAAC;QAClD,UAAU,CAAC,IAAI,CAAC,KAAK;YACnB,CAAC,CAAC,IAAI,CAAC,WAAW,EAAE,6BAA6B,MAAM,CAAC,SAAS,EAAE,CAAC;YACpE,CAAC,CAAC,IAAI,CAAC,WAAW,EAAE,+BAA+B,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;IAC5E,CAAC;IAED,KAAK,MAAM,YAAY,IAAI,MAAM,CAAC,QAAQ,IAAI,EAAE,EAAE,CAAC;QACjD,UAAU,CAAC,IAAI,CAAC,YAAY,CAAC,OAAO,CAAC,IAAI,EAAE,YAAY,CAAC;YACtD,CAAC,CAAC,IAAI,CAAC,YAAY,YAAY,EAAE,EAAE,oBAAoB,YAAY,GAAG,CAAC;YACvE,CAAC,CAAC,IAAI,CAAC,YAAY,YAAY,EAAE,EAAE,4BAA4B,YAAY,GAAG,CAAC,CAAC,CAAC;IACrF,CAAC;IAED,KAAK,MAAM,aAAa,IAAI,MAAM,CAAC,WAAW,IAAI,EAAE,EAAE,CAAC;QACrD,UAAU,CAAC,IAAI,CAAC,CAAC,YAAY,CAAC,OAAO,CAAC,IAAI,EAAE,aAAa,CAAC;YACxD,CAAC,CAAC,IAAI,CAAC,eAAe,aAAa,EAAE,EAAE,4BAA4B,aAAa,GAAG,CAAC;YACpF,CAAC,CAAC,IAAI,CAAC,eAAe,aAAa,EAAE,EAAE,iCAAiC,aAAa,GAAG,CAAC,CAAC,CAAC;IAC/F,CAAC;IAED,OAAO,UAAU,CAAC;AACpB,CAAC;AAED,MAAM,UAAU,uBAAuB,CAAC,UAA6B;IACnE,MAAM,QAAQ,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC;IAC/E,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,SAAS,CAAC;IAC5C,OAAO,QAAQ,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACnE,CAAC"}
@@ -1 +1 @@
1
- {"version":3,"file":"checker.d.ts","sourceRoot":"","sources":["../src/checker.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAa,YAAY,EAAE,WAAW,EAAe,cAAc,EAAe,aAAa,EAAE,MAAM,YAAY,CAAC;AAQhI,wBAAgB,aAAa,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,CAAC,EAAE,aAAa,GAAG,cAAc,CAavF;AAiDD,wBAAsB,cAAc,CAAC,OAAO,EAAE,YAAY,GAAG,OAAO,CAAC,WAAW,CAAC,CA8HhF"}
1
+ {"version":3,"file":"checker.d.ts","sourceRoot":"","sources":["../src/checker.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAa,YAAY,EAAE,WAAW,EAAe,cAAc,EAAe,aAAa,EAAE,MAAM,YAAY,CAAC;AAQhI,wBAAgB,aAAa,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,CAAC,EAAE,aAAa,GAAG,cAAc,CAavF;AA4ED,wBAAsB,cAAc,CAAC,OAAO,EAAE,YAAY,GAAG,OAAO,CAAC,WAAW,CAAC,CAiIhF"}