@k08200/mcp-probe 0.8.0 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.md +267 -13
  2. package/dist/assertions.d.ts +11 -0
  3. package/dist/assertions.d.ts.map +1 -0
  4. package/dist/assertions.js +156 -0
  5. package/dist/assertions.js.map +1 -0
  6. package/dist/checker.d.ts.map +1 -1
  7. package/dist/checker.js +47 -24
  8. package/dist/checker.js.map +1 -1
  9. package/dist/cli.js +76 -3
  10. package/dist/cli.js.map +1 -1
  11. package/dist/config.d.ts.map +1 -1
  12. package/dist/config.js +21 -0
  13. package/dist/config.js.map +1 -1
  14. package/dist/init.d.ts +22 -0
  15. package/dist/init.d.ts.map +1 -0
  16. package/dist/init.js +141 -0
  17. package/dist/init.js.map +1 -0
  18. package/dist/issues.d.ts +5 -0
  19. package/dist/issues.d.ts.map +1 -0
  20. package/dist/issues.js +126 -0
  21. package/dist/issues.js.map +1 -0
  22. package/dist/protocols/mcp-client.d.ts.map +1 -1
  23. package/dist/protocols/mcp-client.js +57 -27
  24. package/dist/protocols/mcp-client.js.map +1 -1
  25. package/dist/redact.d.ts +3 -0
  26. package/dist/redact.d.ts.map +1 -0
  27. package/dist/redact.js +34 -0
  28. package/dist/redact.js.map +1 -0
  29. package/dist/reporters/github.d.ts.map +1 -1
  30. package/dist/reporters/github.js +15 -9
  31. package/dist/reporters/github.js.map +1 -1
  32. package/dist/reporters/json-reporter.d.ts.map +1 -1
  33. package/dist/reporters/json-reporter.js +2 -1
  34. package/dist/reporters/json-reporter.js.map +1 -1
  35. package/dist/reporters/terminal.d.ts.map +1 -1
  36. package/dist/reporters/terminal.js +18 -5
  37. package/dist/reporters/terminal.js.map +1 -1
  38. package/dist/types.d.ts +32 -3
  39. package/dist/types.d.ts.map +1 -1
  40. package/examples/datadog.tools.json +4 -1
  41. package/examples/mcp-probe.config.json +4 -0
  42. package/examples/recipes/datadog.tools.json +7 -2
  43. package/examples/recipes/gmail.tools.json +6 -2
  44. package/examples/recipes/supabase.tools.json +18 -2
  45. package/examples/self-check.config.json +14 -0
  46. package/examples/self-check.tools.json +37 -0
  47. package/package.json +16 -5
  48. package/schemas/mcp-probe.config.schema.json +74 -0
  49. package/schemas/mcp-probe.sidecar.schema.json +68 -0
package/README.md CHANGED
@@ -5,9 +5,74 @@
5
5
  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
6
6
  [![Node.js](https://img.shields.io/node/v/@k08200/mcp-probe)](package.json)
7
7
 
8
- **Quality checker for MCP servers.** Validates protocol handshake, discovery, optional tool-call dry-runs, and response latency in one command.
8
+ **CI readiness gate for MCP servers.** Validates protocol handshake, discovery, optional tool-call dry-runs, stderr noise, and response latency in one command.
9
9
 
10
- The `npm audit` for the [MCP](https://modelcontextprotocol.io) ecosystem — because [awesome-mcp-servers](https://github.com/punkpeye/awesome-mcp-servers) lists 200+ servers and there was no way to know if they actually worked.
10
+ The `npm audit` for the [MCP](https://modelcontextprotocol.io) ecosystem — because an MCP server can start, pass `tools/list`, and still fail every real tool call when auth handoff, browser OAuth, or downstream permissions are broken.
11
+
12
+ Read the v1 launch post: [mcp-probe v1.0.0: A CI readiness gate for MCP servers](https://dev.to/k08200/mcp-probe-v100-a-ci-readiness-gate-for-mcp-servers-4ch0)
13
+
14
+ ## Quick Start for CI
15
+
16
+ Scaffold the config, sidecar, and GitHub Actions workflow:
17
+
18
+ ```bash
19
+ npx @k08200/mcp-probe@latest init \
20
+ --target @your-org/your-mcp-server \
21
+ --discover \
22
+ --github-actions
23
+ ```
24
+
25
+ Add this workflow to any project that depends on MCP servers:
26
+
27
+ ```yaml
28
+ name: MCP Probe
29
+
30
+ on:
31
+ pull_request:
32
+ push:
33
+ branches: [main]
34
+
35
+ jobs:
36
+ mcp-probe:
37
+ runs-on: ubuntu-latest
38
+ timeout-minutes: 5
39
+
40
+ steps:
41
+ - uses: actions/checkout@v4
42
+
43
+ - name: Validate MCP server
44
+ run: |
45
+ npx @k08200/mcp-probe @your-org/your-mcp-server \
46
+ --probe-tools \
47
+ --github-summary
48
+ ```
49
+
50
+ For teams running several MCP servers, use a config file:
51
+
52
+ ```bash
53
+ npx @k08200/mcp-probe --config mcp-probe.config.json --github-summary
54
+ ```
55
+
56
+ For production CI, add sidecar inputs so dry-runs call real read-only paths instead of schema-minimum placeholders:
57
+
58
+ ```json
59
+ {
60
+ "tools": {
61
+ "logs_query": {
62
+ "input": {
63
+ "query": "service:web status:error",
64
+ "timeframe": "1h"
65
+ },
66
+ "expect": {
67
+ "status": "pass",
68
+ "not_error_code": [401, 403],
69
+ "requiredFields": ["source", "freshness"],
70
+ "maxRows": 100
71
+ }
72
+ }
73
+ }
74
+ }
75
+ ```
11
76
 
12
77
  ```bash
13
78
  npx @k08200/mcp-probe @modelcontextprotocol/server-memory
@@ -44,6 +109,8 @@ mcp-probe @modelcontextprotocol/server-memory
44
109
 
45
110
  ## Install
46
111
 
112
+ Requires Node.js 20.19 or newer.
113
+
47
114
  ```bash
48
115
  # No install needed
49
116
  npx @k08200/mcp-probe <target>
@@ -58,6 +125,25 @@ npm install -g @k08200/mcp-probe
58
125
  # Check an npm package
59
126
  mcp-probe @modelcontextprotocol/server-memory
60
127
 
128
+ # Scaffold config + .mcp-probe.json + optional GitHub Actions workflow
129
+ mcp-probe init --target @modelcontextprotocol/server-memory --github-actions
130
+
131
+ # Discover tool names first and scaffold sidecar entries automatically
132
+ mcp-probe init --target @modelcontextprotocol/server-memory --discover --github-actions
133
+
134
+ # Scaffold a remote server config with auth from an env var
135
+ mcp-probe init \
136
+ --target https://mcp.example.com/mcp \
137
+ --transport http \
138
+ --header-env MCP_TOKEN \
139
+ --github-actions
140
+
141
+ # Choose custom scaffold paths
142
+ mcp-probe init \
143
+ --target @your-org/your-mcp-server \
144
+ --config-file ci/mcp-probe.config.json \
145
+ --sidecar-file ci/mcp-tools.json
146
+
61
147
  # Check a server that requires arguments (e.g. directories to serve)
62
148
  mcp-probe @modelcontextprotocol/server-filesystem /tmp /Users/me/projects
63
149
 
@@ -73,6 +159,9 @@ mcp-probe https://mcp.example.com/sse --transport sse
73
159
  # Pass headers to remote servers
74
160
  mcp-probe https://mcp.example.com/mcp --header "Authorization: Bearer $TOKEN"
75
161
 
162
+ # Ignore known noisy stderr lines when classifying startup failures
163
+ mcp-probe @scope/server --stderr-allow "^Warning:" --stderr-fatal "panic|FATAL"
164
+
76
165
  # JSON output for CI / scripting
77
166
  mcp-probe @scope/server --output json
78
167
 
@@ -107,8 +196,65 @@ mcp-probe @scope/server --tools-file .mcp-probe.json
107
196
  | **Prompts discovery** | Runs `prompts/list` when the server advertises prompts. |
108
197
  | **Tool call dry-run** | Optional `tools/call` checks via `--probe-tools` or `--tools-file`. |
109
198
 
199
+ ## Issue codes and remediation hints
200
+
201
+ When a check warns or fails, mcp-probe attaches stable issue metadata:
202
+
203
+ ```json
204
+ {
205
+ "name": "Tool call dry-run",
206
+ "status": "warn",
207
+ "message": "1 auth/permission errors (1 sidecar, 0 auto)",
208
+ "issue": {
209
+ "code": "TOOL_CALL_AUTH",
210
+ "hint": "At least one tool call hit auth or permission handling. This often means CI needs tokens or the server needs non-browser auth."
211
+ }
212
+ }
213
+ ```
214
+
215
+ These hints appear in terminal output, JSON output, GitHub Actions summaries, and workflow annotations so PR failures point at the likely fix instead of only showing raw MCP errors.
216
+
217
+ Common issue codes:
218
+
219
+ | Code | Meaning |
220
+ |------|---------|
221
+ | `TARGET_NOT_FOUND` | The npm package, local file, or executable could not be started. |
222
+ | `HANDSHAKE_TIMEOUT` | The server did not complete MCP `initialize` before the timeout. |
223
+ | `HANDSHAKE_AUTH` | Initialization failed with an auth-like error. |
224
+ | `NO_TOOLS` | The server responded but did not expose tools. |
225
+ | `TOOL_SCHEMA_INVALID` | A discovered tool has an invalid schema. |
226
+ | `TOOL_CALL_AUTH` | A real tool call reached auth or permission handling. |
227
+ | `CONTRACT_ASSERTION_FAILED` | A tool call completed but failed one or more sidecar assertions. |
228
+ | `AUTO_DRY_RUN_INPUT` | Auto-generated schema-minimum input failed; add sidecar inputs. |
229
+ | `TOOL_CALL_FAILED` | A sidecar tool call returned a non-auth error. |
230
+
110
231
  ## Batch CI gate
111
232
 
233
+ If you are starting from scratch, generate the files:
234
+
235
+ ```bash
236
+ mcp-probe init --target @your-org/your-mcp-server --discover --github-actions
237
+ ```
238
+
239
+ This creates:
240
+
241
+ | File | Purpose |
242
+ |------|---------|
243
+ | `mcp-probe.config.json` | Batch config with one server and `probeTools: true`. |
244
+ | `.mcp-probe.json` | Sidecar template for real tool-call sample inputs. |
245
+ | `.github/workflows/mcp-probe.yml` | GitHub Actions readiness gate. |
246
+
247
+ Existing files are skipped unless you pass `--force`.
248
+
249
+ Generated config and sidecar files include JSON Schema references:
250
+
251
+ | Schema | File |
252
+ |--------|------|
253
+ | [`mcp-probe.config.schema.json`](schemas/mcp-probe.config.schema.json) | `mcp-probe.config.json` |
254
+ | [`mcp-probe.sidecar.schema.json`](schemas/mcp-probe.sidecar.schema.json) | `.mcp-probe.json` |
255
+
256
+ When `--discover` is enabled, mcp-probe connects to the target server, runs discovery, and pre-populates `.mcp-probe.json` with the discovered tool names and schema-minimum sample inputs. Review those values before using them as a production CI gate.
257
+
112
258
  Use `--config` when a project depends on several MCP servers and you want one CI command to validate all of them:
113
259
 
114
260
  ```json
@@ -127,6 +273,10 @@ Use `--config` when a project depends on several MCP servers and you want one CI
127
273
  "headers": {
128
274
  "Authorization": "Bearer ${DATADOG_MCP_TOKEN}"
129
275
  },
276
+ "stderr": {
277
+ "allow": ["^Warning:", "missing optional config"],
278
+ "fatal": ["panic", "FATAL"]
279
+ },
130
280
  "toolsFile": "./recipes/datadog.tools.json"
131
281
  }
132
282
  ]
@@ -151,9 +301,39 @@ Config fields:
151
301
  | `servers[].serverArgs` | Optional arguments passed to the MCP server. |
152
302
  | `servers[].transport` | Optional transport override: `stdio`, `http`, or `sse`. URL targets default to `http`; package/path targets default to `stdio`. |
153
303
  | `servers[].headers` | Optional HTTP headers for remote MCP servers. `${ENV_VAR}` placeholders are expanded at runtime. |
304
+ | `servers[].stderr.allow` | Optional regex patterns for stderr lines that should be ignored when startup fails. |
305
+ | `servers[].stderr.fatal` | Optional regex patterns for stderr lines that should always be treated as the startup failure reason. |
154
306
  | `servers[].probeTools` | Enables dry-run tool calls for that server. |
155
307
  | `servers[].toolsFile` | Sidecar input file for meaningful `tools/call` samples. Relative paths resolve from the config file directory. |
156
308
 
309
+ ## Stderr classification
310
+
311
+ Many MCP servers write harmless warnings to stderr during startup: optional config notices, update checks, deprecation warnings, and similar noise. If the server later fails to initialize, raw stderr can make those warnings look like the root cause.
312
+
313
+ mcp-probe has built-in warning filters and also lets you declare server-specific regexes:
314
+
315
+ ```bash
316
+ mcp-probe @scope/server \
317
+ --stderr-allow "^Warning:" \
318
+ --stderr-allow "missing optional config" \
319
+ --stderr-fatal "panic|FATAL"
320
+ ```
321
+
322
+ For batch checks, put the rules in `mcp-probe.config.json`:
323
+
324
+ ```json
325
+ {
326
+ "name": "datadog",
327
+ "target": "https://mcp.example.com/mcp",
328
+ "stderr": {
329
+ "allow": ["^Warning:", "missing optional config"],
330
+ "fatal": ["panic", "FATAL"]
331
+ }
332
+ }
333
+ ```
334
+
335
+ `fatal` patterns win over `allow` patterns. If every stderr line is allowed noise, mcp-probe reports the actual connection/init error instead of the warning text.
336
+
157
337
  ## Tool call dry-runs
158
338
 
159
339
  Discovery proves that a server starts and registers tools. It does **not** prove that the tools actually work in an agent loop. Use `--probe-tools` to call every discovered tool.
@@ -190,6 +370,55 @@ mcp-probe @your-org/datadog-mcp --tools-file ./ci/mcp-tools.json
190
370
 
191
371
  Sidecar inputs are used first; generated minimal inputs are fallback only. Auth and permission failures such as 401/403 are surfaced as warnings so CI can distinguish "OAuth handoff needed" from transport or runtime failure.
192
372
 
373
+ ## Tool call contract assertions
374
+
375
+ For production MCP servers, especially database-backed servers, a successful `tools/call` is still not enough. Agents depend on a contract: read-only roles, scoped data, stable error codes, safe limits, and no leaked internals.
376
+
377
+ Add assertions to `.mcp-probe.json` to validate that contract:
378
+
379
+ ```json
380
+ {
381
+ "tools": {
382
+ "execute_sql": {
383
+ "input": {
384
+ "project_id": "YOUR_PROJECT_ID",
385
+ "query": "select 1 as health_check"
386
+ },
387
+ "expect": {
388
+ "status": "pass",
389
+ "requiredFields": ["rowCount", "limit", "source", "freshness"],
390
+ "maxRows": 100
391
+ }
392
+ },
393
+ "execute_sql_write_denied": {
394
+ "input": {
395
+ "project_id": "YOUR_PROJECT_ID",
396
+ "query": "delete from users where id = 1"
397
+ },
398
+ "expect": {
399
+ "status": "fail",
400
+ "errorCode": "WRITE_NOT_ALLOWED",
401
+ "notContains": ["DATABASE_URL", "password", "stack"]
402
+ }
403
+ }
404
+ }
405
+ }
406
+ ```
407
+
408
+ Supported assertions:
409
+
410
+ | Assertion | Purpose |
411
+ |-----------|---------|
412
+ | `status` | Expected call status: `pass`, `fail`, or `warn`. Use `fail` for denied-write probes. |
413
+ | `requiredFields` | Field names that must appear anywhere in the tool result payload. |
414
+ | `maxRows` | Maximum allowed row count, using `rowCount`, `rowsReturned`, or common row arrays. |
415
+ | `errorCode` | Stable error code expected in an error response. |
416
+ | `contains` | Text snippets that must appear in the result or error payload. |
417
+ | `notContains` | Text snippets that must not appear; useful for stack traces, secrets, and raw internals. |
418
+ | `not_error_code` | HTTP/status codes that should be warnings instead of failures, usually auth handoff codes. |
419
+
420
+ If an assertion fails, mcp-probe returns `CONTRACT_ASSERTION_FAILED` and includes per-assertion details in JSON and GitHub Actions summaries.
421
+
193
422
  ## Status badges
194
423
 
195
424
  Use `--badge-file` to write a [shields.io endpoint](https://shields.io/badges/endpoint-badge) JSON file:
@@ -290,6 +519,8 @@ Copy-ready examples live in [`examples/github-actions`](examples/github-actions)
290
519
  | [`fleet.yml`](examples/github-actions/fleet.yml) | Validate several MCP servers from `mcp-probe.config.json` on PRs and hourly schedules. |
291
520
  | [`remote-server.yml`](examples/github-actions/remote-server.yml) | Validate a remote Streamable HTTP MCP server with auth headers. |
292
521
 
522
+ mcp-probe also dogfoods itself in CI with [`examples/self-check.config.json`](examples/self-check.config.json), which validates batch mode, sidecar inputs, GitHub summaries, and badge output against a local fixture MCP server.
523
+
293
524
  ## Recipes
294
525
 
295
526
  Production MCP checks work best with sidecar inputs that exercise real call paths instead of generated empty values. Copy-ready starting points live in [`examples/recipes`](examples/recipes):
@@ -305,25 +536,43 @@ Tool names vary by MCP server implementation. Run your server once with `--outpu
305
536
  ## JSON output
306
537
 
307
538
  ```bash
308
- mcp-probe @modelcontextprotocol/server-memory --probe-tools --output json
539
+ mcp-probe @your-org/datadog-mcp --tools-file .mcp-probe.json --output json
309
540
  ```
310
541
 
311
542
  ```json
312
543
  {
313
- "target": "@modelcontextprotocol/server-memory",
544
+ "target": "@your-org/datadog-mcp",
314
545
  "timestamp": "2026-05-17T12:00:00.000Z",
315
- "overallStatus": "pass",
546
+ "overallStatus": "warn",
316
547
  "checks": [
317
- { "name": "Target resolution", "status": "pass", "message": "npx --yes @modelcontextprotocol/server-memory" },
318
- { "name": "MCP protocol handshake", "status": "pass", "message": "memory-server v0.6.3", "latencyMs": 1392 },
319
- { "name": "Tools discovery", "status": "pass", "message": "Found 9 tools", "latencyMs": 33 },
548
+ { "name": "Target resolution", "status": "pass", "message": "npx --yes @your-org/datadog-mcp" },
549
+ { "name": "MCP protocol handshake", "status": "pass", "message": "datadog-mcp v1.0.0", "latencyMs": 1392 },
550
+ { "name": "Tools discovery", "status": "pass", "message": "Found 12 tools", "latencyMs": 33 },
320
551
  { "name": "Tool schema validation", "status": "pass", "message": "All tool schemas are valid" },
321
- { "name": "Tool call dry-run", "status": "pass", "message": "9 passed (2 sidecar, 7 auto)" }
552
+ {
553
+ "name": "Tool call dry-run",
554
+ "status": "warn",
555
+ "message": "1 auth/permission errors (1 sidecar, 0 auto)",
556
+ "issue": {
557
+ "code": "TOOL_CALL_AUTH",
558
+ "hint": "At least one tool call hit auth or permission handling. This often means CI needs tokens or the server needs non-browser auth."
559
+ }
560
+ }
322
561
  ],
323
- "serverInfo": { "name": "memory-server", "version": "0.6.3", "capabilities": ["tools"] },
324
- "tools": [{ "name": "create_entities", "description": "Create multiple new entities in the knowledge graph" }],
562
+ "serverInfo": { "name": "datadog-mcp", "version": "1.0.0", "capabilities": ["tools"] },
563
+ "tools": [{ "name": "logs_query", "description": "Query Datadog logs" }],
325
564
  "toolCallResults": [
326
- { "tool": "read_graph", "status": "pass", "latencyMs": 41, "source": "auto" }
565
+ {
566
+ "tool": "logs_query",
567
+ "status": "warn",
568
+ "latencyMs": 41,
569
+ "source": "sidecar",
570
+ "error": "401 Unauthorized",
571
+ "issue": {
572
+ "code": "TOOL_CALL_AUTH",
573
+ "hint": "The server registered this tool, but the call path hit auth or permission handling. Check OAuth/browser handoff, service tokens, and CI secrets."
574
+ }
575
+ }
327
576
  ],
328
577
  "totalLatencyMs": 1455
329
578
  }
@@ -339,17 +588,22 @@ mcp-probe @modelcontextprotocol/server-memory --probe-tools --output json
339
588
 
340
589
  ## Roadmap
341
590
 
591
+ - [x] Self-check batch workflow for mcp-probe itself
342
592
  - [x] HTTP/SSE transport support
343
593
  - [x] Batch checking from a config file (`mcp-probe --config mcp-probe.config.json`)
344
594
  - [x] GitHub Actions summary and annotations
345
595
  - [x] Badge generation (`mcp-probe --badge-file mcp-probe-badge.json`)
346
- - [ ] Structured stderr conventions for MCP server authors
596
+ - [x] Structured stderr classification rules
347
597
  - [x] Server-specific recipe examples for Datadog, Supabase, and Gmail MCP servers
348
598
 
349
599
  ## Contributing
350
600
 
351
601
  Issues and PRs are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).
352
602
 
603
+ ## Changelog
604
+
605
+ See [CHANGELOG.md](CHANGELOG.md).
606
+
353
607
  ## License
354
608
 
355
609
  [MIT](LICENSE)
@@ -0,0 +1,11 @@
1
+ import type { AssertionResult, CheckStatus, ToolExpectations } from './types.js';
2
+ type EvaluationInput = {
3
+ result?: unknown;
4
+ error?: string;
5
+ actualStatus: CheckStatus;
6
+ expect?: ToolExpectations;
7
+ };
8
+ export declare function evaluateToolAssertions(input: EvaluationInput): AssertionResult[];
9
+ export declare function assertionFailureMessage(assertions: AssertionResult[]): string | undefined;
10
+ export {};
11
+ //# sourceMappingURL=assertions.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"assertions.d.ts","sourceRoot":"","sources":["../src/assertions.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,eAAe,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAEjF,KAAK,eAAe,GAAG;IACrB,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,YAAY,EAAE,WAAW,CAAC;IAC1B,MAAM,CAAC,EAAE,gBAAgB,CAAC;CAC3B,CAAC;AA6GF,wBAAgB,sBAAsB,CAAC,KAAK,EAAE,eAAe,GAAG,eAAe,EAAE,CAsDhF;AAED,wBAAgB,uBAAuB,CAAC,UAAU,EAAE,eAAe,EAAE,GAAG,MAAM,GAAG,SAAS,CAIzF"}
@@ -0,0 +1,156 @@
1
+ function stableStringify(value) {
2
+ if (typeof value === 'string')
3
+ return value;
4
+ try {
5
+ return JSON.stringify(value);
6
+ }
7
+ catch {
8
+ return String(value);
9
+ }
10
+ }
11
+ function parseMaybeJson(text) {
12
+ try {
13
+ return JSON.parse(text);
14
+ }
15
+ catch {
16
+ return text;
17
+ }
18
+ }
19
+ function extractPayload(result, error) {
20
+ const values = [];
21
+ const textParts = [];
22
+ if (error) {
23
+ values.push(error);
24
+ textParts.push(error);
25
+ }
26
+ if (result !== undefined) {
27
+ values.push(result);
28
+ textParts.push(stableStringify(result));
29
+ }
30
+ if (result && typeof result === 'object') {
31
+ const content = result.content;
32
+ if (Array.isArray(content)) {
33
+ for (const part of content) {
34
+ if (!part || typeof part !== 'object')
35
+ continue;
36
+ const record = part;
37
+ const text = typeof record.text === 'string' ? record.text : undefined;
38
+ if (text !== undefined) {
39
+ values.push(parseMaybeJson(text));
40
+ textParts.push(text);
41
+ }
42
+ for (const key of ['json', 'data', 'resource']) {
43
+ if (record[key] !== undefined) {
44
+ values.push(record[key]);
45
+ textParts.push(stableStringify(record[key]));
46
+ }
47
+ }
48
+ }
49
+ }
50
+ }
51
+ return { values, text: textParts.join('\n') };
52
+ }
53
+ function objectValues(value) {
54
+ if (Array.isArray(value))
55
+ return value.flatMap(objectValues);
56
+ if (!value || typeof value !== 'object')
57
+ return [];
58
+ const record = value;
59
+ return [record, ...Object.values(record).flatMap(objectValues)];
60
+ }
61
+ function hasField(value, field) {
62
+ if (!value || typeof value !== 'object')
63
+ return false;
64
+ if (Array.isArray(value))
65
+ return value.some((entry) => hasField(entry, field));
66
+ const record = value;
67
+ if (Object.prototype.hasOwnProperty.call(record, field))
68
+ return true;
69
+ return Object.values(record).some((entry) => hasField(entry, field));
70
+ }
71
+ function findNumberField(value, field) {
72
+ for (const candidate of objectValues(value)) {
73
+ const record = candidate;
74
+ const found = record[field];
75
+ if (typeof found === 'number')
76
+ return found;
77
+ if (typeof found === 'string' && found.trim() !== '' && Number.isFinite(Number(found))) {
78
+ return Number(found);
79
+ }
80
+ }
81
+ return undefined;
82
+ }
83
+ function findRowsLength(value) {
84
+ for (const candidate of objectValues(value)) {
85
+ const record = candidate;
86
+ const rows = record.rows ?? record.data ?? record.items ?? record.records;
87
+ if (Array.isArray(rows))
88
+ return rows.length;
89
+ }
90
+ return undefined;
91
+ }
92
+ function includesText(haystack, needle) {
93
+ return haystack.toLowerCase().includes(needle.toLowerCase());
94
+ }
95
+ function pass(name, message) {
96
+ return { name, status: 'pass', message };
97
+ }
98
+ function fail(name, message) {
99
+ return { name, status: 'fail', message };
100
+ }
101
+ export function evaluateToolAssertions(input) {
102
+ const { expect } = input;
103
+ if (!expect)
104
+ return [];
105
+ const payload = extractPayload(input.result, input.error);
106
+ const assertions = [];
107
+ if (expect.status) {
108
+ assertions.push(input.actualStatus === expect.status
109
+ ? pass('status', `Tool status matched expected ${expect.status}`)
110
+ : fail('status', `Expected tool status ${expect.status}, got ${input.actualStatus}`));
111
+ }
112
+ for (const field of expect.requiredFields ?? []) {
113
+ const found = payload.values.some((value) => hasField(value, field));
114
+ assertions.push(found
115
+ ? pass(`requiredFields.${field}`, `Found required field "${field}"`)
116
+ : fail(`requiredFields.${field}`, `Missing required field "${field}"`));
117
+ }
118
+ if (expect.maxRows !== undefined) {
119
+ const rowCount = payload.values
120
+ .map((value) => findNumberField(value, 'rowCount') ?? findNumberField(value, 'rowsReturned') ?? findRowsLength(value))
121
+ .find((value) => value !== undefined);
122
+ if (rowCount === undefined) {
123
+ assertions.push(fail('maxRows', 'Could not determine row count from result metadata'));
124
+ }
125
+ else {
126
+ assertions.push(rowCount <= expect.maxRows
127
+ ? pass('maxRows', `Row count ${rowCount} is within maxRows ${expect.maxRows}`)
128
+ : fail('maxRows', `Row count ${rowCount} exceeds maxRows ${expect.maxRows}`));
129
+ }
130
+ }
131
+ if (expect.errorCode) {
132
+ const found = payload.values.some((value) => hasField(value, 'code') && includesText(stableStringify(value), expect.errorCode))
133
+ || includesText(payload.text, expect.errorCode);
134
+ assertions.push(found
135
+ ? pass('errorCode', `Found expected error code ${expect.errorCode}`)
136
+ : fail('errorCode', `Missing expected error code ${expect.errorCode}`));
137
+ }
138
+ for (const expectedText of expect.contains ?? []) {
139
+ assertions.push(includesText(payload.text, expectedText)
140
+ ? pass(`contains.${expectedText}`, `Output contains "${expectedText}"`)
141
+ : fail(`contains.${expectedText}`, `Output does not contain "${expectedText}"`));
142
+ }
143
+ for (const forbiddenText of expect.notContains ?? []) {
144
+ assertions.push(!includesText(payload.text, forbiddenText)
145
+ ? pass(`notContains.${forbiddenText}`, `Output does not contain "${forbiddenText}"`)
146
+ : fail(`notContains.${forbiddenText}`, `Output leaked forbidden text "${forbiddenText}"`));
147
+ }
148
+ return assertions;
149
+ }
150
+ export function assertionFailureMessage(assertions) {
151
+ const failures = assertions.filter((assertion) => assertion.status === 'fail');
152
+ if (failures.length === 0)
153
+ return undefined;
154
+ return failures.map((assertion) => assertion.message).join('; ');
155
+ }
156
+ //# sourceMappingURL=assertions.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"assertions.js","sourceRoot":"","sources":["../src/assertions.ts"],"names":[],"mappings":"AAcA,SAAS,eAAe,CAAC,KAAc;IACrC,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,KAAK,CAAC;IAC5C,IAAI,CAAC;QACH,OAAO,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC;IAC/B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC;IACvB,CAAC;AACH,CAAC;AAED,SAAS,cAAc,CAAC,IAAY;IAClC,IAAI,CAAC;QACH,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAC1B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;IACd,CAAC;AACH,CAAC;AAED,SAAS,cAAc,CAAC,MAAe,EAAE,KAAc;IACrD,MAAM,MAAM,GAAc,EAAE,CAAC;IAC7B,MAAM,SAAS,GAAa,EAAE,CAAC;IAC/B,IAAI,KAAK,EAAE,CAAC;QACV,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACnB,SAAS,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IACxB,CAAC;IAED,IAAI,MAAM,KAAK,SAAS,EAAE,CAAC;QACzB,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACpB,SAAS,CAAC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,CAAC,CAAC;IAC1C,CAAC;IAED,IAAI,MAAM,IAAI,OAAO,MAAM,KAAK,QAAQ,EAAE,CAAC;QACzC,MAAM,OAAO,GAAI,MAAgC,CAAC,OAAO,CAAC;QAC1D,IAAI,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;YAC3B,KAAK,MAAM,IAAI,IAAI,OAAO,EAAE,CAAC;gBAC3B,IAAI,CAAC,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ;oBAAE,SAAS;gBAChD,MAAM,MAAM,GAAG,IAA+B,CAAC;gBAC/C,MAAM,IAAI,GAAG,OAAO,MAAM,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;gBACvE,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;oBACvB,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,CAAC,CAAC;oBAClC,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;gBACvB,CAAC;gBACD,KAAK,MAAM,GAAG,IAAI,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,CAAC,EAAE,CAAC;oBAC/C,IAAI,MAAM,CAAC,GAAG,CAAC,KAAK,SAAS,EAAE,CAAC;wBAC9B,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;wBACzB,SAAS,CAAC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;oBAC/C,CAAC;gBACH,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;IAED,OAAO,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;AAChD,CAAC;AAED,SAAS,YAAY,CAAC,KAAc;IAClC,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,OAAO,KAAK,CAAC,OAAO,CAAC,YAAY,CAAC,CAAC;IAC7D,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,EAAE,CAAC;IACnD,MAAM,MAAM,GAAG,KAAgC,CAAC;IAChD,OAAO,CAAC,MAAM,EAAE,GAAG,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,OAAO,CAAC,YAAY,CAAC,CAAC,CAAC;AAClE,CAAC;AAED,SAAS,QAAQ,CAAC,KAAc,EAAE,KAAa;IAC7C,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,KAAK,CAAC;IACtD,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,OAAO,KAAK,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC;IAC/E,MAAM,MAAM,GAAG,KAAgC,CAAC;IAChD,IAAI,MAAM,CAAC,SAAS,CAAC,cAAc,CAAC,IAAI,CAAC,MAAM,EAAE,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC;IACrE,OAAO,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC;AACvE,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,KAAa;IACpD,KAAK,MAAM,SAAS,IAAI,YAAY,CAAC,KAAK,CAAC,EAAE,CAAC;QAC5C,MAAM,MAAM,GAAG,SAAoC,CAAC;QACpD,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;QAC5B,IAAI,OAAO,KAAK,KAAK,QAAQ;YAAE,OAAO,KAAK,CAAC;QAC5C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC;YACvF,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC;QACvB,CAAC;IACH,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED,SAAS,cAAc,CAAC,KAAc;IACpC,KAAK,MAAM,SAAS,IAAI,YAAY,CAAC,KAAK,CAAC,EAAE,CAAC;QAC5C,MAAM,MAAM,GAAG,SAAoC,CAAC;QACpD,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,MAAM,CAAC,IAAI,IAAI,MAAM,CAAC,KAAK,IAAI,MAAM,CAAC,OAAO,CAAC;QAC1E,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;YAAE,OAAO,IAAI,CAAC,MAAM,CAAC;IAC9C,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED,SAAS,YAAY,CAAC,QAAgB,EAAE,MAAc;IACpD,OAAO,QAAQ,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC;AAC/D,CAAC;AAED,SAAS,IAAI,CAAC,IAAY,EAAE,OAAe;IACzC,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC;AAC3C,CAAC;AAED,SAAS,IAAI,CAAC,IAAY,EAAE,OAAe;IACzC,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC;AAC3C,CAAC;AAED,MAAM,UAAU,sBAAsB,CAAC,KAAsB;IAC3D,MAAM,EAAE,MAAM,EAAE,GAAG,KAAK,CAAC;IACzB,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IAEvB,MAAM,OAAO,GAAG,cAAc,CAAC,KAAK,CAAC,MAAM,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC;IAC1D,MAAM,UAAU,GAAsB,EAAE,CAAC;IAEzC,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;QAClB,UAAU,CAAC,IAAI,CAAC,KAAK,CAAC,YAAY,KAAK,MAAM,CAAC,MAAM;YAClD,CAAC,CAAC,IAAI,CAAC,QAAQ,EAAE,gCAAgC,MAAM,CAAC,MAAM,EAAE,CAAC;YACjE,CAAC,CAAC,IAAI,CAAC,QAAQ,EAAE,wBAAwB,MAAM,CAAC,MAAM,SAAS,KAAK,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC;IAC1F,CAAC;IAED,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,cAAc,IAAI,EAAE,EAAE,CAAC;QAChD,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC;QACrE,UAAU,CAAC,IAAI,CAAC,KAAK;YACnB,CAAC,CAAC,IAAI,CAAC,kBAAkB,KAAK,EAAE,EAAE,yBAAyB,KAAK,GAAG,CAAC;YACpE,CAAC,CAAC,IAAI,CAAC,kBAAkB,KAAK,EAAE,EAAE,2BAA2B,KAAK,GAAG,CAAC,CAAC,CAAC;IAC5E,CAAC;IAED,IAAI,MAAM,CAAC,OAAO,KAAK,SAAS,EAAE,CAAC;QACjC,MAAM,QAAQ,GAAG,OAAO,CAAC,MAAM;aAC5B,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,eAAe,CAAC,KAAK,EAAE,UAAU,CAAC,IAAI,eAAe,CAAC,KAAK,EAAE,cAAc,CAAC,IAAI,cAAc,CAAC,KAAK,CAAC,CAAC;aACrH,IAAI,CAAC,CAAC,KAAK,EAAmB,EAAE,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC;QACzD,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAC3B,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,oDAAoD,CAAC,CAAC,CAAC;QACzF,CAAC;aAAM,CAAC;YACN,UAAU,CAAC,IAAI,CAAC,QAAQ,IAAI,MAAM,CAAC,OAAO;gBACxC,CAAC,CAAC,IAAI,CAAC,SAAS,EAAE,aAAa,QAAQ,sBAAsB,MAAM,CAAC,OAAO,EAAE,CAAC;gBAC9E,CAAC,CAAC,IAAI,CAAC,SAAS,EAAE,aAAa,QAAQ,oBAAoB,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC;QAClF,CAAC;IACH,CAAC;IAED,IAAI,MAAM,CAAC,SAAS,EAAE,CAAC;QACrB,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC,IAAI,YAAY,CAAC,eAAe,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC,SAAU,CAAC,CAAC;eAC3H,YAAY,CAAC,OAAO,CAAC,IAAI,EAAE,MAAM,CAAC,SAAS,CAAC,CAAC;QAClD,UAAU,CAAC,IAAI,CAAC,KAAK;YACnB,CAAC,CAAC,IAAI,CAAC,WAAW,EAAE,6BAA6B,MAAM,CAAC,SAAS,EAAE,CAAC;YACpE,CAAC,CAAC,IAAI,CAAC,WAAW,EAAE,+BAA+B,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;IAC5E,CAAC;IAED,KAAK,MAAM,YAAY,IAAI,MAAM,CAAC,QAAQ,IAAI,EAAE,EAAE,CAAC;QACjD,UAAU,CAAC,IAAI,CAAC,YAAY,CAAC,OAAO,CAAC,IAAI,EAAE,YAAY,CAAC;YACtD,CAAC,CAAC,IAAI,CAAC,YAAY,YAAY,EAAE,EAAE,oBAAoB,YAAY,GAAG,CAAC;YACvE,CAAC,CAAC,IAAI,CAAC,YAAY,YAAY,EAAE,EAAE,4BAA4B,YAAY,GAAG,CAAC,CAAC,CAAC;IACrF,CAAC;IAED,KAAK,MAAM,aAAa,IAAI,MAAM,CAAC,WAAW,IAAI,EAAE,EAAE,CAAC;QACrD,UAAU,CAAC,IAAI,CAAC,CAAC,YAAY,CAAC,OAAO,CAAC,IAAI,EAAE,aAAa,CAAC;YACxD,CAAC,CAAC,IAAI,CAAC,eAAe,aAAa,EAAE,EAAE,4BAA4B,aAAa,GAAG,CAAC;YACpF,CAAC,CAAC,IAAI,CAAC,eAAe,aAAa,EAAE,EAAE,iCAAiC,aAAa,GAAG,CAAC,CAAC,CAAC;IAC/F,CAAC;IAED,OAAO,UAAU,CAAC;AACpB,CAAC;AAED,MAAM,UAAU,uBAAuB,CAAC,UAA6B;IACnE,MAAM,QAAQ,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC;IAC/E,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,SAAS,CAAC;IAC5C,OAAO,QAAQ,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACnE,CAAC"}
@@ -1 +1 @@
1
- {"version":3,"file":"checker.d.ts","sourceRoot":"","sources":["../src/checker.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAa,YAAY,EAAE,WAAW,EAAe,cAAc,EAAe,aAAa,EAAE,MAAM,YAAY,CAAC;AAQhI,wBAAgB,aAAa,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,CAAC,EAAE,aAAa,GAAG,cAAc,CAavF;AAiDD,wBAAsB,cAAc,CAAC,OAAO,EAAE,YAAY,GAAG,OAAO,CAAC,WAAW,CAAC,CA6HhF"}
1
+ {"version":3,"file":"checker.d.ts","sourceRoot":"","sources":["../src/checker.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAa,YAAY,EAAE,WAAW,EAAe,cAAc,EAAe,aAAa,EAAE,MAAM,YAAY,CAAC;AAQhI,wBAAgB,aAAa,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,CAAC,EAAE,aAAa,GAAG,cAAc,CAavF;AA4ED,wBAAsB,cAAc,CAAC,OAAO,EAAE,YAAY,GAAG,OAAO,CAAC,WAAW,CAAC,CAiIhF"}