@emseepea/testing 0.20.5 → 0.21.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +103 -0
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/mcp-cli-qualification.d.ts +85 -0
- package/dist/mcp-cli-qualification.d.ts.map +1 -0
- package/dist/mcp-cli-qualification.js +507 -0
- package/dist/mcp-cli-qualification.js.map +1 -0
- package/mcp-cli/cli.mjs +37 -0
- package/package.json +3 -1
- package/semantic/cli.mjs +12 -6
- package/semantic/provider.mjs +230 -12
- package/semantic/test.d.mts +2 -0
- package/semantic/test.mjs +27 -4
package/README.md
CHANGED
|
@@ -57,6 +57,29 @@ advertised-tool text, an answer wrapper, or prepared MCP material. Exact tool
|
|
|
57
57
|
assertions come from the provider's native MCP events. Follow-up messages use
|
|
58
58
|
the same conversation.
|
|
59
59
|
|
|
60
|
+
Call `await conversation.fresh()` to discard provider transcript history while
|
|
61
|
+
keeping the same running test application and its saved data. The next `send`
|
|
62
|
+
starts a new provider-native session in each trial.
|
|
63
|
+
|
|
64
|
+
The runner supports `claude-local`, `claude-ci`, `codex-local`, and `codex-ci`.
|
|
65
|
+
Codex providers require an explicit `EMSEEPEA_CODEX_MODEL`. For example:
|
|
66
|
+
|
|
67
|
+
```sh
|
|
68
|
+
EMSEEPEA_CODEX_MODEL=gpt-5.5 emseepea-test --provider codex-local eval
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
`codex-local` uses the existing Codex CLI sign-in without copying it.
|
|
72
|
+
`codex-ci` requires `OPENAI_API_KEY` from the CI secret store. Both providers
|
|
73
|
+
run `codex exec --json`, resume the actual Codex session for follow-ups, ignore
|
|
74
|
+
user configuration, and pre-approve only the discovered tools from the test
|
|
75
|
+
server.
|
|
76
|
+
|
|
77
|
+
An isolated command policy denies shell launches and spawned shell environments
|
|
78
|
+
inherit no provider credentials.
|
|
79
|
+
|
|
80
|
+
Codex CLI evidence proves that CLI journey. It does not prove native ChatGPT
|
|
81
|
+
connection, attachment, retrieval, or MCP App rendering behavior.
|
|
82
|
+
|
|
60
83
|
Optional `context` is application context, not test guidance. Use it only when
|
|
61
84
|
the deployed application supplies the same context. Leaving it out is the best
|
|
62
85
|
default for testing whether tool names, descriptions, and schemas stand on
|
|
@@ -67,6 +90,86 @@ never production. Resources and prompts need deterministic protocol tests;
|
|
|
67
90
|
this library does not pretend that manually injecting their content proves a
|
|
68
91
|
native user journey.
|
|
69
92
|
|
|
93
|
+
## Qualify an MCP journey from the command line
|
|
94
|
+
|
|
95
|
+
Use `emseepea-qualify` for repeatable protocol checks against an explicitly
|
|
96
|
+
configured MCP endpoint. The command can verify OAuth protected-resource
|
|
97
|
+
metadata, tool and resource catalogues, resource templates, exact authorized
|
|
98
|
+
reads, original byte hashes, tool-returned resource links, progress,
|
|
99
|
+
completion, client cancellation, and access denial.
|
|
100
|
+
|
|
101
|
+
Keep the endpoint, fixture values, expected results, and token environment
|
|
102
|
+
names in an adopter-owned scenario module. Scenario modules are trusted
|
|
103
|
+
application code: importing one executes it with the command process's full
|
|
104
|
+
authority, including access to that process's environment.
|
|
105
|
+
|
|
106
|
+
```js
|
|
107
|
+
import { defineMcpCliQualification } from "@emseepea/testing";
|
|
108
|
+
|
|
109
|
+
export default defineMcpCliQualification({
|
|
110
|
+
name: "synthetic document journey",
|
|
111
|
+
endpoint: new URL(process.env.TEST_MCP_URL),
|
|
112
|
+
tokenEnvironment: "TEST_MCP_TOKEN",
|
|
113
|
+
checkpoints: [
|
|
114
|
+
{ id: "tools", operation: "tools/list", expectedNames: ["get-document"] },
|
|
115
|
+
{ id: "resources", operation: "resources/list", expectedUris: ["fixture://documents/example.pdf"] },
|
|
116
|
+
{
|
|
117
|
+
id: "original",
|
|
118
|
+
operation: "resources/read",
|
|
119
|
+
uri: "fixture://documents/example.pdf",
|
|
120
|
+
expectedContents: [{
|
|
121
|
+
mimeType: "application/pdf",
|
|
122
|
+
bytes: 1240,
|
|
123
|
+
sha256: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
|
124
|
+
}],
|
|
125
|
+
},
|
|
126
|
+
{
|
|
127
|
+
id: "limited-user-denied",
|
|
128
|
+
operation: "resources/read",
|
|
129
|
+
uri: "fixture://documents/example.pdf",
|
|
130
|
+
tokenEnvironment: "TEST_LIMITED_MCP_TOKEN",
|
|
131
|
+
expect: "denied",
|
|
132
|
+
},
|
|
133
|
+
],
|
|
134
|
+
});
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Expose the command through a project-local package script:
|
|
138
|
+
|
|
139
|
+
```json
|
|
140
|
+
{
|
|
141
|
+
"scripts": {
|
|
142
|
+
"qualify:mcp": "emseepea-qualify --scenario test/mcp-qualification.mjs --output artifacts/mcp-cli/evidence.json"
|
|
143
|
+
}
|
|
144
|
+
}
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Run the script with:
|
|
148
|
+
|
|
149
|
+
```sh
|
|
150
|
+
npm run qualify:mcp
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
The framework reads configured tokens from the named environment variables. It
|
|
154
|
+
does not accept them as command arguments or include values it handles in its
|
|
155
|
+
evidence. Your trusted scenario code still has normal process access and is
|
|
156
|
+
responsible for anything it reads or writes.
|
|
157
|
+
|
|
158
|
+
Evidence contains bounded counts, hashes, MIME types, categorical outcomes,
|
|
159
|
+
and checkpoint status. It is written with mode `0600` where supported. Use only
|
|
160
|
+
synthetic or explicitly approved fixture data in scenario names and
|
|
161
|
+
expectations.
|
|
162
|
+
|
|
163
|
+
Each checkpoint records `passed`, `failed`, `blocked`, or `incomplete`.
|
|
164
|
+
`record` checkpoints can preserve a known missing fixture or unavailable
|
|
165
|
+
authority without turning it into success. The report is always labelled
|
|
166
|
+
`evidenceKind: "mcp-cli"` and `nativeChatgpt: "not-tested"`.
|
|
167
|
+
|
|
168
|
+
This command does not drive ChatGPT. It cannot prove ChatGPT connection or
|
|
169
|
+
consent screens, attachments, fresh-conversation behavior, Sources previews,
|
|
170
|
+
visible images, generated links, MCP App cards, viewport interactions, or
|
|
171
|
+
native streaming and cancellation presentation.
|
|
172
|
+
|
|
70
173
|
## Scripted Confirmation Input
|
|
71
174
|
|
|
72
175
|
For tools that request form elicitation, supply an explicit simulated human
|
package/dist/index.d.ts
CHANGED
|
@@ -3,6 +3,7 @@ import { serveEmseepea, type AuthenticationOptions } from "@emseepea/server";
|
|
|
3
3
|
export { createMcpAppHostSimulator, type McpAppHostSimulator, type McpAppHostSimulatorOptions, type McpAppMessageRequest, } from "./mcp-app-host.js";
|
|
4
4
|
export { assertPublishedMcpContractCompatible, comparePublishedMcpContracts, extractPublishedMcpContract, writePublishedMcpContractBaseline, type PublishedMcpContract, type PublishedMcpContractBaseline, type PublishedMcpContractBaselineInput, type PublishedMcpContractBreak, type PublishedMcpContractCheckPolicy, type PublishedMcpContractCheckPolicyInput, type PublishedMcpContractClient, type PublishedMcpResource, } from "./published-contract.js";
|
|
5
5
|
export { assertWcagContrast, wcagContrastRatio } from "./contrast.js";
|
|
6
|
+
export { defineMcpCliQualification, runMcpCliQualification, type McpCliCheckpoint, type McpCliCheckpointEvidence, type McpCliContentExpectation, type McpCliQualificationEvidence, type McpCliQualificationScenario, type McpCliQualificationStatus, type McpCliResourceLinkExpectation, } from "./mcp-cli-qualification.js";
|
|
6
7
|
export type SupportedProtocolVersion = "2026-07-28" | "2025-11-25" | "2025-06-18" | "2025-03-26" | "2024-11-05" | "2024-10-07";
|
|
7
8
|
export interface TestCleanup {
|
|
8
9
|
after(cleanup: () => Promise<void>): void;
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,MAAM,EAAiC,MAAM,8BAA8B,CAAC;AACrF,OAAO,EAAE,aAAa,EAAE,KAAK,qBAAqB,EAAE,MAAM,kBAAkB,CAAC;AAE7E,OAAO,EACL,yBAAyB,EACzB,KAAK,mBAAmB,EACxB,KAAK,0BAA0B,EAC/B,KAAK,oBAAoB,GAC1B,MAAM,mBAAmB,CAAC;AAC3B,OAAO,EACL,oCAAoC,EACpC,4BAA4B,EAC5B,2BAA2B,EAC3B,iCAAiC,EACjC,KAAK,oBAAoB,EACzB,KAAK,4BAA4B,EACjC,KAAK,iCAAiC,EACtC,KAAK,yBAAyB,EAC9B,KAAK,+BAA+B,EACpC,KAAK,oCAAoC,EACzC,KAAK,0BAA0B,EAC/B,KAAK,oBAAoB,GAC1B,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,kBAAkB,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,MAAM,EAAiC,MAAM,8BAA8B,CAAC;AACrF,OAAO,EAAE,aAAa,EAAE,KAAK,qBAAqB,EAAE,MAAM,kBAAkB,CAAC;AAE7E,OAAO,EACL,yBAAyB,EACzB,KAAK,mBAAmB,EACxB,KAAK,0BAA0B,EAC/B,KAAK,oBAAoB,GAC1B,MAAM,mBAAmB,CAAC;AAC3B,OAAO,EACL,oCAAoC,EACpC,4BAA4B,EAC5B,2BAA2B,EAC3B,iCAAiC,EACjC,KAAK,oBAAoB,EACzB,KAAK,4BAA4B,EACjC,KAAK,iCAAiC,EACtC,KAAK,yBAAyB,EAC9B,KAAK,+BAA+B,EACpC,KAAK,oCAAoC,EACzC,KAAK,0BAA0B,EAC/B,KAAK,oBAAoB,GAC1B,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,kBAAkB,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;AACtE,OAAO,EACL,yBAAyB,EACzB,sBAAsB,EACtB,KAAK,gBAAgB,EACrB,KAAK,wBAAwB,EAC7B,KAAK,wBAAwB,EAC7B,KAAK,2BAA2B,EAChC,KAAK,2BAA2B,EAChC,KAAK,yBAAyB,EAC9B,KAAK,6BAA6B,GACnC,MAAM,4BAA4B,CAAC;AAEpC,MAAM,MAAM,wBAAwB,GAChC,YAAY,GACZ,YAAY,GACZ,YAAY,GACZ,YAAY,GACZ,YAAY,GACZ,YAAY,CAAC;AAEjB,MAAM,WAAW,WAAW;IAC1B,KAAK,CAAC,OAAO,EAAE,MAAM,OAAO,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC;CAC3C;AAED,MAAM,WAAW,qBAAqB;IACpC,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,WAAW,CAAC,EAAE,MAAM,CAAC,UAAU,CAAC;IAChC,eAAe,CAAC,EAAE,wBAAwB,CAAC;IAC3C,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,gBAAgB;IAC/B,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;IACvB,OAAO,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC;IACzC,MAAM,IAAI,QAAQ,CAAC;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACvD,GAAG,EAAE,GAAG,CAAC;CACV;AAED,6DAA6D;AAC7D,wBAAgB,0BAA0B,CACxC,WAAW,EAAE,SAAS,MAAM,EAAE,EAC9B,SAAS,GAAE,QAAQ,GAAG,WAAyB,GAC9C,qBAAqB,CAyBvB;AAED,KAAK,oBAAoB,GAAG,IAAI,CAAC,qBAAqB,EAAE,YAAY,GAAG,iBAAiB,GAAG,OAAO,CAAC,CAAC;AACpG,KAAK,WAAW,GAAG,UAAU,CAAC,OAAO,aAAa,CAAC,CAAC,CAAC,CAAC,CAAC;AAEvD,wBAAgB,aAAa,CAC3B,GAAG,EAAE,WAAW,EAChB,OAAO,CAAC,EAAE,oBAAoB,GAC7B,OAAO,CAAC,gBAAgB,CAAC,CAAC;AAC7B,wBAAgB,aAAa,CAC3B,IAAI,EAAE,WAAW,EACjB,GAAG,EAAE,WAAW,EAChB,OAAO,CAAC,EAAE,oBAAoB,GAC7B,OAAO,CAAC,gBAAgB,CAAC,CAAC;AA4B7B,wBAAgB,cAAc,CAC5B,SAAS,EAAE,GAAG,EACd,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,CAAC,CAAC;AAC7B,wBAAgB,cAAc,CAC5B,IAAI,EAAE,WAAW,EACjB,SAAS,EAAE,GAAG,EACd,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,CAAC,CAAC"}
|
package/dist/index.js
CHANGED
|
@@ -5,6 +5,7 @@ import { serveEmseepea } from "@emseepea/server";
|
|
|
5
5
|
export { createMcpAppHostSimulator, } from "./mcp-app-host.js";
|
|
6
6
|
export { assertPublishedMcpContractCompatible, comparePublishedMcpContracts, extractPublishedMcpContract, writePublishedMcpContractBaseline, } from "./published-contract.js";
|
|
7
7
|
export { assertWcagContrast, wcagContrastRatio } from "./contrast.js";
|
|
8
|
+
export { defineMcpCliQualification, runMcpCliQualification, } from "./mcp-cli-qualification.js";
|
|
8
9
|
/** Test-only verifier. Never use it in a deployed server. */
|
|
9
10
|
export function insecureTestAuthentication(permissions, discovery = "protected") {
|
|
10
11
|
const resourceServerUrl = new URL("https://test.example/mcp");
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AAEzC,OAAO,EAAE,MAAM,EAAE,6BAA6B,EAAE,MAAM,8BAA8B,CAAC;AACrF,OAAO,EAAE,aAAa,EAA8B,MAAM,kBAAkB,CAAC;AAE7E,OAAO,EACL,yBAAyB,GAI1B,MAAM,mBAAmB,CAAC;AAC3B,OAAO,EACL,oCAAoC,EACpC,4BAA4B,EAC5B,2BAA2B,EAC3B,iCAAiC,GASlC,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,kBAAkB,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AAEzC,OAAO,EAAE,MAAM,EAAE,6BAA6B,EAAE,MAAM,8BAA8B,CAAC;AACrF,OAAO,EAAE,aAAa,EAA8B,MAAM,kBAAkB,CAAC;AAE7E,OAAO,EACL,yBAAyB,GAI1B,MAAM,mBAAmB,CAAC;AAC3B,OAAO,EACL,oCAAoC,EACpC,4BAA4B,EAC5B,2BAA2B,EAC3B,iCAAiC,GASlC,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,kBAAkB,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;AACtE,OAAO,EACL,yBAAyB,EACzB,sBAAsB,GAQvB,MAAM,4BAA4B,CAAC;AA6BpC,6DAA6D;AAC7D,MAAM,UAAU,0BAA0B,CACxC,WAA8B,EAC9B,YAAoC,WAAW;IAE/C,MAAM,iBAAiB,GAAG,IAAI,GAAG,CAAC,0BAA0B,CAAC,CAAC;IAC9D,OAAO;QACL,SAAS;QACT,QAAQ,EAAE;YACR,KAAK,CAAC,iBAAiB,CAAC,KAAK;gBAC3B,OAAO;oBACL,KAAK;oBACL,QAAQ,EAAE,sBAAsB;oBAChC,MAAM,EAAE,CAAC,GAAG,WAAW,CAAC;oBACxB,SAAS,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,GAAG,EAAE;oBAC9C,QAAQ,EAAE,iBAAiB;iBAC5B,CAAC;YACJ,CAAC;SACF;QACD,QAAQ,EAAE;YACR,iBAAiB;YACjB,aAAa,EAAE;gBACb,MAAM,EAAE,2BAA2B;gBACnC,sBAAsB,EAAE,qCAAqC;gBAC7D,cAAc,EAAE,iCAAiC;gBACjD,wBAAwB,EAAE,CAAC,MAAM,CAAC;aACnC;SACF;KACF,CAAC;AACJ,CAAC;AAcD,MAAM,CAAC,KAAK,UAAU,aAAa,CACjC,SAAoC,EACpC,eAAmD,EAAE,EACrD,kBAAwC,EAAE;IAE1C,MAAM,YAAY,GAAG,OAAQ,YAAqC,CAAC,MAAM,KAAK,UAAU,CAAC;IACzF,MAAM,IAAI,GAAG,YAAY,CAAC,CAAC,CAAC,SAAwB,CAAC,CAAC,CAAC,SAAS,CAAC;IACjE,MAAM,GAAG,GAAG,YAAY,CAAC,CAAC,CAAC,YAA2B,CAAC,CAAC,CAAC,SAAwB,CAAC;IAClF,MAAM,OAAO,GAAG,YAAY,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,YAAoC,CAAC;IACtF,MAAM,OAAO,GAAG,MAAM,aAAa,CAAC,GAAG,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC,CAAC;IACtD,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,MAAM,KAAK,GAAG,WAAW,CAAC,OAAO,EAAE,OAAO,CAAC,KAAK,CAAC,CAAC;IAClD,IAAI,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC;IACnB,OAAO;QACL,GAAG,OAAO;QACV,KAAK;QACL,MAAM,EAAE,GAAG,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,CAAC;QACvD,OAAO,EAAE,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,OAAO,CACzC,OAAO,CAAC,GAAG,EACX,OAAO,EACP,OAAO,CAAC,UAAU,EAClB,KAAK,EACL,OAAO,CAAC,eAAe,CACxB;KACF,CAAC;AACJ,CAAC;AAWD,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,eAAkC,EAClC,qBAAkD,EAAE,EACpD,kBAAyC,EAAE;IAE3C,MAAM,YAAY,GAAG,CAAC,CAAC,eAAe,YAAY,GAAG,CAAC,CAAC;IACvD,MAAM,IAAI,GAAG,YAAY,CAAC,CAAC,CAAC,eAA8B,CAAC,CAAC,CAAC,SAAS,CAAC;IACvE,MAAM,SAAS,GAAG,YAAY,CAAC,CAAC,CAAC,kBAAyB,CAAC,CAAC,CAAC,eAAe,CAAC;IAC7E,MAAM,OAAO,GAAG,YAAY,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,kBAA2C,CAAC;IAC7F,MAAM,KAAK,GAAG,KAAK,CAAC,OAAO,CAAC,QAAQ,EAAE,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC,EAAE;QAChE,GAAG,EAAE;YACH,GAAG,OAAO,CAAC,GAAG;YACd,QAAQ,EAAE,MAAM;YAChB,IAAI,EAAE,GAAG;YACT,GAAG,OAAO,CAAC,WAAW;SACvB;QACD,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;KAClC,CAAC,CAAC;IACH,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,MAAM,GAAG,EAAE,CAAC;IAChB,IAAI,MAAM,GAAG,EAAE,CAAC;IAChB,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,KAAa,EAAE,EAAE,GAAG,MAAM,GAAG,GAAG,MAAM,GAAG,KAAK,EAAE,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC7F,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,KAAa,EAAE,EAAE,GAAG,MAAM,GAAG,GAAG,MAAM,GAAG,KAAK,EAAE,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAE7F,MAAM,GAAG,GAAG,MAAM,IAAI,OAAO,CAAM,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;QACrD,MAAM,MAAM,GAAG,CAAC,KAAa,EAAE,KAAW,EAAE,EAAE;YAC5C,YAAY,CAAC,KAAK,CAAC,CAAC;YACpB,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;YAClC,KAAK,CAAC,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;YAC3B,KAAK,CAAC,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;YAC1B,IAAI,KAAK;gBAAE,MAAM,CAAC,KAAK,CAAC,CAAC;iBACpB,IAAI,KAAK;gBAAE,OAAO,CAAC,KAAK,CAAC,CAAC;QACjC,CAAC,CAAC;QACF,MAAM,OAAO,GAAG,GAAG,EAAE;YACnB,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,gCAAgC,CAAC,CAAC;YAC7D,IAAI,KAAK,EAAE,CAAC,CAAC,CAAC;gBAAE,MAAM,CAAC,SAAS,EAAE,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QACvD,CAAC,CAAC;QACF,MAAM,MAAM,GAAG,CAAC,KAAY,EAAE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QAC/C,MAAM,MAAM,GAAG,CAAC,IAAmB,EAAE,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,qBAAqB,MAAM,CAAC,IAAI,CAAC,KAAK,MAAM,EAAE,CAAC,CAAC,CAAC;QAC1G,MAAM,KAAK,GAAG,UAAU,CACtB,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,iCAAiC,MAAM,EAAE,CAAC,CAAC,EAClE,OAAO,CAAC,gBAAgB,IAAI,MAAM,CACnC,CAAC;QACF,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QACjC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC5B,KAAK,CAAC,IAAI,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC3B,OAAO,EAAE,CAAC;IACZ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,EAAE,KAAc,EAAE,EAAE;QAChC,MAAM,WAAW,CAAC,KAAK,CAAC,CAAC;QACzB,MAAM,KAAK,CAAC;IACd,CAAC,CAAC,CAAC;IAEH,MAAM,KAAK,GAAG,WAAW,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,WAAW,CAAC,KAAK,CAAC,CAAC,CAAC;IAC7D,IAAI,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC;IAEnB,OAAO;QACL,KAAK;QACL,GAAG;QACH,MAAM,EAAE,GAAG,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC;QAC/D,OAAO,EAAE,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,OAAO,CACzC,GAAG,EACH,OAAO,EACP,OAAO,CAAC,UAAU,EAClB,KAAK,EACL,OAAO,CAAC,eAAe,CACxB;KACF,CAAC;AACJ,CAAC;AAED,KAAK,UAAU,OAAO,CACpB,GAAQ,EACR,OAAiB,EACjB,UAAU,GAAG,eAAe,EAC5B,KAAc,EACd,kBAA4C,YAAY;IAExD,MAAM,MAAM,GAAG,IAAI,MAAM,CACvB,EAAE,IAAI,EAAE,UAAU,EAAE,OAAO,EAAE,OAAO,EAAE,EACtC,eAAe,KAAK,YAAY;QAC9B,CAAC,CAAC,EAAE,kBAAkB,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE;QAC5D,CAAC,CAAC,EAAE,yBAAyB,EAAE,CAAC,eAAe,CAAC,EAAE,kBAAkB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,EAAE,CAC7F,CAAC;IACF,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IACrB,IAAI,CAAC;QACH,MAAM,MAAM,CAAC,OAAO,CAAC,IAAI,6BAA6B,CACpD,GAAG,EACH,KAAK,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,EAAE,KAAK,EAAE,KAAK,IAAI,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CACnE,CAAC,CAAC;QACH,OAAO,MAAM,CAAC;IAChB,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,MAAM,CAAC,KAAK,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;QAC5C,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC;AAED,SAAS,WAAW,CAAC,OAAiB,EAAE,IAAyB;IAC/D,IAAI,OAAkC,CAAC;IACvC,OAAO,GAAG,EAAE,CAAC,OAAO,KAAK,CAAC,KAAK,IAAI,EAAE;QACnC,MAAM,OAAO,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;QAClE,MAAM,IAAI,EAAE,CAAC;IACf,CAAC,CAAC,EAAE,CAAC;AACP,CAAC;AAED,KAAK,UAAU,WAAW,CAAC,KAA+B;IACxD,IAAI,KAAK,CAAC,QAAQ,KAAK,IAAI;QAAE,OAAO;IACpC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACtB,MAAM,OAAO,CAAC,IAAI,CAAC;QACjB,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;QACpE,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;KAC3D,CAAC,CAAC;IACH,IAAI,KAAK,CAAC,QAAQ,KAAK,IAAI;QAAE,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;AACrD,CAAC"}
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
export type McpCliQualificationStatus = "passed" | "failed" | "blocked" | "incomplete";
|
|
2
|
+
export interface McpCliContentExpectation {
|
|
3
|
+
readonly sha256: string;
|
|
4
|
+
readonly bytes?: number;
|
|
5
|
+
readonly mimeType?: string;
|
|
6
|
+
}
|
|
7
|
+
export interface McpCliResourceLinkExpectation {
|
|
8
|
+
readonly uri: string;
|
|
9
|
+
readonly mimeType?: string;
|
|
10
|
+
}
|
|
11
|
+
interface McpCliCheckpointBase {
|
|
12
|
+
readonly id: string;
|
|
13
|
+
readonly tokenEnvironment?: string;
|
|
14
|
+
}
|
|
15
|
+
export type McpCliCheckpoint = (McpCliCheckpointBase & {
|
|
16
|
+
readonly operation: "oauth/discover";
|
|
17
|
+
readonly expectedScopes?: readonly string[];
|
|
18
|
+
}) | (McpCliCheckpointBase & {
|
|
19
|
+
readonly operation: "tools/list";
|
|
20
|
+
readonly expectedNames?: readonly string[];
|
|
21
|
+
readonly expect?: "success" | "denied";
|
|
22
|
+
}) | (McpCliCheckpointBase & {
|
|
23
|
+
readonly operation: "resources/list";
|
|
24
|
+
readonly expectedUris?: readonly string[];
|
|
25
|
+
readonly expect?: "success" | "denied";
|
|
26
|
+
}) | (McpCliCheckpointBase & {
|
|
27
|
+
readonly operation: "resources/templates/list";
|
|
28
|
+
readonly expectedUriTemplates?: readonly string[];
|
|
29
|
+
readonly expect?: "success" | "denied";
|
|
30
|
+
}) | (McpCliCheckpointBase & {
|
|
31
|
+
readonly operation: "resources/read";
|
|
32
|
+
readonly uri: string;
|
|
33
|
+
readonly expectedContents?: readonly McpCliContentExpectation[];
|
|
34
|
+
readonly maxContentBytes?: number;
|
|
35
|
+
readonly expect?: "success" | "denied";
|
|
36
|
+
}) | (McpCliCheckpointBase & {
|
|
37
|
+
readonly operation: "tools/call";
|
|
38
|
+
readonly name: string;
|
|
39
|
+
readonly arguments?: Readonly<Record<string, unknown>>;
|
|
40
|
+
readonly expectedResourceLinks?: readonly McpCliResourceLinkExpectation[];
|
|
41
|
+
readonly minimumProgress?: number;
|
|
42
|
+
readonly cancelAfterMs?: number;
|
|
43
|
+
readonly expect?: "success" | "denied" | "cancelled";
|
|
44
|
+
}) | {
|
|
45
|
+
readonly id: string;
|
|
46
|
+
readonly operation: "record";
|
|
47
|
+
readonly outcome: "blocked" | "incomplete";
|
|
48
|
+
readonly category: string;
|
|
49
|
+
};
|
|
50
|
+
export interface McpCliQualificationScenario {
|
|
51
|
+
readonly name: string;
|
|
52
|
+
readonly endpoint: URL;
|
|
53
|
+
readonly protocolVersion?: "2026-07-28" | "2025-11-25" | "2025-06-18" | "2025-03-26";
|
|
54
|
+
readonly requestTimeoutMs?: number;
|
|
55
|
+
readonly tokenEnvironment?: string;
|
|
56
|
+
readonly checkpoints: readonly McpCliCheckpoint[];
|
|
57
|
+
}
|
|
58
|
+
export interface McpCliCheckpointEvidence {
|
|
59
|
+
readonly id: string;
|
|
60
|
+
readonly operation: McpCliCheckpoint["operation"];
|
|
61
|
+
readonly status: McpCliQualificationStatus;
|
|
62
|
+
readonly category: string;
|
|
63
|
+
readonly observation?: Readonly<Record<string, unknown>>;
|
|
64
|
+
}
|
|
65
|
+
export interface McpCliQualificationEvidence {
|
|
66
|
+
readonly schemaVersion: 1;
|
|
67
|
+
readonly evidenceKind: "mcp-cli";
|
|
68
|
+
readonly nativeChatgpt: "not-tested";
|
|
69
|
+
readonly name: string;
|
|
70
|
+
readonly endpointSha256: string;
|
|
71
|
+
readonly protocolVersion: string;
|
|
72
|
+
readonly startedAt: string;
|
|
73
|
+
readonly finishedAt: string;
|
|
74
|
+
readonly status: McpCliQualificationStatus;
|
|
75
|
+
readonly checkpoints: readonly McpCliCheckpointEvidence[];
|
|
76
|
+
}
|
|
77
|
+
/** Preserve TypeScript checking for an adopter-owned qualification scenario. */
|
|
78
|
+
export declare function defineMcpCliQualification(scenario: McpCliQualificationScenario): McpCliQualificationScenario;
|
|
79
|
+
/**
|
|
80
|
+
* Run deterministic MCP protocol checks and write bounded private evidence.
|
|
81
|
+
* This does not exercise or qualify native ChatGPT.
|
|
82
|
+
*/
|
|
83
|
+
export declare function runMcpCliQualification(scenario: McpCliQualificationScenario, output: string): Promise<McpCliQualificationEvidence>;
|
|
84
|
+
export {};
|
|
85
|
+
//# sourceMappingURL=mcp-cli-qualification.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"mcp-cli-qualification.d.ts","sourceRoot":"","sources":["../src/mcp-cli-qualification.ts"],"names":[],"mappings":"AAeA,MAAM,MAAM,yBAAyB,GAAG,QAAQ,GAAG,QAAQ,GAAG,SAAS,GAAG,YAAY,CAAC;AAEvF,MAAM,WAAW,wBAAwB;IACvC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;CAC5B;AAED,MAAM,WAAW,6BAA6B;IAC5C,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC;IACrB,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;CAC5B;AAED,UAAU,oBAAoB;IAC5B,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;CACpC;AAED,MAAM,MAAM,gBAAgB,GACxB,CAAC,oBAAoB,GAAG;IACtB,QAAQ,CAAC,SAAS,EAAE,gBAAgB,CAAC;IACrC,QAAQ,CAAC,cAAc,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;CAC7C,CAAC,GACF,CAAC,oBAAoB,GAAG;IACtB,QAAQ,CAAC,SAAS,EAAE,YAAY,CAAC;IACjC,QAAQ,CAAC,aAAa,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IAC3C,QAAQ,CAAC,MAAM,CAAC,EAAE,SAAS,GAAG,QAAQ,CAAC;CACxC,CAAC,GACF,CAAC,oBAAoB,GAAG;IACtB,QAAQ,CAAC,SAAS,EAAE,gBAAgB,CAAC;IACrC,QAAQ,CAAC,YAAY,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IAC1C,QAAQ,CAAC,MAAM,CAAC,EAAE,SAAS,GAAG,QAAQ,CAAC;CACxC,CAAC,GACF,CAAC,oBAAoB,GAAG;IACtB,QAAQ,CAAC,SAAS,EAAE,0BAA0B,CAAC;IAC/C,QAAQ,CAAC,oBAAoB,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IAClD,QAAQ,CAAC,MAAM,CAAC,EAAE,SAAS,GAAG,QAAQ,CAAC;CACxC,CAAC,GACF,CAAC,oBAAoB,GAAG;IACtB,QAAQ,CAAC,SAAS,EAAE,gBAAgB,CAAC;IACrC,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC;IACrB,QAAQ,CAAC,gBAAgB,CAAC,EAAE,SAAS,wBAAwB,EAAE,CAAC;IAChE,QAAQ,CAAC,eAAe,CAAC,EAAE,MAAM,CAAC;IAClC,QAAQ,CAAC,MAAM,CAAC,EAAE,SAAS,GAAG,QAAQ,CAAC;CACxC,CAAC,GACF,CAAC,oBAAoB,GAAG;IACtB,QAAQ,CAAC,SAAS,EAAE,YAAY,CAAC;IACjC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,SAAS,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC;IACvD,QAAQ,CAAC,qBAAqB,CAAC,EAAE,SAAS,6BAA6B,EAAE,CAAC;IAC1E,QAAQ,CAAC,eAAe,CAAC,EAAE,MAAM,CAAC;IAClC,QAAQ,CAAC,aAAa,CAAC,EAAE,MAAM,CAAC;IAChC,QAAQ,CAAC,MAAM,CAAC,EAAE,SAAS,GAAG,QAAQ,GAAG,WAAW,CAAC;CACtD,CAAC,GACF;IACE,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,SAAS,EAAE,QAAQ,CAAC;IAC7B,QAAQ,CAAC,OAAO,EAAE,SAAS,GAAG,YAAY,CAAC;IAC3C,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;CAC3B,CAAC;AAEN,MAAM,WAAW,2BAA2B;IAC1C,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,GAAG,CAAC;IACvB,QAAQ,CAAC,eAAe,CAAC,EAAE,YAAY,GAAG,YAAY,GAAG,YAAY,GAAG,YAAY,CAAC;IACrF,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC,QAAQ,CAAC,WAAW,EAAE,SAAS,gBAAgB,EAAE,CAAC;CACnD;AAED,MAAM,WAAW,wBAAwB;IACvC,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,SAAS,EAAE,gBAAgB,CAAC,WAAW,CAAC,CAAC;IAClD,QAAQ,CAAC,MAAM,EAAE,yBAAyB,CAAC;IAC3C,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,WAAW,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC;CAC1D;AAED,MAAM,WAAW,2BAA2B;IAC1C,QAAQ,CAAC,aAAa,EAAE,CAAC,CAAC;IAC1B,QAAQ,CAAC,YAAY,EAAE,SAAS,CAAC;IACjC,QAAQ,CAAC,aAAa,EAAE,YAAY,CAAC;IACrC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,cAAc,EAAE,MAAM,CAAC;IAChC,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;IACjC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,QAAQ,CAAC,MAAM,EAAE,yBAAyB,CAAC;IAC3C,QAAQ,CAAC,WAAW,EAAE,SAAS,wBAAwB,EAAE,CAAC;CAC3D;AAYD,gFAAgF;AAChF,wBAAgB,yBAAyB,CACvC,QAAQ,EAAE,2BAA2B,GACpC,2BAA2B,CAE7B;AAED;;;GAGG;AACH,wBAAsB,sBAAsB,CAC1C,QAAQ,EAAE,2BAA2B,EACrC,MAAM,EAAE,MAAM,GACb,OAAO,CAAC,2BAA2B,CAAC,CA6BtC"}
|