@emseepea/testing 0.0.1 → 0.1.0

package/README.md

@@ -1,45 +1,90 @@
 # `@emseepea/testing`
 
-Test what a Model Context Protocol (MCP) server returns and whether a language
-model understands it.
+Test whether an AI understands your MCP server's results, not just whether
+the server returns valid data. Write the checks in JavaScript, with ordinary
+imports, setup hooks, loops, and assertions.
 
-The package starts a Streamable HTTP server, connects through the official MCP
-client, and records the MCP operations used by each check. Semantic checks use
-Promptfoo and a model command supplied by the caller.
+The code-first API below is being prepared for the next package release.
 
-The package is intended to publish under the `next` tag with the first
-pre-alpha release.
+## Keep the Two Kinds of Test Separate
 
-## Add Semantic Checks to an Example
+Each example has two directories:
 
-Install the package as a development dependency:
+- `test/` holds ordinary tests. Run them with `npm test`.
+- `eval/` holds tests that ask a language model to interpret MCP results.
+  Run them with `npm run test:llm`.
 
-```sh
-npm install --save-dev @emseepea/testing@next
-```
+The commands do not run each other's tests. Both directories are linted.
+Organize larger suites into subdirectories, such as `eval/inventory/`.
+The LLM runner finds every `*.test.mjs` file inside `eval/`.
 
-Create `eval.yaml` beside the example. Say which server to start, which MCP
-operation to run, and what a correct answer must mean:
-
-```yaml
-description: Coffee details keep each concept distinct
-server: ./dist/server.js
-operations:
-  - method: tools/call
-    name: get-bean-details
-    arguments:
-      name: Highland Bloom
-question: What are this coffee's origin, variety, process, roast, and tasting notes?
-criticalFacts: [Sample Highlands, Bourbon, natural, medium, berry, cocoa]
-criteria: Keep origin, variety, process, roast, and tasting notes distinct.
-```
+## Write an LLM Test
+
+Import `semanticTest` from `@emseepea/testing/semantic` in a file such as
+`eval/coffee.test.mjs`. Start with the
+[basic coffee test](https://github.com/windyroad/emseepea/blob/main/examples/basic-no-ui/eval/meaning.test.mjs)
+or the
+[report test with repeated calls](https://github.com/windyroad/emseepea/blob/main/examples/multi-instance/eval/meaning.test.mjs).
+
+Each test describes:
+
+- `server`: the file URL of your built server entry point. It must print its
+  local `http://127.0.0.1:PORT/mcp` address when ready.
+- `exercise`: code that calls tools, reads resources, or requests prompts
+  through the supplied MCP client. Use assertions to check the returned data.
+- `question`: what to ask the model about those results.
+- `criticalFacts`: text or regular expressions that every answer must match.
+  Text checks ignore letter case; patterns use their own flags.
+- `criteria`: what a correct answer must mean, including mistakes to avoid.
+- `requiredPaths`: which MCP operations must supply the evidence, written as
+  `method:target`, such as `tools/call:get-bean-details`.
+
+Use the optional `assertAnswer(answer)` callback for your own exact checks on
+the model's answer. Each test needs a unique name within its file.
+
+The repeated-report example asks for JSON and checks every field; it does not
+test free-form prose.
+
+The callback's client supports `callTool`, `readResource`, and `getPrompt`.
+Pass request parameters as the single argument to each method. The helper
+records every call and includes tool progress in the model's input.
 
-Build the example, then run:
+## Run the Checks
+
+In a copied example, install its dependencies and run:
 
 ```sh
-npx emseepea-test eval.yaml
+npm test
+npm run lint
+npm run test:llm
 ```
 
-The check calls the server through the official MCP client. It runs three
-answers and three independent judgments per answer, and saves the evidence
-under `artifacts/llm-eval/`.
+The last command requires the Claude CLI on your command path and a signed-in
+Claude account. If needed, run `claude auth login` first. This package does not
+bundle the CLI or copy your login credentials.
+
+For your own project, add `@emseepea/testing` as a development dependency and
+set `test:llm` to build your server and run `emseepea-test eval`.
+
+For a server that requires sign-in, set `authTokenEnvironment` to the name of
+an environment variable containing its test access token. Do not put real
+tokens in test files. The optional `environment` object supplies ordinary test
+settings to the server; model-provider credentials are not passed through.
+
+## What a Passing Check Means
+
+For each test, the runner asks the same question three times. Each answer gets
+three independent model judgments, for nine judgments in total. A missing
+fact, failed assertion, rejected answer, or missing MCP call fails the test.
+Failed answers are not retried or taken from a cache.
+
+Your test code makes the MCP calls. The model interprets the collected results;
+it cannot call tools itself. These checks prove interpretation of that material,
+not that the model can independently choose the right tool.
+
+Results are saved to `artifacts/llm-eval/evidence.json`. The report contains
+outcomes and operation hashes, not raw MCP results or model answers.
+
+Repository tests also use a simulated model to check the runner without spending
+model credits. Those `--smoke` checks test the wiring only. They do not prove
+that a real model understands the result and cannot approve a release.

package/dist/index.d.ts

@@ -0,0 +1,16 @@
+import { Client } from "@modelcontextprotocol/client";
+export interface TestCleanup {
+    after(cleanup: () => Promise<void>): void;
+}
+export interface StartMcpServerOptions {
+    clientName?: string;
+    environment?: NodeJS.ProcessEnv;
+    startupTimeoutMs?: number;
+    token?: string;
+}
+export interface RunningMcpServer {
+    connect(token?: string): Promise<Client>;
+    url: URL;
+}
+export declare function startMcpServer(test: TestCleanup, serverUrl: URL, options?: StartMcpServerOptions): Promise<RunningMcpServer>;
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,MAAM,EAAiC,MAAM,8BAA8B,CAAC;AAErF,MAAM,WAAW,WAAW;IAC1B,KAAK,CAAC,OAAO,EAAE,MAAM,OAAO,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC;CAC3C;AAED,MAAM,WAAW,qBAAqB;IACpC,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,WAAW,CAAC,EAAE,MAAM,CAAC,UAAU,CAAC;IAChC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,gBAAgB;IAC/B,OAAO,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC;IACzC,GAAG,EAAE,GAAG,CAAC;CACV;AAED,wBAAsB,cAAc,CAClC,IAAI,EAAE,WAAW,EACjB,SAAS,EAAE,GAAG,EACd,OAAO,GAAE,qBAA0B,GAClC,OAAO,CAAC,gBAAgB,CAAC,CAgE3B"}

package/dist/index.js

@@ -0,0 +1,71 @@
+import { spawn } from "node:child_process";
+import { fileURLToPath } from "node:url";
+import { Client, StreamableHTTPClientTransport } from "@modelcontextprotocol/client";
+export async function startMcpServer(test, serverUrl, options = {}) {
+    const child = spawn(process.execPath, [fileURLToPath(serverUrl)], {
+        env: {
+            ...process.env,
+            NODE_ENV: "test",
+            PORT: "0",
+            ...options.environment,
+        },
+        stdio: ["ignore", "pipe", "pipe"],
+    });
+    const clients = [];
+    let output = "";
+    let errors = "";
+    child.stdout.on("data", (chunk) => { output = `${output}${chunk}`.slice(-16_384); });
+    child.stderr.on("data", (chunk) => { errors = `${errors}${chunk}`.slice(-16_384); });
+    const url = await new Promise((resolve, reject) => {
+        const finish = (error, value) => {
+            clearTimeout(timer);
+            child.stdout.off("data", inspect);
+            child.off("error", failed);
+            child.off("exit", exited);
+            if (error)
+                reject(error);
+            else if (value)
+                resolve(value);
+        };
+        const inspect = () => {
+            const match = output.match(/http:\/\/127\.0\.0\.1:\d+\/mcp/);
+            if (match?.[0])
+                finish(undefined, new URL(match[0]));
+        };
+        const failed = (error) => finish(error);
+        const exited = (code) => finish(new Error(`MCP server exited ${String(code)}: ${errors}`));
+        const timer = setTimeout(() => finish(new Error(`MCP server startup timed out: ${errors}`)), options.startupTimeoutMs ?? 15_000);
+        child.stdout.on("data", inspect);
+        child.once("error", failed);
+        child.once("exit", exited);
+        inspect();
+    }).catch(async (error) => {
+        await stopProcess(child);
+        throw error;
+    });
+    test.after(async () => {
+        await Promise.allSettled(clients.map((client) => client.close()));
+        await stopProcess(child);
+    });
+    return {
+        url,
+        async connect(token = options.token) {
+            const client = new Client({ name: options.clientName ?? "emseepea-test", version: "0.0.0" }, { versionNegotiation: { mode: { pin: "2026-07-28" } } });
+            await client.connect(new StreamableHTTPClientTransport(url, token ? { authProvider: { token: async () => token } } : undefined));
+            clients.push(client);
+            return client;
+        },
+    };
+}
+async function stopProcess(child) {
+    if (child.exitCode !== null)
+        return;
+    child.kill("SIGTERM");
+    await Promise.race([
+        new Promise((resolve) => child.once("close", () => resolve())),
+        new Promise((resolve) => setTimeout(resolve, 3_000)),
+    ]);
+    if (child.exitCode === null)
+        child.kill("SIGKILL");
+}
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AAEzC,OAAO,EAAE,MAAM,EAAE,6BAA6B,EAAE,MAAM,8BAA8B,CAAC;AAkBrF,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,IAAiB,EACjB,SAAc,EACd,UAAiC,EAAE;IAEnC,MAAM,KAAK,GAAG,KAAK,CAAC,OAAO,CAAC,QAAQ,EAAE,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC,EAAE;QAChE,GAAG,EAAE;YACH,GAAG,OAAO,CAAC,GAAG;YACd,QAAQ,EAAE,MAAM;YAChB,IAAI,EAAE,GAAG;YACT,GAAG,OAAO,CAAC,WAAW;SACvB;QACD,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;KAClC,CAAC,CAAC;IACH,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,MAAM,GAAG,EAAE,CAAC;IAChB,IAAI,MAAM,GAAG,EAAE,CAAC;IAChB,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,KAAa,EAAE,EAAE,GAAG,MAAM,GAAG,GAAG,MAAM,GAAG,KAAK,EAAE,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC7F,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,KAAa,EAAE,EAAE,GAAG,MAAM,GAAG,GAAG,MAAM,GAAG,KAAK,EAAE,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAE7F,MAAM,GAAG,GAAG,MAAM,IAAI,OAAO,CAAM,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;QACrD,MAAM,MAAM,GAAG,CAAC,KAAa,EAAE,KAAW,EAAE,EAAE;YAC5C,YAAY,CAAC,KAAK,CAAC,CAAC;YACpB,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;YAClC,KAAK,CAAC,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;YAC3B,KAAK,CAAC,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;YAC1B,IAAI,KAAK;gBAAE,MAAM,CAAC,KAAK,CAAC,CAAC;iBACpB,IAAI,KAAK;gBAAE,OAAO,CAAC,KAAK,CAAC,CAAC;QACjC,CAAC,CAAC;QACF,MAAM,OAAO,GAAG,GAAG,EAAE;YACnB,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,gCAAgC,CAAC,CAAC;YAC7D,IAAI,KAAK,EAAE,CAAC,CAAC,CAAC;gBAAE,MAAM,CAAC,SAAS,EAAE,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QACvD,CAAC,CAAC;QACF,MAAM,MAAM,GAAG,CAAC,KAAY,EAAE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QAC/C,MAAM,MAAM,GAAG,CAAC,IAAmB,EAAE,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,qBAAqB,MAAM,CAAC,IAAI,CAAC,KAAK,MAAM,EAAE,CAAC,CAAC,CAAC;QAC1G,MAAM,KAAK,GAAG,UAAU,CACtB,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,iCAAiC,MAAM,EAAE,CAAC,CAAC,EAClE,OAAO,CAAC,gBAAgB,IAAI,MAAM,CACnC,CAAC;QACF,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QACjC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC5B,KAAK,CAAC,IAAI,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC3B,OAAO,EAAE,CAAC;IACZ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,EAAE,KAAc,EAAE,EAAE;QAChC,MAAM,WAAW,CAAC,KAAK,CAAC,CAAC;QACzB,MAAM,KAAK,CAAC;IACd,CAAC,CAAC,CAAC;IAEH,IAAI,CAAC,KAAK,CAAC,KAAK,IAAI,EAAE;QACpB,MAAM,OAAO,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;QAClE,MAAM,WAAW,CAAC,KAAK,CAAC,CAAC;IAC3B,CAAC,CAAC,CAAC;IAEH,OAAO;QACL,GAAG;QACH,KAAK,CAAC,OAAO,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK;YACjC,MAAM,MAAM,GAAG,IAAI,MAAM,CACvB,EAAE,IAAI,EAAE,OAAO,CAAC,UAAU,IAAI,eAAe,EAAE,OAAO,EAAE,OAAO,EAAE,EACjE,EAAE,kBAAkB,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,YAAY,EAAE,EAAE,EAAE,CACxD,CAAC;YACF,MAAM,MAAM,CAAC,OAAO,CAAC,IAAI,6BAA6B,CACpD,GAAG,EACH,KAAK,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,EAAE,KAAK,EAAE,KAAK,IAAI,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CACnE,CAAC,CAAC;YACH,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;YACrB,OAAO,MAAM,CAAC;QAChB,CAAC;KACF,CAAC;AACJ,CAAC;AAED,KAAK,UAAU,WAAW,CAAC,KAA+B;IACxD,IAAI,KAAK,CAAC,QAAQ,KAAK,IAAI;QAAE,OAAO;IACpC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACtB,MAAM,OAAO,CAAC,IAAI,CAAC;QACjB,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;QACpE,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;KAC3D,CAAC,CAAC;IACH,IAAI,KAAK,CAAC,QAAQ,KAAK,IAAI;QAAE,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;AACrD,CAAC"}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@emseepea/testing",
-  "version": "0.0.1",
+  "version": "0.1.0",
   "description": "MCP integration and semantic testing helpers",
   "license": "MIT",
   "repository": {
@@ -20,6 +20,10 @@
     ".": {
       "types": "./dist/index.d.ts",
       "import": "./dist/index.js"
+    },
+    "./semantic": {
+      "types": "./semantic/test.d.mts",
+      "import": "./semantic/test.mjs"
     }
   },
   "bin": {
@@ -33,12 +37,12 @@
   ],
   "scripts": {
     "build": "tsc -p tsconfig.json",
+    "prepack": "npm run build",
     "test": "npm run build && npm run test:built",
     "test:built": "node --test test/*.test.mjs"
   },
   "dependencies": {
-    "@modelcontextprotocol/client": "2.0.0",
-    "promptfoo": "0.122.1"
+    "@modelcontextprotocol/client": "2.0.0"
   },
   "engines": {
     "node": ">=22"

package/semantic/case.mjs

@@ -1,189 +1,44 @@
-import { readFile } from "node:fs/promises";
-import { dirname, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
-
-const methods = new Set(["tools/call", "resources/read", "prompts/get"]);
-
-export async function loadSemanticCase(path) {
-  const absolutePath = path instanceof URL ? fileURLToPath(path) : resolve(path);
-  const value = parseCase(await readFile(absolutePath, "utf8"));
-  if (!value || typeof value !== "object" || Array.isArray(value)) fail("must be an object");
-  for (const key of ["description", "question", "criteria", "server"]) {
-    if (typeof value[key] !== "string" || value[key].trim() === "") fail(`${key} must be text`);
-  }
-  if (!Array.isArray(value.criticalFacts) || value.criticalFacts.some((fact) => typeof fact !== "string" || !fact)) {
-    fail("criticalFacts must be a non-empty text list");
-  }
-  if (!Array.isArray(value.operations) || value.operations.length === 0) fail("operations must not be empty");
-  for (const operation of value.operations) {
-    if (!operation || typeof operation !== "object" || !methods.has(operation.method)) {
-      fail("each operation needs a supported method");
-    }
-    const target = operation.name ?? operation.uri;
-    if (typeof target !== "string" || target === "") fail("each operation needs a name or uri");
-  }
-  if (value.environment !== undefined && (!value.environment || typeof value.environment !== "object" || Array.isArray(value.environment))) {
-    fail("environment must be an object");
-  }
-  if (value.authTokenEnvironment !== undefined && typeof value.authTokenEnvironment !== "string") {
-    fail("authTokenEnvironment must be text");
-  }
-  if (value.authToken !== undefined && typeof value.authToken !== "string") fail("authToken must be text");
-  return {
-    ...value,
-    path: absolutePath,
-    directory: dirname(absolutePath),
-    server: resolve(dirname(absolutePath), value.server),
-  };
-}
-
-export function requiredPaths(testCase) {
-  return testCase.operations.map((operation) => (
-    `${operation.method}:${operation.name ?? operation.uri}`
-  ));
-}
-
-function fail(message) {
-  throw new Error(`Invalid semantic case: ${message}`);
-}
-
-function parseCase(source) {
-  const lines = source.split(/\r?\n/);
-  const value = {};
-  for (let index = 0; index < lines.length;) {
-    const line = lines[index];
-    if (line.trim() === "" || line.trimStart().startsWith("#")) {
-      index += 1;
-      continue;
-    }
-    const top = /^([A-Za-z][A-Za-z0-9]*):(?:\s*(.*))?$/.exec(line);
-    if (!top) fail(`unsupported line: ${line}`);
-    const [, key, raw = ""] = top;
-    const rest = raw.trimEnd();
-    if (key === "operations") {
-      const parsed = parseOperations(lines, index + 1);
-      value.operations = parsed.value;
-      index = parsed.next;
-    } else if (rest === ">-") {
-      const parsed = parseFoldedText(lines, index + 1);
-      value[key] = parsed.value;
-      index = parsed.next;
-    } else if (key === "environment" && rest === "") {
-      const parsed = parseObject(lines, index + 1, "  ");
-      value.environment = parsed.value;
-      index = parsed.next;
-    } else if (rest === "") {
-      const parsed = parseList(lines, index + 1);
-      value[key] = parsed.value;
-      index = parsed.next;
-    } else if (rest.startsWith("[") && rest.endsWith("]")) {
-      value[key] = rest.slice(1, -1).split(",").map((item) => unquote(item.trim())).filter(Boolean);
-      index += 1;
-    } else {
-      value[key] = unquote(rest);
-      index += 1;
-    }
-  }
-  return value;
-}
-
-function parseOperations(lines, start) {
-  const operations = [];
-  let index = start;
-  while (index < lines.length) {
-    const line = lines[index];
-    if (line.trim() === "") {
-      index += 1;
-      continue;
-    }
-    if (!line.startsWith("  - ")) break;
-    const operation = {};
-    const first = line.slice(4);
-    applyPair(operation, first);
-    index += 1;
-    while (index < lines.length) {
-      const current = lines[index];
-      if (current.trim() === "") {
-        index += 1;
-        continue;
-      }
-      if (current.startsWith("  - ") || !current.startsWith("    ")) break;
-      const pair = /^    ([A-Za-z][A-Za-z0-9]*):(?:\s*(.*))?$/.exec(current);
-      if (!pair) fail(`unsupported operation line: ${current}`);
-      const [, key, raw = ""] = pair;
-      if (key === "arguments" && raw.trimEnd() === "") {
-        const parsed = parseObject(lines, index + 1, "      ", true);
-        operation.arguments = parsed.value;
-        index = parsed.next;
-      } else {
-        operation[key] = unquote(raw.trimEnd());
-        index += 1;
-      }
+import { dirname } from "node:path";
+
+export function validateSemanticCase(value) {
+  if (!value || typeof value !== "object") throw new Error("Semantic test needs options");
+  for (const key of ["question", "criteria"]) {
+    if (typeof value[key] !== "string" || !value[key].trim()) throw new Error(`${key} must be text`);
+  }
+  if (!(value.server instanceof URL) || value.server.protocol !== "file:") throw new Error("server must be a file URL");
+  for (const key of ["criticalFacts", "requiredPaths"]) {
+    if (!Array.isArray(value[key]) || !value[key].length || value[key].some((item) =>
+      !(key === "criticalFacts" && item instanceof RegExp) && (typeof item !== "string" || !item.trim()))) {
+      throw new Error(`${key} must contain text${key === "criticalFacts" ? " or regular expressions" : ""}`);
     }
-    operations.push(operation);
-  }
-  return { value: operations, next: index };
-}
-
-function parseList(lines, start) {
-  const items = [];
-  let index = start;
-  while (index < lines.length) {
-    const item = /^  -\s+(.+)$/.exec(lines[index]);
-    if (!item) break;
-    items.push(unquote(item[1].trimEnd()));
-    index += 1;
-  }
-  if (items.length === 0) fail("expected a list");
-  return { value: items, next: index };
-}
-
-function parseObject(lines, start, indent, parseScalars = false) {
-  const object = {};
-  let index = start;
-  while (index < lines.length) {
-    if (!lines[index].startsWith(indent)) break;
-    applyPair(object, lines[index].slice(indent.length), parseScalars);
-    index += 1;
   }
-  return { value: object, next: index };
-}
-
-function parseFoldedText(lines, start) {
-  const parts = [];
-  let index = start;
-  while (index < lines.length) {
-    if (!lines[index].startsWith("  ")) break;
-    parts.push(lines[index].trim());
-    index += 1;
+  if (typeof value.exercise !== "function") throw new Error("exercise must be a function");
+  if (value.assertAnswer !== undefined && typeof value.assertAnswer !== "function") throw new Error("assertAnswer must be a function");
+  if (value.requiredPaths.some((path) => !/^(tools\/call|resources\/read|prompts\/get):.+$/.test(path))) {
+    throw new Error("requiredPaths must name a supported MCP method and target");
   }
-  return { value: parts.join(" "), next: index };
-}
-
-function applyPair(target, source, parseScalars = false) {
-  const pair = /^([A-Za-z][A-Za-z0-9_]*):\s*(.*)$/.exec(source);
-  if (!pair) fail(`expected key-value pair: ${source}`);
-  target[pair[1]] = parseScalars ? parseArgumentScalar(pair[2]) : unquote(pair[2].trimEnd());
-}
-
-function parseArgumentScalar(value) {
-  const trimmed = value.trim();
-  if ((trimmed.startsWith('"') && trimmed.endsWith('"')) || (trimmed.startsWith("'") && trimmed.endsWith("'"))) {
-    return unquote(trimmed);
+  for (const key of ["authToken", "authTokenEnvironment"]) {
+    if (value[key] !== undefined && (typeof value[key] !== "string" || !value[key].trim())) throw new Error(`${key} must be non-empty text`);
   }
-  if (trimmed === "true") return true;
-  if (trimmed === "false") return false;
-  if (/^-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?$/.test(trimmed)) {
-    const number = Number(trimmed);
-    if (Number.isFinite(number)) return number;
+  if (value.authToken !== undefined && value.authTokenEnvironment !== undefined) throw new Error("Choose one authentication source");
+  if (value.environment !== undefined && (!value.environment || typeof value.environment !== "object" ||
+    Array.isArray(value.environment) || Object.values(value.environment).some((item) => typeof item !== "string"))) {
+    throw new Error("environment must contain string values");
   }
-  return trimmed;
+  const server = fileURLToPath(value.server);
+  return { ...value, server, directory: dirname(server) };
 }
 
-function unquote(value) {
-  const trimmed = value.trim();
-  if ((trimmed.startsWith('"') && trimmed.endsWith('"')) || (trimmed.startsWith("'") && trimmed.endsWith("'"))) {
-    return trimmed.slice(1, -1);
+export function checkMeaningEvidence(options, answer, paths) {
+  const seen = new Set(paths.map(({ method, target }) => `${method}:${target}`));
+  if (options.requiredPaths.some((path) => !seen.has(path))) throw new Error("Required MCP path evidence is missing");
+  const missingFactIndices = options.criticalFacts.flatMap((fact, index) =>
+    (fact instanceof RegExp ? new RegExp(fact.source, fact.flags).test(answer)
+      : answer.toLowerCase().includes(fact.toLowerCase())) ? [] : [index]);
+  if (missingFactIndices.length) {
+    throw Object.assign(new Error("Answer is missing a required fact"), {
+      code: "missing-critical-facts", missingFactIndices,
+    });
   }
-  return trimmed;
 }