@marinoscar/kvox-eval 0.0.0-stage → 0.1.0-next.0

package/CHANGELOG.md

@@ -0,0 +1,14 @@
+# @marinoscar/kvox-eval
+
+## 0.1.0-next.0
+
+### Minor Changes
+
+- 7310f9d: Add the headless Ask evaluation (#608). The new `@marinoscar/kvox-eval/ask-run` subpath (`runAskEval`, `runAskReplay`, `startAskEvalHost`, `seedGoldenGraph`, `runAskCommand`) runs the golden question set through the real `ask.respond` handler on an in-memory PGlite migrated with the platform and domain history, seeding the golden graph through `GraphWriteService`. It never touches a real database. `kvox-eval ask` (`--provider`, `--model`, `--base-url`, `--only`, `--category`, `--out`, `--min-match`, `--replay`) replaces the stub that exited 2; `--replay` needs no key. Exit codes are 0, 1 and 2, and the key is read only from `KVOX_EVAL_OPENAI_API_KEY`.
+
+  The root entry does not export `ask-run`; import the subpath. The package now ships `host/prisma` and `host/generated`, built by `scripts/prepare-host.mjs`.
+
+  New required peers: `@electric-sql/pglite` (0.5.10), `@electric-sql/pglite-pgvector` (0.0.11), `@electric-sql/pglite-socket` (0.2.13), `@prisma/client`, `@prisma/adapter-pg`, `pg`, `@nestjs/common`, `@nestjs/core`, `reflect-metadata`, `rxjs`.
+
+- c060a77: New package `@marinoscar/kvox-eval`: the KVox knowledge-graph and Ask evaluation (#580), written only against the model port (`KvoxAiPort`, bound with `createKvoxAi` of `@marinoscar/kvox-ai`). It moves in from the web app unchanged in behaviour: the synthetic golden set (`./fixtures`: 36 meetings and the Ask question file, with their schema and loaders), the prediction schema, the scorer, the report and the gold, `extract` and `extract+resolve` runners, `runKgEval`, the pure half of the Ask evaluation (question schema, scorer, report), the real-data path guard, and the `kvox-eval` CLI (`kvox-eval kg`; `kvox-eval ask` is not headless yet). Live runs go through `createDirectRuntime` over the platform's OpenAI and OpenAI-compatible adapters with the key from `KVOX_EVAL_OPENAI_API_KEY` (`KG_EVAL_OPENAI_API_KEY` and `ASK_EVAL_OPENAI_API_KEY` are read for one more release); the web app's private OpenAI client is gone.
+- 8dfebb9: Harness parity (#580). `@marinoscar/kvox-ai/testing` gains request recorders: `createRecordingRuntime` (every runtime round trip, without user, org, job or signal), `createRecordingAdapter` (a scripted provider adapter for `createDirectRuntime` that records what it receives) and `normalizeRecordedValue` (ids, timestamps and request ids numbered by first appearance, nothing dropped). `@marinoscar/kvox-eval/parity` drives notes generation, `kg.extract`, adjudication and three Ask questions through the real package code over any `KvoxAiPort` and ships the request snapshots (`parity/requests.*.json`) every host binding must reproduce; `kg` also exports `runExtraction` and `fixtureToInput`. `@marinoscar/kvox-api/notes` exports `budgetFromTarget` and the `AiTokenCaps` type.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 marinoscar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,166 @@
-# Temporary Holding Version
+# @marinoscar/kvox-eval
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+The KVox knowledge-graph and Ask evaluation: the synthetic golden set, its schemas, the scorers, the report, the gold and live runners, the headless Ask run and the `kvox-eval` command. Written only against the model port, so every host measures the prompts, schemas and validators the product ships. CommonJS.
+
+## Purpose and scope
+
+One package for "how good is extraction (and Ask) on the golden set", shared by the web edition and KVox Desktop. A live run binds the same `KvoxAiPort` every host binds (`createKvoxAi` of `@marinoscar/kvox-ai`, over `createDirectRuntime` and the platform's provider adapters); the extraction pipeline it scores is `@marinoscar/kvox-api`'s own (`graph-extraction` internals), minus the loader and the writer. The package owns no prompt, schema or pipeline of the product.
+
+The Ask run is headless (issue #608): `kvox-eval ask` starts an in-memory PGlite, applies the shipped migration SQL in order, seeds the golden graph through `GraphWriteService` and drives the real `ask.respond` handler, with no web app and no real database. `npm run ask:eval --workspace=api` is a thin wrapper over the same command. Exporting one of your own notes as an unlabelled skeleton (`--export-note`) reads the web app's tables and stays in `apps/api`.
+
+Status: implemented by issue #580 (moved from `apps/api/scripts/kg-eval/`, `apps/api/scripts/ask-eval/` and `apps/api/test/fixtures/kg-golden/`), unreleased like the other packages. The web app's private OpenAI client (`direct-structured.ts`) is deleted; `npm run kg:eval --workspace=api` is a thin wrapper over `runKgCommand`.
+
+Entry points:
+
+- `@marinoscar/kvox-eval`: the package root, which re-exports the `kg`, `ask`, `live`, `real-data` and `cli` slices (not `ask-run`: import the subpath below). The root is still slow to import (about 2 s) because the kg runners import `@marinoscar/kvox-api/graph`. Slice READMEs: [kg](src/kg/README.md), [ask](src/ask/README.md), [live](src/live/README.md), [real-data](src/real-data/README.md), [cli](src/cli/README.md).
+- `@marinoscar/kvox-eval/fixtures`: the golden set (`loadGoldenMeetings`, `GOLDEN_MEETINGS_DIR`, `ASK_QUESTIONS_PATH`, `loadGoldenAskQuestions`) and its schema (`goldenFixtureSchema`, `GoldenFixture`). Slice README: [fixtures](src/fixtures/README.md). The data is under [fixtures/kg-golden](fixtures/kg-golden/README.md).
+- `@marinoscar/kvox-eval/ask-run`: the headless Ask evaluation (`runAskEval`, `runAskReplay`, `startAskEvalHost`, `seedGoldenGraph`, `runAskCommand`): an in-memory PGlite, the golden graph seeded through the graph write service, the real `ask.respond` handler. Slice README: [ask-run](src/ask-run/README.md).
+- `@marinoscar/kvox-eval/parity`: the harness-parity scenario and its committed request snapshots (`parity/requests.*.json`), which prove two bindings of the model port send the same provider requests (`runParityScenario`, `createParityScript`, `loadParitySnapshot`). Slice README: [parity](src/parity/README.md).
+- `kvox-eval` (the `bin`, `dist/cli/main.js`): `kvox-eval kg` and `kvox-eval ask`.
+
+## Install and peer dependencies
+
+```bash
+npm install @marinoscar/kvox-eval@next @marinoscar/kvox-ai@next @marinoscar/kvox-api@next @marinoscar/kvox-contract@next @marinoscar/platform-api@0.1.0-next.5 zod \
+  @electric-sql/pglite@0.5.10 @electric-sql/pglite-pgvector@0.0.11 @electric-sql/pglite-socket@0.2.13 \
+  @prisma/client @prisma/adapter-pg pg @nestjs/common @nestjs/core reflect-metadata rxjs
+```
+
+Every slice is universal, so every peer is required, including those only `ask-run` imports (PGlite, Prisma, Nest):
+
+| Package | Range | Needed by |
+|---|---|---|
+| `zod` | `^4.4.3` | `fixtures`, `ask`, `kg` (the fixture, prediction, report and question schemas) |
+| `@marinoscar/kvox-contract` | `>=0.0.0-0 \|\| >=0.1.0-0` | `fixtures`, `kg` (the ontology registry), `ask` (the Ask wire types) |
+| `@marinoscar/kvox-api` | `>=0.0.0-0 \|\| >=0.1.0-0` | `kg` (the extraction pipeline, `graph-extraction` and `graph`), `live` and `cli` (`KvoxAiPort`, `KvoxAiError`) |
+| `@marinoscar/kvox-ai` | `>=0.0.0-0 \|\| >=0.1.0-0` | `live` (`createKvoxAi`, `createDirectRuntime`) |
+| `@marinoscar/platform-api` | `0.1.0-next.5` (exact, the repository's platform pin) | `live` and `cli`: the OpenAI and OpenAI-compatible adapters; `ask-run`: the provider event types of the replay. Only `@marinoscar/platform-api/ai`, as in `@marinoscar/kvox-ai`; an interim peer until the lean adapter package of #578. |
+| `@electric-sql/pglite` | `0.5.10` (exact) | `ask-run`: the in-memory PostgreSQL |
+| `@electric-sql/pglite-pgvector` | `0.0.11` (exact) | `ask-run`: the `vector` extension the migrations create |
+| `@electric-sql/pglite-socket` | `0.2.13` (exact) | `ask-run`: the loopback socket the Prisma client connects to |
+| `@prisma/client` | `^7.8.0` | `ask-run`: the client generated into `host/generated` |
+| `@prisma/adapter-pg` | `^7.8.0` | `ask-run`: the driver adapter over `pg` |
+| `pg` | `^8.23.0` | `ask-run` |
+| `@nestjs/common`, `@nestjs/core` | `^11.1.12` | `ask-run`: the application context of the Ask slices |
+| `reflect-metadata` | `^0.2.2` | `ask-run` |
+| `rxjs` | `^7.8.1` | `ask-run` |
+
+`@marinoscar/kvox-api`'s graph slices bring their own peers (Nest, among others). The per-slice data is `packages/kvox-slice-peers.json`; `node scripts/kvox-packages/check-slice-peers.mjs --table` prints it.
+
+## Quick start
+
+The harness self-test, and a score of predictions written earlier:
+
+```bash
+kvox-eval kg --predictions gold                 # 1.000 everywhere, or the scorer has a bug
+kvox-eval kg --predictions ./runs/2026-10-17 --only m01,m07 --json report.json
+```
+
+A live run over the model port (the key comes from the environment only):
+
+```bash
+export KVOX_EVAL_OPENAI_API_KEY=...             # never a flag; never printed
+kvox-eval kg --runner extract --provider openai --model gpt-5-mini --out /tmp/kg-run
+kvox-eval kg --runner extract-resolve --provider openai-compatible --base-url http://localhost:8000/v1 --model local-model
+```
+
+The Ask evaluation, headless (in-memory database, golden graph seeded through the graph write service, the real `ask.respond` handler):
+
+```bash
+kvox-eval ask --replay                          # the scripted self-test: 6 questions and the leak gate; no key, no network
+export KVOX_EVAL_OPENAI_API_KEY=...
+kvox-eval ask --model gpt-4o-mini --out /tmp/ask-run
+kvox-eval ask --only q01,q22 --category negative --min-match 0.7
+kvox-eval ask --provider openai-compatible --base-url http://localhost:8000/v1 --model local-model
+```
+
+Flags: `--provider openai|openai-compatible`, `--model`, `--base-url`, `--only`, `--category`, `--out` (writes `report.json` and `answers.md`; `replay.json` for `--replay`), `--min-match <0..1>`, `--replay`. Exit codes: 0 pass, 1 a gate failed (citation validity below 1, a sensitive-fact leak) or the answer match fell below `--min-match` (or a replay expectation failed), 2 configuration (no key, a refused provider, a bad flag, an unknown question id) or every model call failed. One failed model call marks that turn failed and the run goes on. The seeded graph is 36 meetings, 94 entities, 265 relations, 187 items and 731 evidence rows, 0 refused. Semantic search is off (no embeddings). `--replay` reproduces `apps/api/test/ask/ask-eval.replay.db.spec.ts`.
+
+From code:
+
+```ts
+import { createLiveKvoxAi, formatReport, readEvalApiKey, runKgEval } from '@marinoscar/kvox-eval';
+import { loadGoldenMeetings } from '@marinoscar/kvox-eval/fixtures';
+
+const gold = await runKgEval({ runner: 'gold' });
+console.log(formatReport(gold));
+
+const key = readEvalApiKey(process.env);
+const { ai } = createLiveKvoxAi({ provider: 'openai', model: 'gpt-5-mini', apiKey: key?.apiKey ?? null });
+const live = await runKgEval({ runner: 'extract', ai, model: 'gpt-5-mini', fixtures: ['m01'] });
+
+loadGoldenMeetings(['m01', 'm07']); // the fixtures themselves
+```
+
+```ts
+import { runAskReplay } from '@marinoscar/kvox-eval/ask-run'; // never from the root
+
+const replay = await runAskReplay(); // no key; starts and closes its own PGlite
+```
+
+## Configuration
+
+Environment only, read by the `live` slice and the CLI:
+
+- `KVOX_EVAL_OPENAI_API_KEY`: the key of a live run (`kg` and `ask`; read only from here, never a flag). `kvox-eval ask --replay` needs none. `KG_EVAL_OPENAI_API_KEY` and `ASK_EVAL_OPENAI_API_KEY` are read for one more release, with a one-line deprecation note on standard error.
+- `KG_EVAL_REAL_DIR`: stands in for `--real-dir`.
+
+No settings key and no file. The models a live run uses are the command line's (`--provider`, `--model`, `--base-url`).
+
+## Extension-point catalog
+
+- `KgEvalRunner`: one fixture to one prediction. Register a factory in `KG_EVAL_RUNNERS` (a test's stub runner does) and it becomes `--runner <name>`.
+
+## Data
+
+The golden set, shipped in the tarball under `fixtures/kg-golden/`: 36 synthetic meetings (`meetings/mNN-<slug>.json`) and the Ask question file (`ask-questions.json`). **Synthetic only, ever**: no real recording, transcript, note, name, company, email address or phone number belongs there (the rule, the format and the coverage the set must keep: [fixtures/kg-golden/README.md](fixtures/kg-golden/README.md); `test/kg-golden-fixtures.spec.ts` asserts them). A fixture's and a question's id is permanent once committed. The package owns no table. The `ask-run` slice ships `host/prisma` and `host/generated` (the schema, migrations and Prisma client), generated at build time by `scripts/prepare-host.mjs`; the in-memory database they run on is created per run and discarded.
+
+## Permissions and settings
+
+None. A live run has no signed-in user: its calls carry the user id `kg-eval` and no organization.
+
+## UI
+
+None.
+
+## Infra
+
+None. A live run reaches the provider's API from the machine it runs on. The Ask run needs no database server: it never touches a real database.
+
+## Observability
+
+None: no log, metric or span (the Ask run's Nest context logs errors and warnings only). The CLI prints the report table to standard output and its notes (missing predictions, where predictions were written, deprecations) to standard error.
+
+## Security notes
+
+- The API key is read from the environment only, never a flag. It is handed to `createDirectRuntime`'s `keys()` per call and never logged, printed, put in an error or written into a prediction or a report (a prediction carries `model` alone). A failed live call is reported by its `KvoxAiError` class, the platform error code and the HTTP status, never a message body.
+- Real meetings never enter the repository: `--real-dir`, `--out` of a real-data run and `KG_EVAL_REAL_DIR` are refused (exit 3) when they resolve inside the git work tree, symlinks included (`assertOutsideRepo`).
+- The golden set is synthetic; its spec rejects a real-looking email address, a phone number and the product's name.
+- The Ask run uses an in-memory PGlite only: it has no connection string to a real database, so it cannot read or write one. The key never enters that database.
+- Apart from the `ask-run` slice (Prisma, Nest, PGlite), the source imports no Prisma, no `@app/shared`, and of the platform only `@marinoscar/platform-api/ai` (rule `host-neutral` of `npm run check:kvox-slice-peers`).
+
+## Conformance suite
+
+None of its own to export. The package's specs (`npm run test:run -w @marinoscar/kvox-eval`) pin the scorer case by case, the golden set's validity and coverage, the extraction runners over a fake port (`createFakeKvoxAi` of `@marinoscar/kvox-api/testing`), the Ask question file and scorer, `runKgEval`, the key aliases, the copilot refusal and the real-data guard. `test/parity.spec.ts` runs the harness-parity scenario over the direct binding and compares it with the shipped `parity/` snapshots (`npm run test:run -w @marinoscar/kvox-eval -- parity`; `KVOX_EVAL_UPDATE_PARITY=1` rewrites them, see [parity](src/parity/README.md)); the web edition's twin is `apps/api/test/kvox/ai-parity.spec.ts`. The gold self-test (`kvox-eval kg --predictions gold`) must print 1.000 everywhere.
+
+## Upgrade notes
+
+First release. For code that imported these files from the web app: `apps/api/scripts/kg-eval/{fixture-schema,load,domains}.ts` are `@marinoscar/kvox-eval/fixtures`; `prediction-schema`, `scorer`, `report`, `targets`, `text`, `gold-runner`, `runner` and `runners/*` are the root; `apps/api/scripts/ask-eval/{question-schema,scorer,report}.ts` are the root. `loadPredictionFile` moved from `load.ts` to the root. A runner factory now takes `{ ai }` and a runner no longer receives `apiKey`. `KG_EVAL_API_KEY_ENV` is `KVOX_EVAL_API_KEY_ENV` (`KVOX_EVAL_OPENAI_API_KEY`). `assertOutsideRepo`'s option is `allowInside: <dir>` (the web app keeps its `allowSyntheticRunsDir` wrapper).
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| `Cannot find module '@marinoscar/kvox-eval'` in `apps/api` | Build the packages first: `npm run build:kvox-packages`. |
+| `reads its API key from KVOX_EVAL_OPENAI_API_KEY; it is not set` | Export the key in the shell that runs the command; it is never a flag. |
+| `Cannot find module '@electric-sql/pglite'` (or `@prisma/client`, `@nestjs/core`) on `kvox-eval ask` | Install the `ask-run` peers listed above; they are required. |
+| `the 'copilot' provider is not available yet` | The Copilot adapter arrives with #581; use `--provider openai` or `openai-compatible`. |
+| `real meeting data must never be written inside the repository` (exit 3) | Point `--real-dir` / `--out` outside the git work tree. |
+
+## Links
+
+- The package standard: [docs/PACKAGES.md](../../docs/PACKAGES.md)
+- The quality bar the report checks: [docs/specs/ontology.md](../../docs/specs/ontology.md) (§6)
+- Consuming the packages from KVox Desktop (which runs the same evaluation over its own binding of the model port): [docs/guides/kvox-desktop.md](../../docs/guides/kvox-desktop.md)
+- The model port: [packages/kvox-api/src/ports/README.md](../kvox-api/src/ports/README.md); its implementation: [packages/kvox-ai/README.md](../kvox-ai/README.md)
+- Dev commands: `npm run build -w @marinoscar/kvox-eval`, `npm run typecheck -w @marinoscar/kvox-eval`, `npm run test:run -w @marinoscar/kvox-eval`, `npm run docs -w @marinoscar/kvox-eval`

package/dist/ask/index.d.ts

@@ -0,0 +1,6 @@
+export { ASK_EVAL_CATEGORIES, AskEvalQuestionLoadError, askEvalExpectedSchema, askEvalQuestionFileSchema, askEvalQuestionSchema, loadAskEvalQuestions, parseAskEvalQuestions, selectAskEvalQuestions, } from './question-schema.js';
+export type { AskEvalCategory, AskEvalExpected, AskEvalQuestion } from './question-schema.js';
+export { askEvalReportSchema, buildAskEvalReport, formatAnswersMarkdown, formatAskEvalReport } from './report.js';
+export type { AskEvalReport } from './report.js';
+export { scoreAnswer, scoreAnswerMatch, scoreCitationValidity, scoreCited, scoreCiteFromHit, scoreEntityRecall, scoreSensitiveLeak, } from './scorer.js';
+export type { AskEvalScore, EvidenceSourceMap } from './scorer.js';

package/dist/ask/index.js

@@ -0,0 +1,28 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.scoreSensitiveLeak = exports.scoreEntityRecall = exports.scoreCiteFromHit = exports.scoreCited = exports.scoreCitationValidity = exports.scoreAnswerMatch = exports.scoreAnswer = exports.formatAskEvalReport = exports.formatAnswersMarkdown = exports.buildAskEvalReport = exports.askEvalReportSchema = exports.selectAskEvalQuestions = exports.parseAskEvalQuestions = exports.loadAskEvalQuestions = exports.askEvalQuestionSchema = exports.askEvalQuestionFileSchema = exports.askEvalExpectedSchema = exports.AskEvalQuestionLoadError = exports.ASK_EVAL_CATEGORIES = void 0;
+// The ask slice's public surface (re-exported by the package root): the PURE
+// half of the Ask evaluation. The run itself (seeding a graph, the real
+// `ask.respond` handler) needs the web app's database and stays there.
+var question_schema_js_1 = require("./question-schema.js");
+Object.defineProperty(exports, "ASK_EVAL_CATEGORIES", { enumerable: true, get: function () { return question_schema_js_1.ASK_EVAL_CATEGORIES; } });
+Object.defineProperty(exports, "AskEvalQuestionLoadError", { enumerable: true, get: function () { return question_schema_js_1.AskEvalQuestionLoadError; } });
+Object.defineProperty(exports, "askEvalExpectedSchema", { enumerable: true, get: function () { return question_schema_js_1.askEvalExpectedSchema; } });
+Object.defineProperty(exports, "askEvalQuestionFileSchema", { enumerable: true, get: function () { return question_schema_js_1.askEvalQuestionFileSchema; } });
+Object.defineProperty(exports, "askEvalQuestionSchema", { enumerable: true, get: function () { return question_schema_js_1.askEvalQuestionSchema; } });
+Object.defineProperty(exports, "loadAskEvalQuestions", { enumerable: true, get: function () { return question_schema_js_1.loadAskEvalQuestions; } });
+Object.defineProperty(exports, "parseAskEvalQuestions", { enumerable: true, get: function () { return question_schema_js_1.parseAskEvalQuestions; } });
+Object.defineProperty(exports, "selectAskEvalQuestions", { enumerable: true, get: function () { return question_schema_js_1.selectAskEvalQuestions; } });
+var report_js_1 = require("./report.js");
+Object.defineProperty(exports, "askEvalReportSchema", { enumerable: true, get: function () { return report_js_1.askEvalReportSchema; } });
+Object.defineProperty(exports, "buildAskEvalReport", { enumerable: true, get: function () { return report_js_1.buildAskEvalReport; } });
+Object.defineProperty(exports, "formatAnswersMarkdown", { enumerable: true, get: function () { return report_js_1.formatAnswersMarkdown; } });
+Object.defineProperty(exports, "formatAskEvalReport", { enumerable: true, get: function () { return report_js_1.formatAskEvalReport; } });
+var scorer_js_1 = require("./scorer.js");
+Object.defineProperty(exports, "scoreAnswer", { enumerable: true, get: function () { return scorer_js_1.scoreAnswer; } });
+Object.defineProperty(exports, "scoreAnswerMatch", { enumerable: true, get: function () { return scorer_js_1.scoreAnswerMatch; } });
+Object.defineProperty(exports, "scoreCitationValidity", { enumerable: true, get: function () { return scorer_js_1.scoreCitationValidity; } });
+Object.defineProperty(exports, "scoreCited", { enumerable: true, get: function () { return scorer_js_1.scoreCited; } });
+Object.defineProperty(exports, "scoreCiteFromHit", { enumerable: true, get: function () { return scorer_js_1.scoreCiteFromHit; } });
+Object.defineProperty(exports, "scoreEntityRecall", { enumerable: true, get: function () { return scorer_js_1.scoreEntityRecall; } });
+Object.defineProperty(exports, "scoreSensitiveLeak", { enumerable: true, get: function () { return scorer_js_1.scoreSensitiveLeak; } });

package/dist/ask/question-schema.d.ts

@@ -0,0 +1,121 @@
+import { z } from 'zod';
+/**
+ * Every category a question may be filed under. Coverage counts live in the README.
+ *
+ * @stability experimental
+ */
+export declare const ASK_EVAL_CATEGORIES: readonly ["lookup", "relationship", "temporal", "as_of", "commitments", "decisions", "multi_hop", "negative"];
+/**
+ * One of {@link ASK_EVAL_CATEGORIES}.
+ *
+ * @stability experimental
+ */
+export type AskEvalCategory = (typeof ASK_EVAL_CATEGORIES)[number];
+/**
+ * What a correct answer to one question must (and must not) contain: cited entities,
+ * `answerAll` groups, `answerNone` strings (a sensitive fact's own text belongs here),
+ * `citeFrom` sources and the `notFound` rule.
+ *
+ * @stability experimental
+ */
+export declare const askEvalExpectedSchema: z.ZodObject<{
+    entities: z.ZodDefault<z.ZodArray<z.ZodString>>;
+    answerAll: z.ZodDefault<z.ZodArray<z.ZodArray<z.ZodString>>>;
+    answerNone: z.ZodDefault<z.ZodArray<z.ZodString>>;
+    citeFrom: z.ZodDefault<z.ZodArray<z.ZodString>>;
+    notFound: z.ZodDefault<z.ZodBoolean>;
+}, z.core.$strip>;
+/**
+ * A parsed {@link askEvalExpectedSchema}.
+ *
+ * @stability experimental
+ */
+export type AskEvalExpected = z.infer<typeof askEvalExpectedSchema>;
+/**
+ * One golden Ask question: a permanent `qNN` id, a category, the fixtures it is grounded
+ * in (provenance only), an optional scope entity, the question and its expectations.
+ *
+ * @stability experimental
+ */
+export declare const askEvalQuestionSchema: z.ZodObject<{
+    id: z.ZodString;
+    category: z.ZodEnum<{
+        lookup: "lookup";
+        relationship: "relationship";
+        temporal: "temporal";
+        as_of: "as_of";
+        commitments: "commitments";
+        decisions: "decisions";
+        multi_hop: "multi_hop";
+        negative: "negative";
+    }>;
+    fixtures: z.ZodArray<z.ZodString>;
+    scopeEntity: z.ZodDefault<z.ZodNullable<z.ZodString>>;
+    question: z.ZodString;
+    expected: z.ZodObject<{
+        entities: z.ZodDefault<z.ZodArray<z.ZodString>>;
+        answerAll: z.ZodDefault<z.ZodArray<z.ZodArray<z.ZodString>>>;
+        answerNone: z.ZodDefault<z.ZodArray<z.ZodString>>;
+        citeFrom: z.ZodDefault<z.ZodArray<z.ZodString>>;
+        notFound: z.ZodDefault<z.ZodBoolean>;
+    }, z.core.$strip>;
+}, z.core.$strip>;
+/**
+ * A parsed {@link askEvalQuestionSchema}.
+ *
+ * @stability experimental
+ */
+export type AskEvalQuestion = z.infer<typeof askEvalQuestionSchema>;
+/**
+ * The whole committed file: a bare array of questions.
+ *
+ * @stability experimental
+ */
+export declare const askEvalQuestionFileSchema: z.ZodArray<z.ZodObject<{
+    id: z.ZodString;
+    category: z.ZodEnum<{
+        lookup: "lookup";
+        relationship: "relationship";
+        temporal: "temporal";
+        as_of: "as_of";
+        commitments: "commitments";
+        decisions: "decisions";
+        multi_hop: "multi_hop";
+        negative: "negative";
+    }>;
+    fixtures: z.ZodArray<z.ZodString>;
+    scopeEntity: z.ZodDefault<z.ZodNullable<z.ZodString>>;
+    question: z.ZodString;
+    expected: z.ZodObject<{
+        entities: z.ZodDefault<z.ZodArray<z.ZodString>>;
+        answerAll: z.ZodDefault<z.ZodArray<z.ZodArray<z.ZodString>>>;
+        answerNone: z.ZodDefault<z.ZodArray<z.ZodString>>;
+        citeFrom: z.ZodDefault<z.ZodArray<z.ZodString>>;
+        notFound: z.ZodDefault<z.ZodBoolean>;
+    }, z.core.$strip>;
+}, z.core.$strip>>;
+/**
+ * The question file cannot be used: not JSON, a schema violation, a duplicate or unknown id.
+ *
+ * @stability experimental
+ */
+export declare class AskEvalQuestionLoadError extends Error {
+}
+/**
+ * Parses and validates the question file's raw JSON. Throws on any schema violation.
+ *
+ * @stability experimental
+ */
+export declare function parseAskEvalQuestions(raw: unknown): AskEvalQuestion[];
+/**
+ * Loads and validates the committed question file from disk.
+ *
+ * @stability experimental
+ */
+export declare function loadAskEvalQuestions(path: string): AskEvalQuestion[];
+/**
+ * `--only q01,q07` / `--category lookup`: both narrow; neither given keeps all.
+ *
+ * @stability experimental
+ */
+export declare function selectAskEvalQuestions(questions: AskEvalQuestion[], only: string[] | null, category: string | null): AskEvalQuestion[];

package/dist/ask/question-schema.js

@@ -0,0 +1,167 @@
+"use strict";
+// =============================================================================
+// The Ask golden question set: schema (issue #382, epic #348)
+// =============================================================================
+//
+// One JSON file, `packages/kvox-eval/fixtures/kg-golden/ask-questions.json`, holding
+// every question the Ask-agent evaluation asks against the seeded gold graph
+// (built from #362's synthetic meetings — never real data, per CLAUDE.md's
+// "synthetic fixtures only, ever"). A question's `id` is PERMANENT once
+// committed, exactly like a fixture's `id` (#362's own rule): retire a
+// question by noting it in the README, never by renumbering, so a report is
+// comparable across runs.
+//
+// `fixtures` documents PROVENANCE ONLY — the seeder always loads every fixture
+// under #362's `packages/kvox-eval/fixtures/kg-golden/meetings/`, so a question is
+// never scoped to a subset at seed time. It exists so a reviewer (or a script
+// checking referential integrity, `ask-eval-questions.spec.ts`) can see at a
+// glance which meeting(s) a question's expectations are grounded in.
+// =============================================================================
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.AskEvalQuestionLoadError = exports.askEvalQuestionFileSchema = exports.askEvalQuestionSchema = exports.askEvalExpectedSchema = exports.ASK_EVAL_CATEGORIES = void 0;
+exports.parseAskEvalQuestions = parseAskEvalQuestions;
+exports.loadAskEvalQuestions = loadAskEvalQuestions;
+exports.selectAskEvalQuestions = selectAskEvalQuestions;
+const zod_1 = require("zod");
+/**
+ * Every category a question may be filed under. Coverage counts live in the README.
+ *
+ * @stability experimental
+ */
+exports.ASK_EVAL_CATEGORIES = [
+    'lookup',
+    'relationship',
+    'temporal',
+    'as_of',
+    'commitments',
+    'decisions',
+    'multi_hop',
+    'negative',
+];
+/**
+ * What a correct answer to one question must (and must not) contain: cited entities,
+ * `answerAll` groups, `answerNone` strings (a sensitive fact's own text belongs here),
+ * `citeFrom` sources and the `notFound` rule.
+ *
+ * @stability experimental
+ */
+exports.askEvalExpectedSchema = zod_1.z.object({
+    /**
+     * Entity labels that must be cited: either directly (`[^entN]`) or as the
+     * subject of a cited evidence row (`[^evN]`/a resolved `[^itmN]`/`[^relN]`)
+     * whose citation carries this label. Case-insensitive exact match against
+     * the citation's own `label` field — never a substring, since a label is a
+     * whole name.
+     */
+    entities: zod_1.z.array(zod_1.z.string()).default([]),
+    /**
+     * Every inner group must be satisfied by a case-insensitive SUBSTRING match
+     * somewhere in the answer text; within a group, any one alternative is
+     * enough (synonyms/phrasing variants). `[["Sarah Chen","Sarah"]]` is one
+     * group with two acceptable spellings.
+     */
+    answerAll: zod_1.z.array(zod_1.z.array(zod_1.z.string()).min(1)).default([]),
+    /** None of these substrings may appear anywhere in the answer, case-insensitively. */
+    answerNone: zod_1.z.array(zod_1.z.string()).default([]),
+    /**
+     * Segment ids (`"m04-s012"`, matching a fixture's own segment id verbatim)
+     * or `"mNN#note"` for a meeting's note. At least one VALID citation in the
+     * answer must resolve (through the seeder's `idMap`) to one of these.
+     * Empty for a `notFound` question, which cites nothing.
+     */
+    citeFrom: zod_1.z.array(zod_1.z.string()).default([]),
+    /**
+     * A negative question: the agent must say it could not find an answer and
+     * must cite nothing valid. `answerNone` still applies — this is where a
+     * sensitive fact's own text belongs, so a leak is caught even though the
+     * question is "answered" with a refusal.
+     */
+    notFound: zod_1.z.boolean().default(false),
+});
+/**
+ * One golden Ask question: a permanent `qNN` id, a category, the fixtures it is grounded
+ * in (provenance only), an optional scope entity, the question and its expectations.
+ *
+ * @stability experimental
+ */
+exports.askEvalQuestionSchema = zod_1.z.object({
+    id: zod_1.z.string().regex(/^q\d{2,3}$/, 'must look like "q01" or "q123"'),
+    category: zod_1.z.enum(exports.ASK_EVAL_CATEGORIES),
+    /** Provenance only — see the file header. At least one fixture id (`"m01"`), even for a negative question about an absent entity. */
+    fixtures: zod_1.z.array(zod_1.z.string().regex(/^m\d{2,3}$/)).min(1),
+    /** The label of the entity to scope the conversation to, or `null` for an unscoped conversation. */
+    scopeEntity: zod_1.z.string().nullable().default(null),
+    question: zod_1.z.string().min(5),
+    expected: exports.askEvalExpectedSchema,
+});
+/**
+ * The whole committed file: a bare array of questions.
+ *
+ * @stability experimental
+ */
+exports.askEvalQuestionFileSchema = zod_1.z.array(exports.askEvalQuestionSchema);
+/**
+ * The question file cannot be used: not JSON, a schema violation, a duplicate or unknown id.
+ *
+ * @stability experimental
+ */
+class AskEvalQuestionLoadError extends Error {
+}
+exports.AskEvalQuestionLoadError = AskEvalQuestionLoadError;
+/**
+ * Parses and validates the question file's raw JSON. Throws on any schema violation.
+ *
+ * @stability experimental
+ */
+function parseAskEvalQuestions(raw) {
+    const parsed = exports.askEvalQuestionFileSchema.safeParse(raw);
+    if (!parsed.success) {
+        const issues = parsed.error.issues
+            .slice(0, 8)
+            .map((i) => `${i.path.join('.')}: ${i.message}`)
+            .join('; ');
+        throw new AskEvalQuestionLoadError(`ask-questions.json does not match askEvalQuestionSchema — ${issues}`);
+    }
+    const seen = new Set();
+    for (const q of parsed.data) {
+        if (seen.has(q.id))
+            throw new AskEvalQuestionLoadError(`duplicate question id ${q.id}`);
+        seen.add(q.id);
+    }
+    return parsed.data;
+}
+/**
+ * Loads and validates the committed question file from disk.
+ *
+ * @stability experimental
+ */
+function loadAskEvalQuestions(path) {
+    // Lazy so callers that already have the parsed JSON (e.g. a spec that reads
+    // it once) never need Node's `fs` in their own import graph.
+    const { readFileSync } = require('fs');
+    let raw;
+    try {
+        raw = JSON.parse(readFileSync(path, 'utf8'));
+    }
+    catch (err) {
+        throw new AskEvalQuestionLoadError(`${path}: not valid JSON (${err.message})`);
+    }
+    return parseAskEvalQuestions(raw);
+}
+/**
+ * `--only q01,q07` / `--category lookup`: both narrow; neither given keeps all.
+ *
+ * @stability experimental
+ */
+function selectAskEvalQuestions(questions, only, category) {
+    let out = questions;
+    if (only && only.length > 0) {
+        const unknown = only.filter((id) => !questions.some((q) => q.id === id));
+        if (unknown.length > 0)
+            throw new AskEvalQuestionLoadError(`unknown question id(s): ${unknown.join(', ')}`);
+        out = out.filter((q) => only.includes(q.id));
+    }
+    if (category)
+        out = out.filter((q) => q.category === category);
+    return out;
+}

package/dist/ask/report.d.ts

@@ -0,0 +1,88 @@
+import { z } from 'zod';
+import type { AskEvalScore } from './scorer.js';
+/**
+ * The Ask evaluation report: the per-category rollup, the overall numbers, the two gates
+ * (citation validity 1.000, zero sensitive leaks), the answer-match threshold and every
+ * question's score. The JSON written as `report.json`.
+ *
+ * @stability experimental
+ */
+export declare const askEvalReportSchema: z.ZodObject<{
+    runAt: z.ZodString;
+    model: z.ZodNullable<z.ZodString>;
+    questions: z.ZodNumber;
+    categories: z.ZodArray<z.ZodObject<{
+        category: z.ZodString;
+        count: z.ZodNumber;
+        answerMatch: z.ZodNumber;
+        cited: z.ZodNumber;
+        entityRecall: z.ZodNumber;
+        meanSteps: z.ZodNumber;
+        meanToolCalls: z.ZodNumber;
+        meanPromptTokens: z.ZodNumber;
+        meanCompletionTokens: z.ZodNumber;
+        meanMs: z.ZodNullable<z.ZodNumber>;
+    }, z.core.$strip>>;
+    overall: z.ZodObject<{
+        answerMatch: z.ZodNumber;
+        citationValidity: z.ZodNumber;
+        sensitiveLeaks: z.ZodNumber;
+    }, z.core.$strip>;
+    gatesPassed: z.ZodBoolean;
+    answerMatchPassed: z.ZodBoolean;
+    minMatch: z.ZodNumber;
+    perQuestion: z.ZodArray<z.ZodObject<{
+        id: z.ZodString;
+        category: z.ZodString;
+        citationValidity: z.ZodNumber;
+        cited: z.ZodBoolean;
+        citeFromHit: z.ZodBoolean;
+        answerMatch: z.ZodBoolean;
+        entityRecall: z.ZodNumber;
+        sensitiveLeak: z.ZodBoolean;
+        steps: z.ZodNumber;
+        toolCalls: z.ZodNumber;
+        promptTokens: z.ZodNumber;
+        completionTokens: z.ZodNumber;
+        ms: z.ZodNullable<z.ZodNumber>;
+        notes: z.ZodArray<z.ZodString>;
+    }, z.core.$strip>>;
+}, z.core.$strip>;
+/**
+ * A parsed {@link askEvalReportSchema}.
+ *
+ * @stability experimental
+ */
+export type AskEvalReport = z.infer<typeof askEvalReportSchema>;
+/**
+ * Rolls the per-question scores of one run up into a report.
+ *
+ * @param scores - one score per answered question.
+ * @param meta - the model, the answer-match threshold and (for a test) the run time.
+ * @returns the report.
+ *
+ * @stability experimental
+ */
+export declare function buildAskEvalReport(scores: AskEvalScore[], meta: {
+    model: string | null;
+    minMatch: number;
+    runAt?: Date;
+}): AskEvalReport;
+/**
+ * The report as the fixed-column table `ask:eval` prints, then the gates and the failing questions.
+ *
+ * @param r - the report.
+ * @returns the table.
+ *
+ * @stability experimental
+ */
+export declare function formatAskEvalReport(r: AskEvalReport): string;
+/**
+ * `answers.md`: every question, its answer text and its per-question numbers — for reading, not for scripts.
+ *
+ * @stability experimental
+ */
+export declare function formatAnswersMarkdown(report: AskEvalReport, answers: ReadonlyMap<string, {
+    question: string;
+    content: string;
+}>): string;