@gaunt-sloth/batch 2.0.0-alpha.19 → 2.0.0-alpha.20

package/README.md

@@ -0,0 +1,140 @@
+# @gaunt-sloth/batch
+
+The batch / eval / workflow runtime for Gaunt Sloth. You reach it three ways: through the commands
+the `gaunt-sloth` app registers (`gth batch`, `gth eval`, `gth workflow`), through this package's own
+`gth-batch` binary — a thin standalone matrix runner for shell pipelines (see [below](#pipeline-runner-gth-batch)) —
+or by importing the package to embed the runtime. Install `gaunt-sloth` for the full command set;
+install `@gaunt-sloth/batch` for the standalone binary, or depend on it to embed the runtime.
+
+## Grade a suite of prompts (`gth eval`)
+
+You want to check that your agent still passes a set of graded cases before you ship a prompt or
+config change. Write a suite YAML, install the CLI, then run `gth eval` — it exits `0` iff every
+case passes ("pytest for prompts"). The commands run on `@gaunt-sloth/batch`, but they ship in the
+`gaunt-sloth` app, so that is what you install:
+
+```bash
+npm install -g gaunt-sloth@alpha
+```
+
+Write `prompts.eval.yaml`:
+
+```yaml
+target: { type: gth-agent }
+cases:
+  - id: greets-and-signs-off
+    prompt: "Greet the user, then say goodbye."
+    must_contain: ["hello", "goodbye"]
+  - id: summarizes-as-json
+    prompt: "Summarize the last release as a JSON object with a title field."
+    must_not_contain: ["Sorry"]
+    judge: "Returns a single JSON object with a non-empty title field."
+    pass_threshold: 7
+```
+
+Run it:
+
+```bash
+gth eval prompts.eval.yaml
+```
+
+A case passes when its deterministic checks hold (`must_contain` / `must_not_contain` /
+`should_contain_any`) and, if a `judge` rubric is set, the LLM judge rates the answer at or above
+`pass_threshold` (0–10 scale, suite default `6`). `gth eval` prints a `PASS`/`FAIL` line per case
+plus a suite total, writes structured per-case JSON and a `results.json` summary to a timestamped
+output dir (override with `-o <dir>`), and exits non-zero if any case failed. Add `-j <n>` to cap
+in-flight cases.
+
+### Examples
+
+```bash
+# Run one prompt-executable over a matrix of two models × the rows of a CSV.
+gth batch summarize.md --over inputs.csv --models gemini-2.5-pro,gemini-3.5-flash -j 4
+
+# Run a local orchestration script, passing it JSON as ctx.args.
+gth workflow rank-models.mjs --args '{"topic":"robotics"}'
+```
+
+`gth batch <script.md>` runs a markdown prompt-executable over a matrix of models (`--models a,b,c`,
+comma-separated; omit to use the configured model) and/or content-bound input rows (`--over
+<file.csv|file.jsonl>` — one cell per row, with `{{field}}` placeholders bound from the row). It
+writes the same structured per-cell output as `eval` but — unlike `eval` — exits `0` as long as the
+cells *ran*: a poor answer is not a harness failure. `-j <n>` caps concurrency, `--retry <n>` retries
+a failed cell (default `0`), `-o <dir>` sets the output dir.
+
+`gth workflow <script.mjs>` runs a local ESM script whose default export is `async (ctx) => result`;
+the return value is printed (a string as-is, anything else as pretty JSON), and `--args <json>` is
+handed to the script as `ctx.args`. The script is arbitrary local ESM run with full Node privileges
+(it can read files and spawn processes) — run only scripts you trust, as you would any local script.
+
+## Pipeline runner (`gth-batch`)
+
+`gth batch` above is the full command, wired into the `gaunt-sloth` app. When you want the matrix
+runtime on its own — inside a shell pipeline, without installing the whole CLI — this package ships a
+thin binary, `gth-batch`, that runs the same matrix and emits the **same per-cell records** `gth batch`
+produces, one JSON object per line (JSONL) on stdout:
+
+```bash
+# Fan a prompt-executable over rows piped in as JSON (or YAML), across two models.
+echo '[{"topic":"gears"},{"topic":"levers"}]' \
+  | gth-batch explain.md --models gemini-2.5-flash,gemini-2.5-pro -j 4 \
+  | jq -c 'select(.ok) | {id, model, answer}'
+```
+
+It takes the script path as its argument and the input axis as **inline `--over` data** (a JSON/YAML
+array of row objects) **or on stdin** — a pipeline already has the shell to produce the data, so
+unlike `gth batch` there is no CSV/JSONL file path. `--models a,b,c`, `-j <n>` (concurrency) and
+`--retry <n>` behave as in `gth batch`.
+
+Each stdout line is the full `CellResult` (`id`, `model`, `inputIndex`, `inputRow`, `ok`, `answer`,
+`tokensInput`/`tokensOutput`, `tools`, `durationMs`, `retries`) — stdout is kept a clean data channel
+(all progress/errors go to stderr), so it pipes straight into `jq`, `grep`, or a file. Following the
+`gth batch` exit-code contract, `gth-batch` exits `0` as long as the cells *ran* — a poor or failed
+cell is recorded as `"ok": false` in its line, not reflected in the exit code; only a harness error
+(bad arguments, an unreadable script, malformed `--over`, or a config failure) exits non-zero.
+
+It resolves your model from the same `.gsloth.config.*` as the rest of Gaunt Sloth (discovered from
+the working directory), so configure a provider there first.
+
+## Programmatic use
+
+Depend on `@gaunt-sloth/batch` to drive the same engine from your own code. Every module is exported
+via the `./*.js` subpath map, matching the other `@gaunt-sloth/*` packages:
+
+```js
+import { runEvalSuite } from '@gaunt-sloth/batch/evalRunner.js';
+```
+
+The public API (see the package's `index.ts`) groups by command:
+
+- **Matrix** (`gth batch`): `buildMatrix`, `bindCellContent`, `parseOverFile`, `runBatchMatrix`,
+  `buildBatchSummary`, `writeBatchOutput`, `DEFAULT_CONCURRENCY`.
+- **Eval** (`gth eval`): `parseEvalSuite`, `runDeterministicChecks`, `judgeEvalCase`, `runEvalSuite`,
+  `writeEvalOutput`, `EvalVerdictSchema`, `DEFAULT_EVAL_PASS_THRESHOLD`.
+- **Workflow** (`gth workflow`): `runWorkflow`.
+
+The corresponding TypeScript types (`MatrixCell`, `BatchSummary`, `CellResult`, `EvalSuite`,
+`EvalCaseResult`, `WorkflowContext`, …) are exported alongside them.
+
+## Dependencies
+
+- `@gaunt-sloth/core` — config, provider factory, the lean single-shot runtime, `askStructured`
+- `@gaunt-sloth/agent` — the resolvers and lean agent factory the batch/eval/workflow cell path runs on
+- `@langchain/core`, `yaml`, `zod`
+
+## Related packages
+
+- [`@gaunt-sloth/core`](https://www.npmjs.com/package/@gaunt-sloth/core) — Core utilities, config,
+  and agent infrastructure
+  ([source](https://github.com/pukeko-robotics/gaunt-sloth/tree/main/packages/core))
+- [`@gaunt-sloth/agent`](https://www.npmjs.com/package/@gaunt-sloth/agent) — Agent runtime: built-in
+  tools, filesystem toolkit, middleware registry, API server, AG-UI, MCP, and A2A integration
+  ([source](https://github.com/pukeko-robotics/gaunt-sloth/tree/main/packages/agent))
+- [`@gaunt-sloth/review`](https://www.npmjs.com/package/@gaunt-sloth/review) — Review and Q&A modules
+  with standalone CLI
+  ([source](https://github.com/pukeko-robotics/gaunt-sloth/tree/main/packages/review))
+- [`@gaunt-sloth/batch`](https://www.npmjs.com/package/@gaunt-sloth/batch) — Batch / eval / workflow
+  runtime (this package)
+  ([source](https://github.com/pukeko-robotics/gaunt-sloth/tree/main/packages/batch))
+- [`gaunt-sloth`](https://www.npmjs.com/package/gaunt-sloth) — Main CLI application
+  ([source](https://github.com/pukeko-robotics/gaunt-sloth/tree/main/packages/app))

package/dist/bin.d.ts

@@ -0,0 +1,13 @@
+#!/usr/bin/env node
+/**
+ * @module bin
+ *
+ * BATCH-9 — the `gth-batch` executable. A minimal shebang wrapper around {@link runBatchCli}
+ * (pipelineCli.ts) whose only job beyond delegating is to keep **stdout a clean machine channel**: the
+ * batch runtime's human/status/streaming output all lands on `process.stdout` (via
+ * `console.*`/`ProgressIndicator`/`stream()`), so this redirects `process.stdout.write` to stderr
+ * for the duration of the run. The JSONL cell records are written straight to fd 1 inside
+ * `runBatchCli` (`fs.writeSync`), bypassing this redirect — the same "protocol channel" discipline
+ * `packages/app/cli.js` uses for the ACP stdio channel.
+ */
+export {};

package/dist/bin.js

@@ -0,0 +1,28 @@
+#!/usr/bin/env node
+/**
+ * @module bin
+ *
+ * BATCH-9 — the `gth-batch` executable. A minimal shebang wrapper around {@link runBatchCli}
+ * (pipelineCli.ts) whose only job beyond delegating is to keep **stdout a clean machine channel**: the
+ * batch runtime's human/status/streaming output all lands on `process.stdout` (via
+ * `console.*`/`ProgressIndicator`/`stream()`), so this redirects `process.stdout.write` to stderr
+ * for the duration of the run. The JSONL cell records are written straight to fd 1 inside
+ * `runBatchCli` (`fs.writeSync`), bypassing this redirect — the same "protocol channel" discipline
+ * `packages/app/cli.js` uses for the ACP stdio channel.
+ */
+import { runBatchCli } from '#src/pipelineCli.js';
+const originalStdoutWrite = process.stdout.write.bind(process.stdout);
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+process.stdout.write = ((chunk, encoding, cb) => process.stderr.write(chunk, encoding, cb));
+runBatchCli(process.argv.slice(2))
+    .then((code) => {
+    process.exitCode = code;
+})
+    .catch((err) => {
+    process.stderr.write(`gth-batch: ${err instanceof Error ? err.message : String(err)}\n`);
+    process.exitCode = 1;
+})
+    .finally(() => {
+    process.stdout.write = originalStdoutWrite;
+});
+//# sourceMappingURL=bin.js.map

package/dist/bin.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"bin.js","sourceRoot":"","sources":["../src/bin.ts"],"names":[],"mappings":";AACA;;;;;;;;;;GAUG;AAEH,OAAO,EAAE,WAAW,EAAE,MAAM,qBAAqB,CAAC;AAElD,MAAM,mBAAmB,GAAG,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;AACtE,8DAA8D;AAC9D,OAAO,CAAC,MAAM,CAAC,KAAK,GAAG,CAAC,CAAC,KAAU,EAAE,QAAc,EAAE,EAAQ,EAAW,EAAE,CACxE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,EAAE,QAAQ,EAAE,EAAE,CAAC,CAAgC,CAAC;AAE5E,WAAW,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;KAC/B,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE;IACb,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;AAC1B,CAAC,CAAC;KACD,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE;IACb,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,cAAc,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IACzF,OAAO,CAAC,QAAQ,GAAG,CAAC,CAAC;AACvB,CAAC,CAAC;KACD,OAAO,CAAC,GAAG,EAAE;IACZ,OAAO,CAAC,MAAM,CAAC,KAAK,GAAG,mBAAmB,CAAC;AAC7C,CAAC,CAAC,CAAC"}

package/dist/deterministicChecks.d.ts

@@ -1,16 +1,48 @@
-import type { DeterministicCheckResult, EvalCase } from '#src/evalTypes.js';
+import type { DeterministicCheckResult, EvalExpectation } from '#src/evalTypes.js';
 /**
- * Case-insensitive substring checks over an SUT answer. Ported, not reinvented, from the field
- * user's proven `deterministic()` function (`docs/batch-eval-user-requirements.md` Appendix A):
+ * Resolve a **minimal** JSON path (BATCH-10) against an already-parsed JSON value.
+ *
+ * Supported subset (deliberately tiny, dependency-free — NOT full JSONPath):
+ * - an optional leading `$` and/or leading `.` (`$.items[0]`, `.items[0]`, and `items[0]` are
+ *   equivalent);
+ * - dot-separated object keys (`data.rows`);
+ * - `[<int>]` array indexing (`items[0]`, `rows[2].id`).
+ *
+ * No wildcards, filters, slices, quoted keys, or negative indices. A segment made only of digits
+ * is treated as an array index; any other segment is an object key. Returns `{ found: false }` for
+ * any missing key, out-of-range index, or type mismatch (indexing a non-array, keying a non-object)
+ * rather than throwing.
+ */
+export declare function resolveJsonPath(root: unknown, path: string): {
+    found: boolean;
+    value?: unknown;
+};
+/**
+ * Deterministic, answer-based checks over an SUT answer. The substring family is ported, not
+ * reinvented, from the field user's proven `deterministic()` function
+ * (`docs/batch-eval-user-requirements.md` Appendix A); BATCH-10 adds regex and minimal JSON-path
+ * assertions alongside them (all over the *answer* — tool-trace assertions live in
+ * `#src/toolChecks.js`, since they read the tool names, not the answer):
  *
  * - `mustContain` — every entry must appear (case-insensitive substring); each miss is reported.
  * - `mustNotContain` — no entry may appear; each hit is reported.
  * - `shouldContainAny` — at least one entry must appear; reported as a single combined failure
  *   (not one per missing option — the check is "at least one", so there is only one way to fail
  *   it) when none do.
+ * - `mustMatch` — every regex must match the raw answer (no case-folding: the pattern owns its
+ *   flags); each miss is reported as `answer did not match /…/`.
+ * - `mustNotMatch` — no regex may match; each hit is reported as `answer matched forbidden /…/`.
+ * - `jsonPath` — the answer is parsed as JSON and each path assertion (`equals`/`contains`) is
+ *   checked; a non-JSON answer fails the whole group once.
+ *
+ * Failures are ordered substring → regex → json_path, so a substring-only case produces exactly the
+ * same output it did before BATCH-10. An expectation with every array empty trivially passes (no
+ * checks to fail) — the suite parser (`#src/evalSuite.js`) is what enforces that a block has *some*
+ * check or a judge rubric; this function itself has no opinion on that.
  *
- * A case with all three arrays empty trivially passes (no checks to fail) — the suite parser
- * (`#src/evalSuite.js`) is what enforces that a case has *some* check or a judge rubric; this
- * function itself has no opinion on that.
+ * BATCH-12: grades one {@link EvalExpectation} block (the assertion bundle) — a flat case's single
+ * unscoped block or one of a matrix case's identity-scoped `expect:` blocks. The signature is the
+ * same `Pick`/`Partial` shape as before, just re-based on `EvalExpectation` (identical field names),
+ * so the answer-check behavior is unchanged.
  */
-export declare function runDeterministicChecks(answer: string, evalCase: Pick<EvalCase, 'mustContain' | 'mustNotContain' | 'shouldContainAny'>): DeterministicCheckResult;
+export declare function runDeterministicChecks(answer: string, expectation: Pick<EvalExpectation, 'mustContain' | 'mustNotContain' | 'shouldContainAny'> & Partial<Pick<EvalExpectation, 'mustMatch' | 'mustNotMatch' | 'jsonPath'>>): DeterministicCheckResult;

package/dist/deterministicChecks.js

@@ -1,34 +1,143 @@
+import { isDeepStrictEqual } from 'node:util';
+/** Render a compiled pattern back as `/source/flags` for failure messages. */
+function formatRegex(re) {
+    return `/${re.source}/${re.flags}`;
+}
 /**
- * Case-insensitive substring checks over an SUT answer. Ported, not reinvented, from the field
- * user's proven `deterministic()` function (`docs/batch-eval-user-requirements.md` Appendix A):
+ * Resolve a **minimal** JSON path (BATCH-10) against an already-parsed JSON value.
+ *
+ * Supported subset (deliberately tiny, dependency-free — NOT full JSONPath):
+ * - an optional leading `$` and/or leading `.` (`$.items[0]`, `.items[0]`, and `items[0]` are
+ *   equivalent);
+ * - dot-separated object keys (`data.rows`);
+ * - `[<int>]` array indexing (`items[0]`, `rows[2].id`).
+ *
+ * No wildcards, filters, slices, quoted keys, or negative indices. A segment made only of digits
+ * is treated as an array index; any other segment is an object key. Returns `{ found: false }` for
+ * any missing key, out-of-range index, or type mismatch (indexing a non-array, keying a non-object)
+ * rather than throwing.
+ */
+export function resolveJsonPath(root, path) {
+    let p = path.trim();
+    if (p.startsWith('$'))
+        p = p.slice(1);
+    if (p.startsWith('.'))
+        p = p.slice(1);
+    // Normalize `[n]` index syntax into dot segments so a single split handles both forms.
+    p = p.replace(/\[(\d+)\]/g, '.$1');
+    const segments = p.split('.').filter((segment) => segment.length > 0);
+    let current = root;
+    for (const segment of segments) {
+        if (current === null || typeof current !== 'object') {
+            return { found: false };
+        }
+        if (/^\d+$/.test(segment)) {
+            const index = Number(segment);
+            if (!Array.isArray(current) || index >= current.length) {
+                return { found: false };
+            }
+            current = current[index];
+        }
+        else {
+            if (Array.isArray(current) || !Object.prototype.hasOwnProperty.call(current, segment)) {
+                return { found: false };
+            }
+            current = current[segment];
+        }
+    }
+    return { found: true, value: current };
+}
+/** Run the `json_path` assertions over the answer parsed as JSON. If the answer is not valid JSON,
+ * the whole group fails with a single reason (not one per entry) and no entry is evaluated. */
+function runJsonPathChecks(answer, checks) {
+    if (checks.length === 0)
+        return [];
+    let root;
+    try {
+        root = JSON.parse(answer.trim());
+    }
+    catch {
+        return ['answer is not JSON (json_path checks require a JSON answer)'];
+    }
+    const failures = [];
+    for (const check of checks) {
+        const { found, value } = resolveJsonPath(root, check.path);
+        if (!found) {
+            failures.push(`json_path "${check.path}" did not resolve (no such path in answer)`);
+            continue;
+        }
+        if (check.contains !== undefined) {
+            if (typeof value !== 'string') {
+                failures.push(`json_path "${check.path}" is ${JSON.stringify(value)} (contains check requires a string)`);
+            }
+            else if (!value.includes(check.contains)) {
+                failures.push(`json_path "${check.path}" does not contain "${check.contains}"`);
+            }
+        }
+        else if (!isDeepStrictEqual(value, check.equals)) {
+            failures.push(`json_path "${check.path}" is ${JSON.stringify(value)}, expected ${JSON.stringify(check.equals)}`);
+        }
+    }
+    return failures;
+}
+/**
+ * Deterministic, answer-based checks over an SUT answer. The substring family is ported, not
+ * reinvented, from the field user's proven `deterministic()` function
+ * (`docs/batch-eval-user-requirements.md` Appendix A); BATCH-10 adds regex and minimal JSON-path
+ * assertions alongside them (all over the *answer* — tool-trace assertions live in
+ * `#src/toolChecks.js`, since they read the tool names, not the answer):
  *
  * - `mustContain` — every entry must appear (case-insensitive substring); each miss is reported.
  * - `mustNotContain` — no entry may appear; each hit is reported.
  * - `shouldContainAny` — at least one entry must appear; reported as a single combined failure
  *   (not one per missing option — the check is "at least one", so there is only one way to fail
  *   it) when none do.
+ * - `mustMatch` — every regex must match the raw answer (no case-folding: the pattern owns its
+ *   flags); each miss is reported as `answer did not match /…/`.
+ * - `mustNotMatch` — no regex may match; each hit is reported as `answer matched forbidden /…/`.
+ * - `jsonPath` — the answer is parsed as JSON and each path assertion (`equals`/`contains`) is
+ *   checked; a non-JSON answer fails the whole group once.
  *
- * A case with all three arrays empty trivially passes (no checks to fail) — the suite parser
- * (`#src/evalSuite.js`) is what enforces that a case has *some* check or a judge rubric; this
- * function itself has no opinion on that.
+ * Failures are ordered substring → regex → json_path, so a substring-only case produces exactly the
+ * same output it did before BATCH-10. An expectation with every array empty trivially passes (no
+ * checks to fail) — the suite parser (`#src/evalSuite.js`) is what enforces that a block has *some*
+ * check or a judge rubric; this function itself has no opinion on that.
+ *
+ * BATCH-12: grades one {@link EvalExpectation} block (the assertion bundle) — a flat case's single
+ * unscoped block or one of a matrix case's identity-scoped `expect:` blocks. The signature is the
+ * same `Pick`/`Partial` shape as before, just re-based on `EvalExpectation` (identical field names),
+ * so the answer-check behavior is unchanged.
  */
-export function runDeterministicChecks(answer, evalCase) {
+export function runDeterministicChecks(answer, expectation) {
     const text = answer.toLowerCase();
     const failures = [];
-    for (const needle of evalCase.mustContain) {
+    for (const needle of expectation.mustContain) {
         if (!text.includes(needle.toLowerCase())) {
             failures.push(`missing "${needle}"`);
         }
     }
-    for (const needle of evalCase.mustNotContain) {
+    for (const needle of expectation.mustNotContain) {
         if (text.includes(needle.toLowerCase())) {
             failures.push(`forbidden "${needle}"`);
         }
     }
-    if (evalCase.shouldContainAny.length > 0 &&
-        !evalCase.shouldContainAny.some((needle) => text.includes(needle.toLowerCase()))) {
-        failures.push(`none of [${evalCase.shouldContainAny.join(' | ')}]`);
+    if (expectation.shouldContainAny.length > 0 &&
+        !expectation.shouldContainAny.some((needle) => text.includes(needle.toLowerCase()))) {
+        failures.push(`none of [${expectation.shouldContainAny.join(' | ')}]`);
+    }
+    // Use `answer.search(re)` rather than `re.test(answer)`: a stored `RegExp` carrying the `g` flag
+    // is stateful via `lastIndex` under `.test()`, whereas `String.prototype.search` ignores it.
+    for (const re of expectation.mustMatch ?? []) {
+        if (answer.search(re) === -1) {
+            failures.push(`answer did not match ${formatRegex(re)}`);
+        }
+    }
+    for (const re of expectation.mustNotMatch ?? []) {
+        if (answer.search(re) !== -1) {
+            failures.push(`answer matched forbidden ${formatRegex(re)}`);
+        }
     }
+    failures.push(...runJsonPathChecks(answer, expectation.jsonPath ?? []));
     return { passed: failures.length === 0, failures };
 }
 //# sourceMappingURL=deterministicChecks.js.map

package/dist/deterministicChecks.js.map

@@ -1 +1 @@
-{"version":3,"file":"deterministicChecks.js","sourceRoot":"","sources":["../src/deterministicChecks.ts"],"names":[],"mappings":"AAEA;;;;;;;;;;;;;GAaG;AACH,MAAM,UAAU,sBAAsB,CACpC,MAAc,EACd,QAA+E;IAE/E,MAAM,IAAI,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC;IAClC,MAAM,QAAQ,GAAa,EAAE,CAAC;IAE9B,KAAK,MAAM,MAAM,IAAI,QAAQ,CAAC,WAAW,EAAE,CAAC;QAC1C,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;YACzC,QAAQ,CAAC,IAAI,CAAC,YAAY,MAAM,GAAG,CAAC,CAAC;QACvC,CAAC;IACH,CAAC;IAED,KAAK,MAAM,MAAM,IAAI,QAAQ,CAAC,cAAc,EAAE,CAAC;QAC7C,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;YACxC,QAAQ,CAAC,IAAI,CAAC,cAAc,MAAM,GAAG,CAAC,CAAC;QACzC,CAAC;IACH,CAAC;IAED,IACE,QAAQ,CAAC,gBAAgB,CAAC,MAAM,GAAG,CAAC;QACpC,CAAC,QAAQ,CAAC,gBAAgB,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC,EAChF,CAAC;QACD,QAAQ,CAAC,IAAI,CAAC,YAAY,QAAQ,CAAC,gBAAgB,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IACtE,CAAC;IAED,OAAO,EAAE,MAAM,EAAE,QAAQ,CAAC,MAAM,KAAK,CAAC,EAAE,QAAQ,EAAE,CAAC;AACrD,CAAC"}
+{"version":3,"file":"deterministicChecks.js","sourceRoot":"","sources":["../src/deterministicChecks.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,WAAW,CAAC;AAI9C,8EAA8E;AAC9E,SAAS,WAAW,CAAC,EAAU;IAC7B,OAAO,IAAI,EAAE,CAAC,MAAM,IAAI,EAAE,CAAC,KAAK,EAAE,CAAC;AACrC,CAAC;AAED;;;;;;;;;;;;;GAaG;AACH,MAAM,UAAU,eAAe,CAAC,IAAa,EAAE,IAAY;IACzD,IAAI,CAAC,GAAG,IAAI,CAAC,IAAI,EAAE,CAAC;IACpB,IAAI,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC;QAAE,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IACtC,IAAI,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC;QAAE,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IACtC,uFAAuF;IACvF,CAAC,GAAG,CAAC,CAAC,OAAO,CAAC,YAAY,EAAE,KAAK,CAAC,CAAC;IACnC,MAAM,QAAQ,GAAG,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IAEtE,IAAI,OAAO,GAAY,IAAI,CAAC;IAC5B,KAAK,MAAM,OAAO,IAAI,QAAQ,EAAE,CAAC;QAC/B,IAAI,OAAO,KAAK,IAAI,IAAI,OAAO,OAAO,KAAK,QAAQ,EAAE,CAAC;YACpD,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;QAC1B,CAAC;QACD,IAAI,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;YAC1B,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC;YAC9B,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,IAAI,KAAK,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;gBACvD,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;YAC1B,CAAC;YACD,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC;QAC3B,CAAC;aAAM,CAAC;YACN,IAAI,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,cAAc,CAAC,IAAI,CAAC,OAAO,EAAE,OAAO,CAAC,EAAE,CAAC;gBACtF,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;YAC1B,CAAC;YACD,OAAO,GAAI,OAAmC,CAAC,OAAO,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC;IACD,OAAO,EAAE,KAAK,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;AACzC,CAAC;AAED;+FAC+F;AAC/F,SAAS,iBAAiB,CAAC,MAAc,EAAE,MAAuB;IAChE,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,CAAC;IAEnC,IAAI,IAAa,CAAC;IAClB,IAAI,CAAC;QACH,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;IACnC,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,CAAC,6DAA6D,CAAC,CAAC;IACzE,CAAC;IAED,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QAC3B,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,CAAC,CAAC;QAC3D,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,QAAQ,CAAC,IAAI,CAAC,cAAc,KAAK,CAAC,IAAI,4CAA4C,CAAC,CAAC;YACpF,SAAS;QACX,CAAC;QACD,IAAI,KAAK,CAAC,QAAQ,KAAK,SAAS,EAAE,CAAC;YACjC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;gBAC9B,QAAQ,CAAC,IAAI,CACX,cAAc,KAAK,CAAC,IAAI,QAAQ,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,qCAAqC,CAC3F,CAAC;YACJ,CAAC;iBAAM,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC;gBAC3C,QAAQ,CAAC,IAAI,CAAC,cAAc,KAAK,CAAC,IAAI,uBAAuB,KAAK,CAAC,QAAQ,GAAG,CAAC,CAAC;YAClF,CAAC;QACH,CAAC;aAAM,IAAI,CAAC,iBAAiB,CAAC,KAAK,EAAE,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC;YACnD,QAAQ,CAAC,IAAI,CACX,cAAc,KAAK,CAAC,IAAI,QAAQ,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,cAAc,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAClG,CAAC;QACJ,CAAC;IACH,CAAC;IACD,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;GA2BG;AACH,MAAM,UAAU,sBAAsB,CACpC,MAAc,EACd,WAC2E;IAE3E,MAAM,IAAI,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC;IAClC,MAAM,QAAQ,GAAa,EAAE,CAAC;IAE9B,KAAK,MAAM,MAAM,IAAI,WAAW,CAAC,WAAW,EAAE,CAAC;QAC7C,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;YACzC,QAAQ,CAAC,IAAI,CAAC,YAAY,MAAM,GAAG,CAAC,CAAC;QACvC,CAAC;IACH,CAAC;IAED,KAAK,MAAM,MAAM,IAAI,WAAW,CAAC,cAAc,EAAE,CAAC;QAChD,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;YACxC,QAAQ,CAAC,IAAI,CAAC,cAAc,MAAM,GAAG,CAAC,CAAC;QACzC,CAAC;IACH,CAAC;IAED,IACE,WAAW,CAAC,gBAAgB,CAAC,MAAM,GAAG,CAAC;QACvC,CAAC,WAAW,CAAC,gBAAgB,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC,EACnF,CAAC;QACD,QAAQ,CAAC,IAAI,CAAC,YAAY,WAAW,CAAC,gBAAgB,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IACzE,CAAC;IAED,iGAAiG;IACjG,6FAA6F;IAC7F,KAAK,MAAM,EAAE,IAAI,WAAW,CAAC,SAAS,IAAI,EAAE,EAAE,CAAC;QAC7C,IAAI,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC;YAC7B,QAAQ,CAAC,IAAI,CAAC,wBAAwB,WAAW,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC;QAC3D,CAAC;IACH,CAAC;IAED,KAAK,MAAM,EAAE,IAAI,WAAW,CAAC,YAAY,IAAI,EAAE,EAAE,CAAC;QAChD,IAAI,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC;YAC7B,QAAQ,CAAC,IAAI,CAAC,4BAA4B,WAAW,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC;QAC/D,CAAC;IACH,CAAC;IAED,QAAQ,CAAC,IAAI,CAAC,GAAG,iBAAiB,CAAC,MAAM,EAAE,WAAW,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC,CAAC;IAExE,OAAO,EAAE,MAAM,EAAE,QAAQ,CAAC,MAAM,KAAK,CAAC,EAAE,QAAQ,EAAE,CAAC;AACrD,CAAC"}

package/dist/evalOutput.d.ts

@@ -1,9 +1,22 @@
 import type { EvalSuiteSummary } from '#src/evalTypes.js';
 /**
- * Write one structured JSON record per case (`<id>.json`) plus one aggregate `results.json`
- * (suite totals + every case's verdict/checks/judge/reasons) into `outputDir`. Creates `outputDir`
- * (and any missing parents) if it doesn't exist. Mirrors BATCH-1's `writeBatchOutput` (`#src/
- * output.js`) — same convention, applied to eval's richer per-case shape.
+ * Write one structured JSON record per cell (`<id>.json`, or `<id>__<identity>.json` for a
+ * BATCH-12 identity-matrix cell) plus one aggregate `results.json` (suite totals + every cell's
+ * verdict/checks/judge/reasons) into `outputDir`. Creates `outputDir` (and any missing parents) if
+ * it doesn't exist. Mirrors BATCH-1's `writeBatchOutput` (`#src/output.js`) — same convention,
+ * applied to eval's richer per-cell shape.
+ *
+ * Both the case `id` and the `identity` are validated at parse time to be plain filename-safe tokens
+ * (`/^[\w.-]+$/`), so joining them with a `__` separator can neither traverse nor escape `outputDir`.
+ * A no-identities cell writes `<id>.json` exactly as before BATCH-12.
+ *
+ * I1 belt-and-suspenders — because both ids and identity names permit `__`, two DISTINCT authored
+ * cells can collapse to the same per-cell filename (e.g. case `x` + identity `y__z` and case `x__y`
+ * + identity `z` both → `x__y__z.json`). Dispatch/grading are keyed by the unique `inputIndex` so
+ * the run itself is always correct and `results.json` holds every cell, but writing per-cell files
+ * by name would silently OVERWRITE one cell's record with another's. Detect that collision up front
+ * and throw BEFORE creating the directory or writing anything (→ the eval command's catch → exit 2,
+ * a suite-authoring signal) rather than emit a misleading, half-complete set of per-cell files.
  *
  * Pure I/O, deliberately separate from {@link ../evalRunner.js}'s `runEvalSuite`: the runner never
  * touches the filesystem, so unit tests can exercise grading logic without a tmp dir.

package/dist/evalOutput.js

@@ -1,18 +1,53 @@
 import { mkdirSync, writeFileSync } from 'node:fs';
 import { join } from 'node:path';
+/** The per-cell output basename (no extension): `<id>` for a no-identities cell, `<id>__<identity>`
+ * for an identity-matrix cell. Both `id` and `identity` are parse-time-validated filename-safe
+ * tokens (`/^[\w.-]+$/`), so the `__` join can neither traverse nor escape the output dir. */
+function outputFileBase(result) {
+    return result.identity !== undefined ? `${result.id}__${result.identity}` : result.id;
+}
+/** A human label for a cell in diagnostics: `<id>` or `<id> [<identity>]`. */
+function cellLabel(result) {
+    return result.identity !== undefined ? `${result.id} [${result.identity}]` : result.id;
+}
 /**
- * Write one structured JSON record per case (`<id>.json`) plus one aggregate `results.json`
- * (suite totals + every case's verdict/checks/judge/reasons) into `outputDir`. Creates `outputDir`
- * (and any missing parents) if it doesn't exist. Mirrors BATCH-1's `writeBatchOutput` (`#src/
- * output.js`) — same convention, applied to eval's richer per-case shape.
+ * Write one structured JSON record per cell (`<id>.json`, or `<id>__<identity>.json` for a
+ * BATCH-12 identity-matrix cell) plus one aggregate `results.json` (suite totals + every cell's
+ * verdict/checks/judge/reasons) into `outputDir`. Creates `outputDir` (and any missing parents) if
+ * it doesn't exist. Mirrors BATCH-1's `writeBatchOutput` (`#src/output.js`) — same convention,
+ * applied to eval's richer per-cell shape.
+ *
+ * Both the case `id` and the `identity` are validated at parse time to be plain filename-safe tokens
+ * (`/^[\w.-]+$/`), so joining them with a `__` separator can neither traverse nor escape `outputDir`.
+ * A no-identities cell writes `<id>.json` exactly as before BATCH-12.
+ *
+ * I1 belt-and-suspenders — because both ids and identity names permit `__`, two DISTINCT authored
+ * cells can collapse to the same per-cell filename (e.g. case `x` + identity `y__z` and case `x__y`
+ * + identity `z` both → `x__y__z.json`). Dispatch/grading are keyed by the unique `inputIndex` so
+ * the run itself is always correct and `results.json` holds every cell, but writing per-cell files
+ * by name would silently OVERWRITE one cell's record with another's. Detect that collision up front
+ * and throw BEFORE creating the directory or writing anything (→ the eval command's catch → exit 2,
+ * a suite-authoring signal) rather than emit a misleading, half-complete set of per-cell files.
  *
  * Pure I/O, deliberately separate from {@link ../evalRunner.js}'s `runEvalSuite`: the runner never
  * touches the filesystem, so unit tests can exercise grading logic without a tmp dir.
  */
 export function writeEvalOutput(outputDir, summary) {
+    const seenBy = new Map();
+    for (const result of summary.cases) {
+        const fileBase = outputFileBase(result);
+        const prior = seenBy.get(fileBase);
+        if (prior) {
+            throw new Error(`eval output filename collision: cells "${cellLabel(prior)}" and "${cellLabel(result)}" ` +
+                `both map to "${fileBase}.json". Case ids and identity names both allow "__", so distinct ` +
+                '(case × identity) cells can collapse to one filename — rename the case id or identity so ' +
+                'every cell has a unique <id>__<identity> name.');
+        }
+        seenBy.set(fileBase, result);
+    }
     mkdirSync(outputDir, { recursive: true });
     for (const result of summary.cases) {
-        writeFileSync(join(outputDir, `${result.id}.json`), `${JSON.stringify(result, null, 2)}\n`);
+        writeFileSync(join(outputDir, `${outputFileBase(result)}.json`), `${JSON.stringify(result, null, 2)}\n`);
     }
     writeFileSync(join(outputDir, 'results.json'), `${JSON.stringify(summary, null, 2)}\n`);
 }

package/dist/evalOutput.js.map

@@ -1 +1 @@
-{"version":3,"file":"evalOutput.js","sourceRoot":"","sources":["../src/evalOutput.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACnD,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AAGjC;;;;;;;;GAQG;AACH,MAAM,UAAU,eAAe,CAAC,SAAiB,EAAE,OAAyB;IAC1E,SAAS,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IAE1C,KAAK,MAAM,MAAM,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QACnC,aAAa,CAAC,IAAI,CAAC,SAAS,EAAE,GAAG,MAAM,CAAC,EAAE,OAAO,CAAC,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;IAC9F,CAAC;IAED,aAAa,CAAC,IAAI,CAAC,SAAS,EAAE,cAAc,CAAC,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;AAC1F,CAAC"}
+{"version":3,"file":"evalOutput.js","sourceRoot":"","sources":["../src/evalOutput.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACnD,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AAGjC;;8FAE8F;AAC9F,SAAS,cAAc,CAAC,MAAsB;IAC5C,OAAO,MAAM,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,EAAE,KAAK,MAAM,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,EAAE,CAAC;AACxF,CAAC;AAED,8EAA8E;AAC9E,SAAS,SAAS,CAAC,MAAsB;IACvC,OAAO,MAAM,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,EAAE,KAAK,MAAM,CAAC,QAAQ,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,EAAE,CAAC;AACzF,CAAC;AAED;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,MAAM,UAAU,eAAe,CAAC,SAAiB,EAAE,OAAyB;IAC1E,MAAM,MAAM,GAAG,IAAI,GAAG,EAA0B,CAAC;IACjD,KAAK,MAAM,MAAM,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QACnC,MAAM,QAAQ,GAAG,cAAc,CAAC,MAAM,CAAC,CAAC;QACxC,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QACnC,IAAI,KAAK,EAAE,CAAC;YACV,MAAM,IAAI,KAAK,CACb,0CAA0C,SAAS,CAAC,KAAK,CAAC,UAAU,SAAS,CAAC,MAAM,CAAC,IAAI;gBACvF,gBAAgB,QAAQ,mEAAmE;gBAC3F,2FAA2F;gBAC3F,gDAAgD,CACnD,CAAC;QACJ,CAAC;QACD,MAAM,CAAC,GAAG,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAC;IAC/B,CAAC;IAED,SAAS,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IAE1C,KAAK,MAAM,MAAM,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QACnC,aAAa,CACX,IAAI,CAAC,SAAS,EAAE,GAAG,cAAc,CAAC,MAAM,CAAC,OAAO,CAAC,EACjD,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CACvC,CAAC;IACJ,CAAC;IAED,aAAa,CAAC,IAAI,CAAC,SAAS,EAAE,cAAc,CAAC,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;AAC1F,CAAC"}

package/dist/evalRunner.d.ts

@@ -1,25 +1,78 @@
 import type { RunCellFn } from '#src/types.js';
-import type { EvalSuite, EvalSuiteSummary, JudgeFn } from '#src/evalTypes.js';
+import type { EvalSuite, EvalSuiteSummary, JudgeFn, RunConversationFn } from '#src/evalTypes.js';
 /** Options for {@link runEvalSuite}. */
 export interface RunEvalSuiteOptions {
-    /** The injectable per-case run function — same seam as BATCH-1's `RunCellFn`, adapted to send
-     * each case's `prompt` through the SUT (production wiring: `evalCommand.ts`, reusing
-     * `batchCommand.ts`'s `buildProductionRunCell`; tests inject a fake). */
-    runCell: RunCellFn;
-    /** The injectable judge function. Only consulted for cases that declare a `judgeRubric`;
-     * omitted entirely = every judge-rubric case fails with a "no judge configured" reason (this
-     * should not happen in production wiring, where `evalCommand.ts` always supplies one, but the
-     * runner degrades safely rather than throwing if it's left out). */
+    /**
+     * The single-run path (suite declares **no** `identities`): one injectable per-case run function,
+     * same seam as BATCH-1's `RunCellFn` (production wiring: `evalCommand.ts` via
+     * `batchCommand.ts`'s `buildProductionRunCell`; tests inject a fake). Required when the suite has
+     * no identities.
+     */
+    runCell?: RunCellFn;
+    /**
+     * The matrix path (BATCH-12; suite declares `identities`): one `RunCellFn` per identity, keyed by
+     * identity name. Each is built once by the command from `initConfig({ …, identityProfile })` — the
+     * same fresh-`.llm` construction `gth batch --models` uses per model — and reused across cases.
+     * The runner fans every case over these identities through the SAME execution + grading path the
+     * single-run mode uses.
+     */
+    runCellByIdentity?: Map<string, RunCellFn>;
+    /**
+     * BATCH-12 Task 2 — the MULTI-TURN seam for the no-identities path: run a whole scripted
+     * conversation (a case whose `turns.length > 1`) and return one {@link TurnRunOutcome} per turn.
+     * Required only when the suite has multi-turn cases and no identities. Single-turn cases keep
+     * using {@link runCell} (the proven `runSingleShot` path, byte-for-byte).
+     */
+    runConversation?: RunConversationFn;
+    /**
+     * BATCH-12 Task 2 — the MULTI-TURN seam per identity (matrix path): one {@link RunConversationFn}
+     * per identity, each built once by the command from that identity's config, reused across cases.
+     * The whole conversation runs once per identity, so per-identity "memory"/authorization is real.
+     */
+    runConversationByIdentity?: Map<string, RunConversationFn>;
+    /** The injectable judge function. Only consulted for expectation blocks that declare a
+     * `judgeRubric`; omitted entirely = every judge-rubric block fails with a "no judge configured"
+     * reason (the runner degrades safely rather than throwing). The judge is orthogonal to identity:
+     * it grades each cell's answer regardless of which identity produced it. */
     judge?: JudgeFn;
-    /** Max in-flight cases — reuses BATCH-1's `runBatchMatrix` pool (`DEFAULT_CONCURRENCY` when
+    /** Max in-flight cells — reuses BATCH-1's `runBatchMatrix` pool (`DEFAULT_CONCURRENCY` when
      * omitted); no second concurrency mechanism is introduced for eval. */
     concurrency?: number;
 }
 /**
- * Run every case in the suite through the SUT ({@link RunEvalSuiteOptions.runCell}, pooled via
- * BATCH-1's `runBatchMatrix`), then grade each answer with deterministic checks and (when the case
- * declares a rubric) the judge. A case PASSES iff its deterministic checks pass AND (it has no
- * judge rubric OR the judge's rate is at/above the case's `passThreshold`) — ported from the field
- * user's proven harness semantics (`docs/batch-eval-user-requirements.md` Appendix A).
+ * Run every (case × identity) cell of the suite through the SUT ({@link RunCellFn}, pooled via
+ * BATCH-1's `runBatchMatrix`), then grade each answer with the applicable expectation blocks'
+ * deterministic checks, tool-trace checks, and (when a block declares a rubric) the judge. A cell
+ * PASSES iff EVERY applicable block passes — ported from the field user's proven harness semantics
+ * (`docs/batch-eval-user-requirements.md` Appendix A), generalized to per-identity blocks.
+ *
+ * ONE concurrency pool: flat and matrix suites both normalize to {@link EvalUnit}s and ride the same
+ * `runBatchMatrix` pool; the only divergence is which `RunCellFn` a unit's identity resolves to.
+ *
+ * BATCH-12 Task 2 — a unit whose case has `turns.length > 1` is a MULTI-TURN conversation: it runs
+ * through the injected {@link RunConversationFn} seam (agent/tools built once, messages accumulated
+ * across turns) instead of {@link RunCellFn}, still inside the SAME pool, and is graded turn-by-turn
+ * by {@link gradeConversationUnit} — the cell PASSES iff EVERY turn's applicable blocks pass. A
+ * single-turn unit keeps the proven `runCell` + {@link gradeUnit} path byte-for-byte.
  */
 export declare function runEvalSuite(suite: EvalSuite, options: RunEvalSuiteOptions): Promise<EvalSuiteSummary>;
+/** The three-way process exit code for a completed `gth eval` run. See {@link classifyEvalExit}. */
+export type EvalExitCode = 0 | 1 | 2;
+/**
+ * BATCH-11 (#405 his #6) — classify a completed suite's {@link EvalSuiteSummary} into a distinct
+ * exit code so CI can tell a product regression from a broken harness. BATCH-12 counts matrix
+ * CELLS (one per case × identity), not cases:
+ *
+ * - `0` — every cell passed (unchanged contract).
+ * - `1` — the suite **ran** but ≥1 cell FAILED (assertion/judge below threshold). A *product*
+ *   signal: real, gradeable results, some below the bar.
+ * - `2` — **harness error**: no gradeable results at all — an empty suite, or **every** cell's SUT
+ *   run failed (`sutOk === false`). (The other harness errors that never reach a summary — suite
+ *   load/parse error, config error, an unresolved identity precondition — are mapped to `2` by the
+ *   caller (`evalCommand.ts`) in a try/catch.)
+ *
+ * Classification is anchored on `sutOk`, not the verdict: a cell that ran (`sutOk === true`) but
+ * whose judge errored (or whose answer failed a check) is a real result → exit `1`, never `2`. A
+ * *mix* of `sutOk:false` and `sutOk:true` cells therefore yields `1`.
+ */
+export declare function classifyEvalExit(summary: EvalSuiteSummary): EvalExitCode;