@agentproto/eval 0.2.11 → 0.3.1

package/README.md

@@ -193,6 +193,43 @@ value >= threshold` (`threshold` defaults to `0.5`).
 A real adapter wiring `JudgeFn` up to an agent session or the supervisor's
 judge-gate is a documented follow-up — not built in this package.
 
+## Style scorers
+
+Five more `eval.*` TOOL contracts, under `src/style/` (see DESIGN.md §7 in the
+`factory` plan for the production bands/gates these back). Two are
+deterministic — bundled into `styleScorersProvider` exactly like
+`evalScorersProvider` — and three are model-backed, each built by a
+`make*Driver(...)` factory that closes over an injected, vendor-neutral
+capability. No LLM SDK, no network dependency, same discipline as
+`eval.llm-judge`.
+
+| Tool id                  | Input                                  | Behavior |
+| ------------------------ | --------------------------------------- | -------- |
+| `eval.text-stats`        | `{ text, thresholds? }`                 | Deterministic. Computes bullet-line ratio, first-person ratio, question rate, and mean sentence length (+ `inBand`) over French text. `passed` is derived from caller-supplied `thresholds` (`maxBulletsRatio` default 0.02, `minFirstPersonRatio` default 0.6, `lengthBand` default `{min: 5, max: 30}` words) — this tool owns no fixed gate. |
+| `eval.lexicon-hit-rate`  | `{ text, lexicon (min 1), threshold? }` | Deterministic. Fraction of `lexicon` terms present as whole words in `text`, Unicode-aware (accented terms like `écrire` match correctly; ASCII `\b` would miss them). Text and terms are NFC-normalized before matching. `passed = hitRate >= threshold` (default 0.5). Build a signature lexicon offline with the pure helper `extractLexicon(corpusTexts, { top?, minLen?, background? })` — with `background`, terms are ranked by a log-odds ratio against that corpus instead of raw frequency (a simple frequency-ratio estimator, not the full variance-weighted informative-Dirichlet estimator). |
+| `eval.style-pairwise`    | `{ reference, a, b, criteria }`         | Model-backed via `makeStylePairwiseDriver(judge: JudgeFn)` — reuses the `eval.llm-judge` `JudgeFn` seam (the raw verdict is validated with `judgeVerdictSchema`; a malformed verdict fails the score instead of propagating `NaN`). `value` encodes preference (1 = `a` wins, 0 = `b` wins, 0.5 = tie). The pure helper `pairwiseWinRate(verdicts)` aggregates `PairwiseVerdict[]` (each carrying a required `item` id) into `{ winRate, kappa, n, nNormal, nSwapped, balanced }`: `winRate` averages the normal-order and swapped-order rates (falling back to whichever order is present, flagged via `balanced: false`, when one is missing entirely); `kappa` is Cohen's kappa between the two most-represented judges, joined by `item` (so a normal+swapped pair on the same item is one observation, not two) — it is `null`, not a default `1`, when there are fewer than two judges or fewer than two items in common, and a `null` must be treated as a gate FAILURE, not skipped. |
+| `eval.style-embedding`   | `{ candidate, references[] (min 1) }`   | Model-backed via `makeStyleEmbeddingDriver(embed: EmbedFn)`, `EmbedFn = (texts) => Promise<number[][]>`. `value` = cosine similarity of `candidate` to the `references` centroid, clamped to `[0, 1]` via `max(0, cosine)` — NOT remapped via `(cosine + 1) / 2`, which put an uninformative orthogonal candidate at the same 0.5 as the default pass threshold. A candidate equal to the centroid scores 1; orthogonal or opposing candidates score 0. Heterogeneous embedding dimensions (mismatched candidate/reference vectors) fail the score with a rationale instead of silently padding/truncating. Pure helper: `cosineToCentroid(candidate, references)` (throws `EmbeddingDimensionError` on dimension mismatch — callers at the tool boundary must catch it). |
+| `eval.outline-fidelity`  | `{ outline, answer }`                   | Model-backed via `makeOutlineFidelityDriver(judge: JudgeFn)`. `value` = judged outline coverage; `passed = value >= 0.95` — a **fixed** gate, never overridden by the judge's own `passed` (unlike `eval.llm-judge`'s threshold semantics). The raw verdict is validated with `judgeVerdictSchema`; a malformed verdict fails the score. |
+
+French text conventions used by `eval.text-stats`: first-person markers are
+`je`, `j'`, `moi`, `mon`/`ma`/`mes`, `me`/`m'`, `nous`, `notre`/`nos`,
+`mien(ne)(s)`; a bullet line starts with `-`, `*`, `•`, or a numbered marker
+like `1.` followed by whitespace (`1.5 million` and `-42 degrés` are prose,
+not bullets). Sentence splitting guards against common French abbreviations
+(`M.`, `Mme`, `Dr`, `etc.`, `cf.`, `p. ex.`) and isolated capital initials
+(`J. Dupont`) so those periods are not treated as sentence ends.
+
+```ts
+import { runTool } from "@agentproto/driver"
+import { textStatsTool, styleScorersProvider } from "@agentproto/eval"
+
+const score = await runTool({
+  tool: textStatsTool,
+  candidates: [styleScorersProvider],
+  input: { text: "Je pense que ceci illustre bien mon propos.", thresholds: { minFirstPersonRatio: 0.5 } },
+})
+```
+
 ## As a CI gate
 
 `toVitest` turns a suite into vitest test registrations — one `it(caseId)` per
@@ -230,6 +267,22 @@ toVitest(
 - `llmJudge` — convenience: build a ready-to-use `ScorerBinding` around a judge
 - `JudgeFn`, `JudgeVerdict`, `judgeVerdictSchema`, `LlmJudgeInput`,
   `MakeLlmJudgeDriverOptions`, `LlmJudgeBinding`
+- `textStatsTool` / `textStatsImpl` — plus pure helpers `bulletsRatio`,
+  `firstPersonRatio`, `questionRate`, `meanSentenceLength`, `splitSentences`,
+  `computeTextStats` (`TextStats`, `LengthBand`)
+- `lexiconHitRateTool` / `lexiconHitRateImpl` — plus `extractLexicon`
+  (`ExtractLexiconOptions`)
+- `styleScorersProvider` — the builtin PROVIDER bundling the two
+  deterministic style scorers
+- `stylePairwiseTool`, `makeStylePairwiseDriver`, `pairwiseWinRate`
+  (`StylePairwiseInput`, `PairwiseWinner`, `PairwiseVerdict`,
+  `PairwiseWinRateResult`)
+- `styleEmbeddingTool`, `makeStyleEmbeddingDriver`, `cosineToCentroid`,
+  `EmbeddingDimensionError`
+  (`StyleEmbeddingInput`, `EmbedFn`, `MakeStyleEmbeddingDriverOptions`)
+- `outlineFidelityTool`, `makeOutlineFidelityDriver` (`OutlineFidelityInput`)
+- `parseVerdict` — shared `style/` helper: validates a raw judge return value
+  against `judgeVerdictSchema`, returning `null` on malformed input
 
 ## License
 

package/dist/index.d.ts

@@ -409,6 +409,300 @@ type ToVitestOptions<I, A> = RunEvalOptions<I, A>;
  */
 declare function toVitest<I, A>(suite: TypedEvalSuite<I, A>, opts: ToVitestOptions<I, A>, hooks: VitestHooks): void;
 
+/**
+ * Split text into non-empty sentences on `.`/`!`/`?` boundaries, guarding
+ * against abbreviations (`M.`, `Mme`, `Dr`, `etc.`, `cf.`, `p. ex.`) and
+ * isolated capital initials (`J. Dupont`) so those periods don't count as
+ * sentence ends.
+ */
+declare function splitSentences(text: string): string[];
+/** Fraction of lines that open with a bullet marker (`-`, `*`, `•`, or `1.`) followed by whitespace — a marker glued to the next character (`1.5 million`, `-42 degrés`) is prose, not a bullet. */
+declare function bulletsRatio(text: string): number;
+/** Fraction of sentences carrying a French first-person marker. */
+declare function firstPersonRatio(text: string): number;
+/** Fraction of sentences ending in `?`. */
+declare function questionRate(text: string): number;
+/** Mean number of words per sentence. */
+declare function meanSentenceLength(text: string): number;
+interface TextStats {
+    readonly bulletsRatio: number;
+    readonly firstPersonRatio: number;
+    readonly questionRate: number;
+    readonly meanSentenceLength: number;
+    readonly inBand: boolean;
+}
+interface LengthBand {
+    readonly min: number;
+    readonly max: number;
+}
+/** Compute every metric in one pass. */
+declare function computeTextStats(text: string, band?: LengthBand): TextStats;
+declare const textStatsTool: _agentproto_tool.ToolHandle<{
+    text: string;
+    thresholds?: {
+        maxBulletsRatio?: number | undefined;
+        minFirstPersonRatio?: number | undefined;
+        lengthBand?: {
+            min: number;
+            max: number;
+        } | undefined;
+    } | undefined;
+}, {
+    value: number;
+    passed: boolean;
+    label: string;
+    rationale?: string | undefined;
+}, _agentproto_tool.ToolContext>;
+declare const textStatsImpl: _agentproto_driver.ToolImplementation<{
+    text: string;
+    thresholds?: {
+        maxBulletsRatio?: number | undefined;
+        minFirstPersonRatio?: number | undefined;
+        lengthBand?: {
+            min: number;
+            max: number;
+        } | undefined;
+    } | undefined;
+}, {
+    value: number;
+    passed: boolean;
+    label: string;
+    rationale?: string | undefined;
+}, _agentproto_tool.ToolContext>;
+
+interface ExtractLexiconOptions {
+    /** Max number of terms to return. Default 20. */
+    readonly top?: number;
+    /** Minimum term length (characters) to keep. Default 3. */
+    readonly minLen?: number;
+    /**
+     * Optional background corpus. When provided, terms are ranked by an
+     * add-one-smoothed log-odds ratio of their rate in `corpusTexts` versus
+     * their rate in `background`, surfacing terms disproportionately frequent
+     * in the foreground corpus rather than merely frequent overall. This is a
+     * simple frequency-ratio estimator, NOT the full variance-weighted
+     * informative-Dirichlet log-odds estimator (Monroe et al. 2008) — it has
+     * no correction for small counts beyond add-one smoothing, so rare terms
+     * in a small background corpus can score unstably high. When omitted,
+     * ranking falls back to raw frequency in `corpusTexts`.
+     */
+    readonly background?: readonly string[];
+}
+/**
+ * Pure helper: extract a corpus's signature lexicon — the most frequent
+ * non-stopword terms across `corpusTexts` — for use as `eval.lexicon-hit-rate`
+ * input. Offline / caller-side; this tool never calls it itself.
+ */
+declare function extractLexicon(corpusTexts: readonly string[], opts?: ExtractLexiconOptions): string[];
+declare const lexiconHitRateTool: _agentproto_tool.ToolHandle<{
+    text: string;
+    lexicon: string[];
+    threshold?: number | undefined;
+}, {
+    value: number;
+    passed: boolean;
+    label: string;
+    rationale?: string | undefined;
+}, _agentproto_tool.ToolContext>;
+declare const lexiconHitRateImpl: _agentproto_driver.ToolImplementation<{
+    text: string;
+    lexicon: string[];
+    threshold?: number | undefined;
+}, {
+    value: number;
+    passed: boolean;
+    label: string;
+    rationale?: string | undefined;
+}, _agentproto_tool.ToolContext>;
+
+/**
+ * `eval.style-pairwise` — model-backed A/B scorer: which of `a`/`b` reads
+ * closer to `reference` under `criteria`. Reuses the existing {@link JudgeFn}
+ * seam from judge.ts (no new judge type) — the driver hands the judge
+ * `{reference, a, b}` as `output` and reads back a single verdict whose
+ * `value` encodes preference: 1 = `a` wins, 0 = `b` wins, 0.5 = tie.
+ *
+ * `pairwiseWinRate` below is a SEPARATE pure helper: callers run this tool
+ * multiple times (varying which side is presented first, and across
+ * multiple judges) and feed the resulting {winner} outcomes to it to get an
+ * order-debiased win rate plus inter-judge agreement (Cohen's kappa).
+ */
+interface StylePairwiseInput {
+    readonly reference: string;
+    readonly a: string;
+    readonly b: string;
+    readonly criteria: string;
+}
+declare const stylePairwiseTool: _agentproto_tool.ToolHandle<{
+    reference: string;
+    a: string;
+    b: string;
+    criteria: string;
+}, {
+    value: number;
+    passed: boolean;
+    label: string;
+    rationale?: string | undefined;
+}, _agentproto_tool.ToolContext>;
+/**
+ * Build a DRIVER that implements `eval.style-pairwise` by delegating to
+ * `judge` — the same {@link JudgeFn} shape as `eval.llm-judge`, reused rather
+ * than threading a bespoke comparison type through this package.
+ */
+declare function makeStylePairwiseDriver(judge: JudgeFn): DriverHandle;
+type PairwiseWinner = "a" | "b" | "tie";
+/**
+ * One collected pairwise outcome. `order` records which physical position
+ * `a` was presented in when this verdict was produced; `winner` is the RAW
+ * outcome as reported for that presentation (before un-swapping) — e.g. a
+ * judge exhibiting pure position bias always picks the same physical slot,
+ * which shows up here as `winner` tracking `order` rather than content.
+ * `pairwiseWinRate` un-swaps `winner` back to the logical a/b before
+ * aggregating, which is what neutralizes that bias.
+ */
+interface PairwiseVerdict {
+    /** Identifies which item this verdict scored — required so verdicts from different items are never zipped together. */
+    readonly item: string;
+    /** Judge identity — distinguishes the (at least two) judges being compared. */
+    readonly judge: string;
+    /** Physical presentation order this verdict was collected under. */
+    readonly order: "normal" | "swapped";
+    /** Raw winner as reported under `order` (not yet un-swapped). */
+    readonly winner: PairwiseWinner;
+}
+interface PairwiseWinRateResult {
+    /** Logical a-win rate (ties count as 0.5), after order de-biasing. Averages the normal-order and swapped-order rates when both are present. */
+    readonly winRate: number;
+    /**
+     * Cohen's kappa agreement between the two most-represented judges, computed
+     * over items both judges rated. `null` when there are fewer than two
+     * judges, or fewer than two items in common — a `null` here means the
+     * agreement gate has no evidence and any threshold check against it (e.g.
+     * "kappa >= 0.4") must be treated as a FAILURE, not skipped.
+     */
+    readonly kappa: number | null;
+    /** Number of verdicts folded in. */
+    readonly n: number;
+    /** Number of verdicts collected under `order: "normal"`. */
+    readonly nNormal: number;
+    /** Number of verdicts collected under `order: "swapped"`. */
+    readonly nSwapped: number;
+    /** `false` when one of the two presentation orders has zero verdicts — `winRate` then reflects only the order that is present, and is not order-debiased. */
+    readonly balanced: boolean;
+}
+/**
+ * Aggregate collected {@link PairwiseVerdict}s into an order-debiased win
+ * rate and inter-judge Cohen's kappa.
+ *
+ * Win rate: each verdict is un-swapped against its `order` (neutralizing
+ * position bias), then the normal-order rate and swapped-order rate are
+ * averaged — not a flat average over all verdicts — so a judge that always
+ * picks the physically-first slot nets out near 0.5 even if one order was
+ * sampled more than the other. When only one order was collected at all,
+ * `balanced` is `false` and `winRate` falls back to that order's rate alone.
+ *
+ * Kappa: verdicts are joined by `item` — for each of the two
+ * most-represented judges, at most one (first-seen, un-swapped) winner is
+ * kept per item, so a normal+swapped pair on the same item is one
+ * observation, not two. Kappa is then computed over the items both judges
+ * rated; see {@link PairwiseWinRateResult.kappa} for the `null` cases.
+ */
+declare function pairwiseWinRate(verdicts: readonly PairwiseVerdict[]): PairwiseWinRateResult;
+
+/** Thrown by {@link centroid} / {@link cosineToCentroid} on mismatched embedding dimensions — callers crossing the tool boundary (the driver) must catch this and fail the `Score`, never let it propagate as a thrown error out of the tool contract. */
+declare class EmbeddingDimensionError extends RangeError {
+}
+/**
+ * Pure helper: cosine similarity between `candidate` and the centroid of
+ * `references`, clamped to `[0, 1]` via `max(0, cosine)` — NOT remapped from
+ * `[-1, 1]` via `(cosine + 1) / 2`. That remap put an orthogonal candidate
+ * (cosine 0, no signal) at 0.5, the same value as the default `passed`
+ * threshold, so an uninformative embedding silently cleared the gate. Under
+ * `max(0, cosine)`, orthogonal and opposing candidates both score 0. A
+ * candidate equal to the centroid still returns exactly 1.
+ *
+ * Throws {@link EmbeddingDimensionError} if `candidate` and the references'
+ * centroid have different dimensionality — callers at the tool boundary must
+ * catch this (see {@link makeStyleEmbeddingDriver}) rather than let it cross
+ * into a thrown error from the TOOL contract.
+ */
+declare function cosineToCentroid(candidate: readonly number[], references: readonly (readonly number[])[]): number;
+interface StyleEmbeddingInput {
+    readonly candidate: string;
+    readonly references: readonly string[];
+}
+declare const styleEmbeddingTool: _agentproto_tool.ToolHandle<{
+    candidate: string;
+    references: string[];
+}, {
+    value: number;
+    passed: boolean;
+    label: string;
+    rationale?: string | undefined;
+}, _agentproto_tool.ToolContext>;
+/**
+ * The seam a real embedding capability satisfies: given a batch of texts,
+ * return one embedding vector per text, same order. Deliberately no model
+ * SDK / network types here — vendor-neutral, like {@link JudgeFn}.
+ */
+type EmbedFn = (texts: readonly string[]) => Promise<number[][]>;
+interface MakeStyleEmbeddingDriverOptions {
+    /** Minimum value to count as passed. Default 0.5. */
+    readonly threshold?: number;
+}
+/**
+ * Build a DRIVER that implements `eval.style-embedding` by delegating to
+ * `embed`. Embeds `[candidate, ...references]` in a single batch call so a
+ * remote embedding backend sees one request per score.
+ */
+declare function makeStyleEmbeddingDriver(embed: EmbedFn, opts?: MakeStyleEmbeddingDriverOptions): DriverHandle;
+
+/**
+ * Validate a raw judge return value against {@link judgeVerdictSchema}. An
+ * injected `JudgeFn` is caller-supplied and only nominally typed — a
+ * misbehaving judge can hand back `NaN`, a missing `value`, or any other
+ * malformed shape at runtime. Callers use this to fail the `Score` with a
+ * rationale instead of letting a bad verdict propagate as `NaN`/`undefined`
+ * downstream.
+ */
+declare function parseVerdict(raw: unknown): JudgeVerdict | null;
+
+/**
+ * `eval.outline-fidelity` — model-backed scorer: does `answer` cover every
+ * point in `outline` without adding unsourced facts. Reuses the existing
+ * {@link JudgeFn} seam (`output = answer`, `expected = outline`) — no new
+ * judge type. Unlike `eval.llm-judge`, `passed` here is a FIXED gate
+ * (`value >= 0.95`, per DESIGN.md §7), never overridden by the judge's own
+ * `passed`.
+ */
+interface OutlineFidelityInput {
+    readonly outline: string;
+    readonly answer: string;
+}
+declare const outlineFidelityTool: _agentproto_tool.ToolHandle<{
+    outline: string;
+    answer: string;
+}, {
+    value: number;
+    passed: boolean;
+    label: string;
+    rationale?: string | undefined;
+}, _agentproto_tool.ToolContext>;
+/**
+ * Build a DRIVER that implements `eval.outline-fidelity` by delegating to
+ * `judge`, the same {@link JudgeFn} shape as `eval.llm-judge`.
+ */
+declare function makeOutlineFidelityDriver(judge: JudgeFn): DriverHandle;
+
+/**
+ * Builtin AIP-30 PROVIDER bundling the two deterministic style scorers —
+ * the pair DESIGN.md §7 names as the only style scorers safe to sample in
+ * prod (free, zero network). The three model-backed style tools have no
+ * static provider: each needs an injected judge/embed via its
+ * `make*Driver(...)` factory, same pattern as `eval.llm-judge`.
+ */
+declare const styleScorersProvider: _agentproto_driver.DriverHandle;
+
 /**
  * @agentproto/eval — deterministic reference scorers.
  *
@@ -446,4 +740,4 @@ declare const SPEC_VERSION: "1.0.0-alpha";
  */
 declare const evalScorersProvider: _agentproto_driver.DriverHandle;
 
-export { type BoundScorer, type CaseReport, type CaseScore, EVAL_EVENT_SCHEMA, type EvalCase, type EvalEvent, type EvalReport, type EvalSuite, type ExpectApi, type JsonObject, type JsonValue, type JudgeFn, type JudgeVerdict, type LlmJudgeBinding, type LlmJudgeInput, type MakeLlmJudgeDriverOptions, type MinimalJsonSchema, type RunEvalOptions, SPEC_NAME, SPEC_VERSION, type Score, type ScorerBinding, type ScorerInputContext, type ToVitestOptions, type TypedEvalSuite, type VitestHooks, bindScorer, evalScorersProvider, exactMatchImpl, exactMatchTool, jsonSchemaValidImpl, jsonSchemaValidTool, jsonValueSchema, judgeVerdictSchema, latencyBudgetImpl, latencyBudgetTool, llmJudge, llmJudgeTool, makeLlmJudgeDriver, regexMatchImpl, regexMatchTool, runEval, scoreSchema, toVitest };
+export { type BoundScorer, type CaseReport, type CaseScore, EVAL_EVENT_SCHEMA, type EmbedFn, EmbeddingDimensionError, type EvalCase, type EvalEvent, type EvalReport, type EvalSuite, type ExpectApi, type ExtractLexiconOptions, type JsonObject, type JsonValue, type JudgeFn, type JudgeVerdict, type LengthBand, type LlmJudgeBinding, type LlmJudgeInput, type MakeLlmJudgeDriverOptions, type MakeStyleEmbeddingDriverOptions, type MinimalJsonSchema, type OutlineFidelityInput, type PairwiseVerdict, type PairwiseWinRateResult, type PairwiseWinner, type RunEvalOptions, SPEC_NAME, SPEC_VERSION, type Score, type ScorerBinding, type ScorerInputContext, type StyleEmbeddingInput, type StylePairwiseInput, type TextStats, type ToVitestOptions, type TypedEvalSuite, type VitestHooks, bindScorer, bulletsRatio, computeTextStats, cosineToCentroid, evalScorersProvider, exactMatchImpl, exactMatchTool, extractLexicon, firstPersonRatio, jsonSchemaValidImpl, jsonSchemaValidTool, jsonValueSchema, judgeVerdictSchema, latencyBudgetImpl, latencyBudgetTool, lexiconHitRateImpl, lexiconHitRateTool, llmJudge, llmJudgeTool, makeLlmJudgeDriver, makeOutlineFidelityDriver, makeStyleEmbeddingDriver, makeStylePairwiseDriver, meanSentenceLength, outlineFidelityTool, pairwiseWinRate, parseVerdict, questionRate, regexMatchImpl, regexMatchTool, runEval, scoreSchema, splitSentences, styleEmbeddingTool, stylePairwiseTool, styleScorersProvider, textStatsImpl, textStatsTool, toVitest };