llm-output-guard 1.0.0 → 1.2.0

package/README.md

@@ -76,6 +76,47 @@ const verdict = checkOutput(raw, {
 if (verdict.ok) use(verdict.json); // already parsed, fence stripped
 ```
 
+`requiredKeys` only asks whether a name is present. A model returning
+`{ "score": "very good" }` where you wanted a number satisfies it and still
+breaks everything downstream that does arithmetic. Pass a **schema** to check
+the shape rather than the spelling:
+
+```ts
+import { z } from 'zod';
+
+const Review = z.object({
+  score: z.number().min(0).max(10),
+  notes: z.string(),
+  followUp: z.array(z.string()),
+});
+
+const verdict = checkOutput(raw, { ...presets.strictJson, schema: Review });
+
+if (verdict.ok) use(verdict.json); // parsed, validated, defaults applied
+```
+
+Any [Standard Schema](https://standardschema.dev) validator works — **Zod 4,
+Valibot, ArkType**, or your own. The spec is types-only, so this costs an
+interface and **no dependency**: your validator is one you already have, and
+`llm-output-guard` still installs with nothing behind it.
+
+On success `verdict.json` is the schema's *output*, so defaults, coercions and
+transforms are applied and the value matches the type you declared. On failure
+you get `INVALID_JSON` with the failing path in the message —
+`score: Expected number, received string`. It is the same reason code as a
+missing key or an unparseable payload because it wants the same handling: retry,
+or fall through to another provider.
+
+The two compose, and keys are checked first, so a missing key is still reported
+as a missing key rather than as whatever the schema calls it.
+
+> **The schema must validate synchronously.** `checkOutput` is synchronous by
+> design — that is what makes it safe on a hot path — so a schema carrying an
+> async refinement throws a `TypeError` telling you so, rather than silently
+> passing. Everything Zod, Valibot and ArkType produce otherwise is synchronous.
+> This is the one thing in the package that throws about your configuration; it
+> still never throws about a response.
+
 ### Streaming, where it stops costing you tokens
 
 Checking a finished response tells you that you already paid for it. A model
@@ -213,7 +254,8 @@ call `checkOutput` on the result yourself.
 
 ### OpenAI SDK — and anything speaking its protocol
 
-One wrap, and both call shapes are guarded:
+One wrap, and both APIs are guarded — `chat.completions.create` and
+`responses.create`, streaming and not:
 
 ```ts
 import OpenAI from 'openai';
@@ -224,12 +266,30 @@ const client = withOutputGuard(new OpenAI(), {
   ...presets.chat,
   onDegenerate: 'abort',
 });
+
+await client.chat.completions.create({ model, messages });   // guarded
+await client.responses.create({ model, input });             // guarded
 ```
 
+The Responses API spells its stop reason `incomplete_details.reason` rather than
+`finish_reason`, and its length stop `max_output_tokens` rather than `length`.
+Both are mapped, so `TRUNCATED` fires the same way on either. `content_filter`
+is deliberately *not* read as truncation — a filtered response is a different
+failure, and reporting it as `TRUNCATED` would send a retry layer after the
+wrong fix.
+
+> **`responses.stream()` is not guarded.** It returns a `ResponseStream` — an
+> event emitter with `.on()` and `.finalResponse()`, not just an async iterable
+> — and wrapping only its iteration would guard a `for await` consumer while
+> leaving `.finalResponse()` unchecked. A guard you believe in and do not have
+> is the failure this package was written about, so it is left plainly
+> unguarded rather than half-wrapped. Use `create({ stream: true })`, which is
+> guarded, or run `checkOutput` on `await stream.finalResponse()` yourself.
+
 This is also how you guard **Groq, Together, OpenRouter, Fireworks, DeepInfra,
 vLLM and Ollama** — anything you reach through an OpenAI-compatible `baseURL`
-works, because the adapter is typed against the chat-completions shape rather
-than against OpenAI the company.
+works, because the adapter is typed against the wire shapes rather than against
+OpenAI the company.
 
 Non-streaming calls are already paid for by the time anything can run, so a
 degenerate one throws `DegenerateOutputError` for your fallback layer to catch:
@@ -288,8 +348,80 @@ deferred to `end()`. `LOW_ENTROPY` is deferred too, for cost — it is ~100x the
 other detectors, and everything it would have caught early is caught by
 `REPETITION`, or by `TAIL_LOOP`'s character mode on non-spaced scripts.
 
-Both adapters share this behaviour because both drive the same
-`createStreamGuard`. Neither reimplements it.
+Every adapter shares this behaviour because they all drive the same
+`createStreamGuard`. None of them reimplements it.
+
+### Anthropic SDK
+
+Same one wrap, same options:
+
+```ts
+import Anthropic from '@anthropic-ai/sdk';
+import { withOutputGuard } from 'llm-output-guard/anthropic';
+import { presets } from 'llm-output-guard';
+
+const client = withOutputGuard(new Anthropic(), {
+  ...presets.chat,
+  onDegenerate: 'abort',
+});
+
+await client.messages.create({ model, max_tokens, messages });               // guarded
+await client.messages.create({ model, max_tokens, messages, stream: true }); // guarded
+```
+
+Two things are specific to this API:
+
+**Extended thinking is not read as the answer.** `thinking` blocks are the
+model's reasoning, they are often longer than the answer, and they repeat
+themselves as a matter of course while working a problem. Folding them into the
+measured text would raise every repetition score on every thinking response and
+flag the ones that thought hardest — so only `text` blocks are measured, and a
+`thinking` block is not mistaken for a tool call either.
+
+**Both of Anthropic's length stops map to `TRUNCATED`.** `max_tokens` passes
+straight through; `model_context_window_exceeded` is the same event under a
+different name and is normalised in the adapter. `refusal` is deliberately *not*
+truncation — a refusal is a complete response that says no, which is a content
+judgement this package does not make.
+
+> **`messages.stream()` is not guarded**, for the same reason `responses.stream()`
+> isn't: it returns a `MessageStream` — an event emitter with `.on()` and
+> `.finalMessage()` — and guarding only its iteration would leave
+> `.finalMessage()` unchecked. Use `create({ stream: true })`, or run
+> `checkOutput` on `await stream.finalMessage()` yourself. `messages.batches` is
+> unguarded too, and less interestingly: a batch is retrieved later as a file of
+> results, so there is no response at `create` time to inspect.
+
+`@anthropic-ai/sdk` is an **optional peer dependency**, declared
+`>=0.60.0 <1.0.0` and verified at 0.60.0, 0.90.0 and 0.117.1 — each installing
+the packed tarball and running the adapter for real, not just typechecking.
+
+### Tool calls and agents
+
+A model that answers by calling a tool returns no assistant text — OpenAI sends
+`content: null` beside `tool_calls`, and the AI SDK sends a `content` array with
+no `text` part. Handed to `checkOutput`, that is an empty string, and an empty
+string scores `EMPTY`.
+
+So **the presence of tool calls means the text, if any, is a preamble rather
+than the answer**, and both adapters judge it as one:
+
+| | On a tool-calling turn |
+|---|---|
+| No text at all | Nothing is judged, and nothing is reported to `onVerdict` |
+| Text beside the call | `REPETITION`, `TAIL_LOOP` and `LOW_ENTROPY` still run |
+| `TOO_SHORT` | Off — "Let me look that up" is sixteen characters and correct |
+| `TRUNCATED` | Off — a preamble ends without terminal punctuation as a matter of course |
+| `INVALID_JSON` | Off — the JSON is in the call arguments, which your provider already validated against the schema |
+
+The redundancy detectors stay on because a model looping in its preamble is
+still a model that is looping. `EMPTY` is not disarmed either: a response with
+neither text nor tool calls still fails, which is the case this package exists
+for.
+
+Nothing is reported to `onVerdict` for a text-free tool call on purpose. Those
+samples are what a `calibrate` run is built from, and a spike of `EMPTY: 1` in
+them would describe your agent's tool use rather than any degeneration.
 
 ---
 
@@ -318,18 +450,34 @@ gives you a number that describes neither.
 
 ## Detectors
 
-| Code | Catches | Signal |
-|---|---|---|
-| `EMPTY` | Whitespace, lone punctuation, `{}`, empty fences | Content presence |
-| `TOO_SHORT` | Non-empty but useless | Length vs. minimum |
-| `REPETITION` | Loops and stutters | Duplicate word n-gram fraction |
-| `TAIL_LOOP` | Good start, then a stuck ending | Periodicity in the trailing window, over words or characters |
-| `LOW_ENTROPY` | Character-level collapse, token artifacts | Hand-rolled LZ77 compression ratio |
-| `TRUNCATED` | Cut off mid-thought | `finish_reason`, unbalanced fences/brackets |
-| `INVALID_JSON` | Prose around the payload, missing keys | Parse + key contract |
-| `LANG_MISMATCH` | Answered in the wrong language | Function-word profile (coarse, opt-in) |
+| Code | Catches | Signal | Exported as |
+|---|---|---|---|
+| `EMPTY` | Whitespace, lone punctuation, `{}`, empty fences | Content presence | `emptinessScore` |
+| `TOO_SHORT` | Non-empty but useless | Length vs. minimum | `shortnessScore` |
+| `REPETITION` | Loops and stutters | Duplicate word n-gram fraction | `repetitionScore` |
+| `TAIL_LOOP` | Good start, then a stuck ending | Periodicity in the trailing window, over words or characters | `tailLoopScore`, `tailLoopDetail` |
+| `LOW_ENTROPY` | Character-level collapse, token artifacts | Hand-rolled LZ77 compression ratio | `compressibilityScore`, `compressionRatio` |
+| `TRUNCATED` | Cut off mid-thought | `finish_reason`, unbalanced fences/brackets | `truncationScore` |
+| `INVALID_JSON` | Prose around the payload, missing keys, wrong types | Parse + key contract + optional schema | `jsonScore`, `stripFence` |
+| `LANG_MISMATCH` | Answered in the wrong language | Function-word profile (coarse, opt-in) | `languageMismatchScore`, `languageProfile`, `supportedLanguages` |
+
+Every detector is exported on its own if you only want one, and every name in
+that last column is covered by semver — see **Stability**.
 
-Every detector is exported on its own if you only want one.
+```ts
+import { repetitionScore, tailLoopDetail, stripFence } from 'llm-output-guard';
+
+repetitionScore(text);                  // 0..1, higher is worse
+repetitionScore(text, { n: 4 });        // n-gram size
+tailLoopDetail(text, { mode: 'char' }); // { score, mode } — which tokenizer ran
+stripFence('```json\n{"a":1}\n```');    // '{"a":1}'
+```
+
+Each takes `(text, options?)` and returns a `0..1` score, with three exceptions
+worth knowing: `shortnessScore(text, minChars)` takes its minimum positionally,
+`stripFence` returns a string, and `jsonScore` / `tailLoopDetail` return a detail
+object rather than a bare number. `supportedLanguages` is a value, not a
+function — the array `['id', 'en', 'es']`.
 
 ## Presets
 
@@ -390,6 +538,42 @@ the bulk and a cluster of outliers is real separation observed in your data
 rather than an assumption about rarity — and when that hole rests on one or
 two samples, the report says so.
 
+### The same thing, as a function
+
+The CLI is a wrapper. If your scores already live somewhere the shell cannot
+reach them — a metrics store, a warehouse query, a test — call `calibrate`
+directly. It takes the same flat objects the JSONL format describes:
+
+```ts
+import { calibrate } from 'llm-output-guard';
+
+const { n, summaries } = calibrate(
+  [
+    { REPETITION: 0.03, TAIL_LOOP: 0 },
+    { REPETITION: 0.91, TAIL_LOOP: 0.88, modes: { TAIL_LOOP: 'char' } },
+    // ...one entry per logged verdict
+  ],
+  { falsePositiveRate: 0.001 },
+);
+
+for (const s of summaries) {
+  s.code;               // 'REPETITION'
+  s.mode;               // 'word' | 'char', when the samples recorded one
+  s.suggested;          // threshold flagging falsePositiveRate of this sample
+  s.gap;                // { below, above, count, share } | null — stronger evidence
+  s.distribution;       // { n, nonZero, min, max, p50, p90, p99, p999 }
+  s.caveats;            // everything that makes `suggested` untrustworthy
+}
+```
+
+`modes` rides along in the same object and is not read as a score. Log it, and
+`summaries` comes back segmented — one entry per `code`+`mode` — for the reason
+in the paragraph above. `summarise(code, scores, options)` is exported too, for
+when you have one detector's numbers already grouped.
+
+**Read `caveats` before `suggested`.** It is where a sample too small for the
+requested rate says so, and a `suggested` number carries no warning of its own.
+
 ## On thresholds
 
 A miss is annoying. **A false positive is worse**: a healthy response gets discarded and retried against a slower provider for nothing.
@@ -531,13 +715,15 @@ onward; under `0.x` they described an intent, and the surface was frozen — exp
 by export — in the 1.0.0 release.
 
 **The public API is:** everything exported from `llm-output-guard`, plus
-`outputGuard` / `OutputGuardOptions` / `DegenerateAction` from `./ai-sdk` and
-`withOutputGuard` / `OutputGuardOptions` / `DegenerateAction` from `./openai`.
-Each subpath is its own contract; the two adapters share an internal base type
-today and are free to diverge, so an option added to one is not a promise about
-the other. Anything not exported from those three entry points is internal, has
-no stability guarantee, and may move in any release. The list is asserted in
-`test/surface.test.ts`, so an export cannot join it by accident.
+`outputGuard` / `OutputGuardOptions` / `DegenerateAction` from `./ai-sdk`, and
+`withOutputGuard` / `OutputGuardOptions` / `DegenerateAction` from each of
+`./openai` and `./anthropic`. Each subpath is its own contract; the adapters
+share internal base types today and are free to diverge, so an option added to
+one is not a promise about the others. Anything not exported from those four
+entry points is internal, has no stability guarantee, and may move in any release
+— `internal/proxy-guard.ts` and `internal/tool-calls.ts` included, however much
+behaviour they carry. The list is asserted in `test/surface.test.ts`, so an
+export cannot join it by accident.
 
 **Threshold and preset values are behaviour, not implementation.** This is the
 interesting case, so it gets a rule of its own:
@@ -577,6 +763,8 @@ patches.
 ## Limitations
 
 - Not a hallucination detector. It measures *shape*, never truth.
+- Tool *arguments* are not checked, only the prose beside them. A model that loops inside a JSON argument string is invisible here — your provider validates those against the schema you gave it.
+- `openai`'s `responses.stream()` helper is not wrapped. See the note above; `create({ stream: true })` is.
 - `REPETITION` does not work on Chinese, Japanese or Thai. See above — this is a known, measured gap, not an oversight.
 - Language detection is a function-word heuristic covering `id`/`en`/`es`. Opt-in, and unreliable under 25 words.
 - Truncation from a missing full stop is weak evidence, scored 0.55 and left below the default thresholds on purpose. Lower `maxTruncation` to ~0.5 to catch it, and expect false positives.

package/dist/{adapter-options-DafMCHtz.d.ts → adapter-options-Da4mVjXh.d.ts}

@@ -1,4 +1,4 @@
-import { V as Verdict } from './stream-Be2WhoKn.js';
+import { V as Verdict } from './stream-D-GVZ6iE.js';
 
 /**
  * The option surface every provider adapter shares.

package/dist/{adapter-options-DQQaELAX.d.cts → adapter-options-LkLsvNy6.d.cts}

@@ -1,4 +1,4 @@
-import { V as Verdict } from './stream-Be2WhoKn.cjs';
+import { V as Verdict } from './stream-D-GVZ6iE.cjs';
 
 /**
  * The option surface every provider adapter shares.

package/dist/ai-sdk.cjs

@@ -154,8 +154,14 @@ function stripFence(text) {
   const fenced = text.trim().match(/^```(?:json)?\s*([\s\S]*?)\s*```$/i);
   return fenced ? fenced[1] : text.trim();
 }
+function describe(issue) {
+  const path = (issue.path ?? []).map(
+    (segment) => typeof segment === "object" && segment !== null && "key" in segment ? String(segment.key) : String(segment)
+  ).join(".");
+  return path ? `${path}: ${issue.message}` : issue.message;
+}
 function jsonScore(text, options = {}) {
-  const { allowFence = true, requiredKeys = [] } = options;
+  const { allowFence = true, requiredKeys = [], schema } = options;
   const candidate = allowFence ? stripFence(text) : text.trim();
   let value;
   try {
@@ -173,6 +179,19 @@ function jsonScore(text, options = {}) {
       return { score: 1, value, reason: "missing-keys", missingKeys: missing };
     }
   }
+  if (schema) {
+    const result = schema["~standard"].validate(value);
+    if (typeof result?.then === "function") {
+      throw new TypeError(
+        "llm-output-guard: `schema` must validate synchronously, and this one returned a promise. checkOutput is synchronous by design. Remove the async refinement, or validate the payload yourself after checkOutput returns."
+      );
+    }
+    const sync = result;
+    if (sync.issues) {
+      return { score: 1, value, reason: "schema", issues: sync.issues.map(describe) };
+    }
+    return { score: 0, value: sync.value };
+  }
   return { score: 0, value };
 }
 
@@ -297,14 +316,15 @@ function checkOutput(text, options = {}) {
   if (opts.expectJson) {
     const result = jsonScore(text, {
       allowFence: opts.allowJsonFence,
-      requiredKeys: opts.requiredKeys
+      requiredKeys: opts.requiredKeys,
+      schema: opts.schema
     });
     parsedJson = result.value;
     add(
       "INVALID_JSON",
       result.score,
       0,
-      result.reason === "missing-keys" ? `JSON is missing required keys: ${result.missingKeys?.join(", ")}.` : "Response is not parseable JSON."
+      result.reason === "missing-keys" ? `JSON is missing required keys: ${result.missingKeys?.join(", ")}.` : result.reason === "schema" ? `JSON does not match the schema: ${result.issues?.join("; ")}.` : "Response is not parseable JSON."
     );
   }
   if (opts.expectLang) {
@@ -405,7 +425,20 @@ function createStreamGuard(options = {}) {
   };
 }
 
+// src/internal/tool-calls.ts
+var TOOL_CALL_PREAMBLE = {
+  minLength: 0,
+  maxTruncation: null,
+  expectJson: false,
+  finishReason: void 0
+};
+function checkPreamble(text, options) {
+  if (text.trim().length === 0) return null;
+  return checkOutput(text, { ...options, ...TOOL_CALL_PREAMBLE });
+}
+
 // src/ai-sdk.ts
+var isToolPart = (part) => part.type.startsWith("tool-");
 function finishReasonOf(value) {
   if (typeof value === "string") return value;
   if (value && typeof value === "object") return value.unified ?? value.raw;
@@ -443,7 +476,13 @@ function outputGuard(options = {}) {
       doGenerate
     }) {
       const result = await doGenerate();
-      const text = (result.content ?? []).filter((part) => part.type === "text").map((part) => part.text ?? "").join("");
+      const content = result.content ?? [];
+      const text = content.filter((part) => part.type === "text").map((part) => part.text ?? "").join("");
+      if (content.some(isToolPart)) {
+        const verdict = checkPreamble(text, guardOptions);
+        if (verdict) act(verdict, false);
+        return result;
+      }
       act(
         checkOutput(text, {
           ...guardOptions,
@@ -459,12 +498,14 @@ function outputGuard(options = {}) {
       const result = await doStream();
       const guard = createStreamGuard(guardOptions);
       let fired = false;
+      let sawToolCall = false;
       let finishReason;
       const guarded = result.stream.pipeThrough(
         new TransformStream({
           transform(part, controller) {
             controller.enqueue(part);
             if (part.type === "finish") finishReason = part.finishReason;
+            if (isToolPart(part)) sawToolCall = true;
             if (part.type !== "text-delta" || fired) return;
             const verdict = guard.push(part.delta ?? "");
             if (!verdict || verdict.ok) return;
@@ -478,9 +519,13 @@ function outputGuard(options = {}) {
             }
           },
           flush() {
-            if (!fired) {
-              onVerdict?.(guard.end(finishReasonOf(finishReason)), { streaming: true });
+            if (fired) return;
+            if (sawToolCall) {
+              const verdict = checkPreamble(guard.text, guardOptions);
+              if (verdict) onVerdict?.(verdict, { streaming: true });
+              return;
             }
+            onVerdict?.(guard.end(finishReasonOf(finishReason)), { streaming: true });
           }
         })
       );