llm-output-guard 1.3.1 → 1.4.0

package/README.md

@@ -1,9 +1,15 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/edwinsatya/llm-output-guard/main/assets/logo.svg"
+       width="240" alt="llm-output-guard" />
+</p>
+
 # llm-output-guard
 
 **Detect LLM responses that failed while returning `200 OK`.**
 
 [![npm](https://img.shields.io/npm/v/llm-output-guard?color=0b7285)](https://www.npmjs.com/package/llm-output-guard)
-[![minzipped](https://img.shields.io/bundlephobia/minzip/llm-output-guard?color=0b7285&label=min%2Bgzip)](https://bundlephobia.com/package/llm-output-guard)
+[![downloads](https://img.shields.io/npm/dm/llm-output-guard?color=0b7285)](https://www.npmjs.com/package/llm-output-guard)
+[![minzipped](https://img.shields.io/bundlejs/size/llm-output-guard?color=0b7285&label=min%2Bgzip)](https://bundlejs.com/?q=llm-output-guard)
 [![dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)](https://github.com/edwinsatya/llm-output-guard/blob/main/package.json)
 [![CI](https://github.com/edwinsatya/llm-output-guard/actions/workflows/ci.yml/badge.svg)](https://github.com/edwinsatya/llm-output-guard/actions/workflows/ci.yml)
 [![license](https://img.shields.io/npm/l/llm-output-guard?color=0b7285)](./LICENSE)
@@ -12,7 +18,7 @@ Your retry layer watches for `429`, `5xx` and timeouts. It cannot see a model th
 looped until `max_tokens`, returned `{}`, stopped mid-sentence, or answered in the
 wrong language — because all of those arrive as a **successful request**.
 
-This produces the signal that layer is missing. Zero dependencies, ~3 KB gzipped,
+This produces the signal that layer is missing. Zero dependencies, ~5 KB gzipped,
 synchronous, no network.
 
 ```bash
@@ -50,7 +56,8 @@ every detector, running on your own pasted output. No API key, no request.
 | `LOW_ENTROPY` | Character-level collapse, token artifacts | Compression ratio |
 | `TRUNCATED` | Cut off mid-thought | `finish_reason`, unbalanced fences |
 | `INVALID_JSON` | Prose around the payload, wrong types | Parse + key + schema contract |
-| `LANG_MISMATCH` | Answered in the wrong language | Function-word profile (opt-in) |
+| `SCRIPT_MISMATCH` | Answered in the wrong alphabet | Share of letters outside the expected scripts (opt-in) |
+| `LANG_MISMATCH` | Answered in the wrong language, same alphabet | Function-word profile (opt-in) |
 
 Every detector runs even after one fails, so a verdict shows the whole picture
 rather than whichever check happened to be ordered first. Each returns **0–1, not
@@ -141,6 +148,28 @@ See **[docs/detectors.md](docs/detectors.md)** for arrays of repeated records
 a JSON array of identical rows reads as a loop under the default scope, and
 `redundancyScope: 'jsonValues'` is the fix.
 
+### Answering in the wrong language
+
+```ts
+checkOutput(raw, { ...presets.chat, expectScript: 'latin' });
+```
+
+A model that ignores "answer in English" does not produce broken English, it
+produces fluent Chinese. That is a `200 OK` your retry layer cannot see, and it
+is detectable by counting characters — no word list, no model, decisive from
+about a dozen letters. A response answered entirely in the wrong script scores
+**1.000**; a healthy response measured against its own script scores
+**0.000–0.028**.
+
+Pass every script the answer may legitimately contain — `['han', 'latin']` for
+Chinese, `['han', 'kana', 'latin']` for Japanese. `'latin'` belongs in nearly all
+of them, because a Chinese answer about React still contains `useEffect`. Code
+fences, inline code and URLs are removed before measuring, so a TypeScript block
+never counts as answering in English.
+
+Same script means no signal: Spanish against English scores 0. That is what
+`expectLang` is for, and the two compose under separate codes.
+
 ## The verdict
 
 ```ts
@@ -191,7 +220,8 @@ because only one of those is evidence. Full guide:
 Korean, Cyrillic, Greek, Arabic and Devanagari separate words and are handled like
 English. **Chinese, Japanese and Thai do not**, so `TAIL_LOOP` switches to
 character mode and reads its own threshold. `REPETITION` is blind on those scripts —
-a known, measured gap, with the numbers behind it in
+a known, measured gap. `SCRIPT_MISMATCH` covers all ten scripts and is the one
+detector these are *not* the weak case for. Numbers behind both in
 **[docs/script-coverage.md](docs/script-coverage.md)**.
 
 ## Design notes
@@ -260,7 +290,8 @@ patches.
 - Tool *arguments* are not checked, only the prose beside them. A model that loops inside a JSON argument string is invisible here — your provider validates those against the schema you gave it.
 - `openai`'s `responses.stream()` helper is not wrapped. See the note above; `create({ stream: true })` is.
 - `REPETITION` does not work on Chinese, Japanese or Thai. See above — this is a known, measured gap, not an oversight.
-- Language detection is a function-word heuristic covering `id`/`en`/`es`. Opt-in, and unreliable under 25 words.
+- Language detection is a function-word heuristic covering `id`/`en`/`es`. Opt-in, and unreliable under 25 words. `expectScript` is the stronger check where the languages differ in alphabet, and says nothing where they do not.
+- `SCRIPT_MISMATCH` does not run mid-stream. A mid-stream check reads a trailing window, and the language of a window is not the language of the response — an English answer quoting a Chinese passage measures 0.114 whole and 0.500 over its last 400 characters.
 - Truncation from a missing full stop is weak evidence, scored 0.55 and left below the default thresholds on purpose. Lower `maxTruncation` to ~0.5 to catch it, and expect false positives.
 - A JSON array of repeated identical records reads as a loop under the default scope, and fails from three records up. Set `redundancyScope: 'jsonValues'` — see **Structured output**.
 - Thresholds calibrated on the bundled corpus. Yours will differ — and the word and character thresholds need calibrating **separately**, because they are separate distributions.

package/dist/{adapter-options-ClbcmvaY.d.ts → adapter-options-BUOzVp3T.d.ts}

@@ -1,4 +1,4 @@
-import { V as Verdict } from './stream-DUEP28_p.js';
+import { V as Verdict } from './stream-NO-TTGYs.js';
 
 /**
  * The option surface every provider adapter shares.

package/dist/{adapter-options-DpTwyLVK.d.cts → adapter-options-CQNPNjES.d.cts}

@@ -1,4 +1,4 @@
-import { V as Verdict } from './stream-DUEP28_p.cjs';
+import { V as Verdict } from './stream-NO-TTGYs.cjs';
 
 /**
  * The option surface every provider adapter shares.

package/dist/ai-sdk.cjs

@@ -236,6 +236,52 @@ function languageMismatchScore(text, expected, options = {}) {
   return Math.min(1, (best - target) / best);
 }
 
+// src/detectors/script.ts
+var SCRIPTS = {
+  latin: /\p{Script=Latin}/u,
+  han: /\p{Script=Han}/u,
+  kana: /[\p{Script=Hiragana}\p{Script=Katakana}]/u,
+  hangul: /\p{Script=Hangul}/u,
+  cyrillic: /\p{Script=Cyrillic}/u,
+  arabic: /\p{Script=Arabic}/u,
+  devanagari: /\p{Script=Devanagari}/u,
+  greek: /\p{Script=Greek}/u,
+  hebrew: /\p{Script=Hebrew}/u,
+  thai: /\p{Script=Thai}/u
+};
+var LETTER_OR_MARK = /[\p{L}\p{M}]/gu;
+var SCRIPT_NEUTRAL = /[\p{sc=Common}\p{sc=Inherited}]/u;
+var FENCED_BLOCK = /```[\s\S]*?```/g;
+var OPEN_FENCE_TO_END = /```[\s\S]*$/;
+var INLINE_CODE = /`[^`\n]*`/g;
+var URL = /\bhttps?:\/\/\S+/gi;
+var stripNonLinguistic = (text) => text.replace(FENCED_BLOCK, " ").replace(OPEN_FENCE_TO_END, " ").replace(INLINE_CODE, " ").replace(URL, " ");
+function scriptMismatchScore(text, expected, options = {}) {
+  const { minLetters = 12 } = options;
+  const wanted = Array.isArray(expected) ? expected : [expected];
+  const known = wanted.filter((name) => typeof name === "string" && Object.hasOwn(SCRIPTS, name));
+  if (known.length === 0) return 0;
+  const letters = judgeableLetters(text, options);
+  if (letters.length < minLetters) return 0;
+  let matched = 0;
+  for (const ch of letters) {
+    for (const name of known) {
+      if (SCRIPTS[name].test(ch)) {
+        matched++;
+        break;
+      }
+    }
+  }
+  return clamp01(1 - matched / letters.length);
+}
+function judgeableLetters(text, options) {
+  const { ignoreCode = true, maxSample = 8e3 } = options;
+  const source = ignoreCode ? stripNonLinguistic(text.slice(0, maxSample)) : text.slice(0, maxSample);
+  const found = source.match(LETTER_OR_MARK);
+  if (!found) return [];
+  return found.filter((ch) => !SCRIPT_NEUTRAL.test(ch));
+}
+
 // src/internal/json-scope.ts
 function stringValues(value, out = []) {
   if (typeof value === "string") out.push(value);
@@ -269,6 +315,7 @@ var DEFAULTS = {
   expectJson: false,
   allowJsonFence: true,
   maxLangMismatch: 0.6,
+  maxScriptMismatch: 0.5,
   ngram: 3,
   redundancyScope: "document"
 };
@@ -366,6 +413,16 @@ function checkOutput(text, options = {}) {
       result.reason === "missing-keys" ? `JSON is missing required keys: ${result.missingKeys?.join(", ")}.` : result.reason === "schema" ? `JSON does not match the schema: ${result.issues?.join("; ")}.` : "Response is not parseable JSON."
     );
   }
+  if (opts.expectScript) {
+    const wanted = Array.isArray(opts.expectScript) ? opts.expectScript : [opts.expectScript];
+    const s = scriptMismatchScore(text, opts.expectScript);
+    add(
+      "SCRIPT_MISMATCH",
+      s,
+      opts.maxScriptMismatch,
+      `${Math.round(s * 100)}% of letters are not in ${wanted.join(" or ")}.`
+    );
+  }
   if (opts.expectLang) {
     const s = languageMismatchScore(text, opts.expectLang);
     add(
@@ -401,6 +458,30 @@ var DEFERRED_TO_END = {
   expectJson: false,
   expectLang: null,
   finishReason: void 0,
+  /*
+   * SCRIPT_MISMATCH is deferred for a reason none of the others share: it is
+   * the *window* that makes it unsafe, not the partial text.
+   *
+   * Which language a model answered in is a property of the whole response.
+   * Every mid-stream check reads the trailing `window` characters, so what it
+   * would actually measure is the language of the last few paragraphs -- and an
+   * English answer that ends by quoting a Chinese passage is a real thing that
+   * happens. Measured on exactly that shape: 0.114 across the document, 0.206
+   * over the last 1000 characters, and 0.500 over the last 400. The document is
+   * healthy and the window says it is half wrong, so a smaller `window` would
+   * buy earlier detection by manufacturing false positives out of quotations.
+   *
+   * The detector itself is decisive from about a dozen letters, so there is a
+   * real early-abort here for someone who wants it -- it just has to read the
+   * whole buffer rather than a window:
+   *
+   *   const v = checkOutput(guard.text, { expectScript: 'latin' });
+   *
+   * Deliberately left to the caller. The guard cannot run one detector over a
+   * different span than the rest without becoming a per-detector span router,
+   * and this is one line for the callers who need it.
+   */
+  expectScript: null,
   /*
    * LOW_ENTROPY is deferred for a second reason: cost. The LZ77 pass is
    * 0.4ms at 500 characters and 11ms at its 4000-character sample cap, which