llm-output-guard 1.12.0 → 1.12.1

package/docs/adapters.md

@@ -0,0 +1,500 @@
+# Provider adapters
+
+One wrap per client. Every adapter drives the same `createStreamGuard`;
+none of them reimplements it.
+
+### Vercel AI SDK
+
+One wrap, and both `generateText` and `streamText` are guarded:
+
+```ts
+import { wrapLanguageModel } from 'ai';
+import { outputGuard } from 'llm-output-guard/ai-sdk';
+import { presets } from 'llm-output-guard';
+
+const model = wrapLanguageModel({
+  model: groq('llama-3.3-70b-versatile'),
+  middleware: outputGuard({ ...presets.chat, onDegenerate: 'throw' }),
+});
+```
+
+On `streamText` this cancels the provider's stream mid-generation. Driven
+through the real SDK over a **mock part stream**, the source was pulled for **17
+of 137 parts** before the guard cut it off. That figure is parts never requested
+from a stub, not tokens never billed by a provider: the SDK's cancellation path
+is exercised for real, the thing on the other end of it is not. On
+`generateText` the tokens are already bought, so it throws
+`DegenerateOutputError` instead, which your fallback layer can act on.
+
+`onDegenerate` takes `'throw'` (default, also cancels the stream), `'abort'`
+(stop cleanly, keep what arrived), or `'ignore'`. Start with `'ignore'` plus
+`onVerdict` to watch your own traffic before letting a threshold fail anything:
+
+```ts
+outputGuard({
+  ...presets.chat,
+  onDegenerate: 'ignore',
+  onVerdict: (verdict, { streaming }) => metrics.record(verdict.scores, { streaming }),
+});
+```
+
+`ai` is an **optional peer dependency** — importing the subpath does not pull it
+in, and the main entry point has no peers at all. Supported: **`ai` v5, v6 and
+v7**. CI installs the packed tarball against each of those and both typechecks
+and runs the adapter, so the range is one that has been executed rather than
+assumed.
+
+**`ai` v4 is not supported, and forcing it will look like a bug in this
+package.** v4's middleware hands back `text` where v5+ hands back a `content`
+array, and streams `{ textDelta }` where v5+ streams `{ delta }`. This adapter
+reads the v5+ shape, so on v4 it sees the empty string for every response —
+which means **every healthy generation is flagged `EMPTY`, and under the default
+`onDegenerate: 'throw'` every call throws `DegenerateOutputError`.** It is not
+that the guard misses things on v4; it rejects everything. The peer range now
+refuses the install so you find out at `npm install` rather than in production.
+If you are pinned to v4, do not override it — stay on the core entry point and
+call `checkOutput` on the result yourself.
+
+### OpenAI SDK — and anything speaking its protocol
+
+One wrap, and both APIs are guarded — `chat.completions.create` and
+`responses.create`, streaming and not:
+
+```ts
+import OpenAI from 'openai';
+import { withOutputGuard } from 'llm-output-guard/openai';
+import { presets } from 'llm-output-guard';
+
+const client = withOutputGuard(new OpenAI(), {
+  ...presets.chat,
+  onDegenerate: 'throw',
+});
+
+await client.chat.completions.create({ model, messages });   // guarded
+await client.responses.create({ model, input });             // guarded
+```
+
+### `throw` stops a bad answer. `abort` saves tokens.
+
+They are not two strengths of the same setting, and picking the wrong one is
+the quiet failure this package exists to catch:
+
+| | non-streaming call | stream |
+|---|---|---|
+| `'throw'` (default) | **fails the call** with `DegenerateOutputError` | fails it *and* cancels the request |
+| `'abort'` | reports to `onVerdict`, **returns the response** | ends the stream, keeps what arrived |
+| `'ignore'` | reports only | reports only |
+
+**`'abort'` on a non-streaming call does not stop anything.** There is no stream
+to end, so the guard measures the response, reports it, and hands it back — a
+guard you believe in and do not have. It is the right choice on a stream where a
+partial answer beats no answer, and only there.
+
+`'throw'` is the default, so the shortest correct wrap is the one with no
+`onDegenerate` at all.
+
+
+The Responses API spells its stop reason `incomplete_details.reason` rather than
+`finish_reason`, and its length stop `max_output_tokens` rather than `length`.
+Both are mapped, so `TRUNCATED` fires the same way on either. `content_filter`
+is deliberately *not* read as truncation — a filtered response is a different
+failure, and reporting it as `TRUNCATED` would send a retry layer after the
+wrong fix.
+
+### `responses.stream()` needed a different mechanism
+
+It is guarded, and it was the last thing here that wasn't. A `ResponseStream` is
+an `EventStream`, and it can be read **six** ways:
+
+| path | detected | cancelled | throws |
+|---|---|---|---|
+| `for await (… of stream)` | yes | yes | yes |
+| `await stream.finalResponse()` | yes | yes | yes |
+| `await stream.done()` | yes | yes | yes |
+| `stream.on('…', cb)` | yes | yes | — |
+| `stream.once()` / `.emitted()` | yes | yes | — |
+| `stream.events('…')` | yes | yes | — |
+
+Replacing the iterator would have covered the first row and left five reading an
+unchecked stream, which is why this shipped unguarded until 1.12.0 — half a
+guard on the SDK's default streaming surface looks like coverage and is not.
+
+What makes it work is that all six are fed by one event pump, so **one listener
+sees every delta however you read it**, and `abort()` reaches the transport for
+all of them. Throwing is the part that cannot be universal: three paths have an
+error channel and three do not. A caller who only attached `on()` callbacks gets
+the `onVerdict` report and a stream that stops — which is the half that saves
+tokens — but no exception, because there is nowhere to put one.
+
+`onDegenerate: 'abort'` ends the iterator cleanly and keeps what arrived.
+`finalResponse()` and `done()` still reject under it, because there is no
+partial value either can honestly return.
+
+This is also how you guard **Groq, Together, OpenRouter, Fireworks, DeepInfra,
+vLLM and Ollama** — anything you reach through an OpenAI-compatible `baseURL`
+works, because the adapter is typed against the wire shapes rather than against
+OpenAI the company.
+
+Non-streaming calls are already paid for by the time anything can run, so a
+degenerate one throws `DegenerateOutputError` for your fallback layer to catch:
+
+```ts
+const completion = await client.chat.completions.create({ model, messages });
+```
+
+Streaming is where it pays. The guard watches deltas and **cancels the HTTP
+request** the moment a loop is detectable:
+
+```ts
+const stream = await client.chat.completions.create({ model, messages, stream: true });
+for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
+```
+
+Driven through the real SDK against a looping model over a **mock transport**,
+the response body was cancelled after **16 of 135 chunks** were generated — 88%
+of the chunks were never produced.
+
+**What that number is, precisely:** chunks a mock server was never asked to
+produce after the client closed the connection, measured against an unguarded
+baseline of the full 135. It is stronger than a "did abort fire" assertion —
+the test observes cancellation at the response body, so a guard that stopped
+iterating while the connection stayed open would fail it. It is **not** a
+billing figure. A real provider sits behind buffering, its own chunking, and
+server-side generation that may already have run ahead of what it has sent;
+none of that exists in the mock. Treat 88% as evidence that cancellation
+reaches the transport promptly, and measure your own provider before putting a
+number in a budget.
+
+`onDegenerate` and `onVerdict` are the same options as the Vercel adapter, from
+the same type — `'throw'` (default, also cancels the stream), `'abort'` (stop
+cleanly, keep what arrived), or `'ignore'`. Start with `'ignore'` plus
+`onVerdict` to watch your own traffic first:
+
+```ts
+withOutputGuard(new OpenAI(), {
+  ...presets.chat,
+  onDegenerate: 'ignore',
+  onVerdict: (verdict, { streaming }) =>
+    metrics.record(verdict.scores, { streaming, modes: verdict.modes }),
+});
+```
+
+`openai` is an **optional peer dependency**, and the wrapper is a proxy: every
+other method on the client, and `create()`'s own `.withResponse()`, pass through
+untouched. `finish_reason: 'length'` is mapped into the final check, so
+`TRUNCATED` fires on a response that hit `max_tokens`.
+
+**What runs when.** Mid-stream only the redundancy detectors are meaningful:
+partial output is genuinely short, genuinely cut off, and genuinely not valid
+JSON, so `TOO_SHORT`, `TRUNCATED`, `INVALID_JSON` and `LANG_MISMATCH` would
+fire on every healthy generation and teach you to ignore the guard. They are
+deferred to `end()`. `LOW_ENTROPY` is deferred too, for cost — it is ~100x the
+other detectors, and everything it would have caught early is caught by
+`REPETITION`, or by `TAIL_LOOP`'s character mode on non-spaced scripts.
+
+Every adapter shares this behaviour because they all drive the same
+`createStreamGuard`. None of them reimplements it.
+
+### Anthropic SDK
+
+Same one wrap, same options:
+
+```ts
+import Anthropic from '@anthropic-ai/sdk';
+import { withOutputGuard } from 'llm-output-guard/anthropic';
+import { presets } from 'llm-output-guard';
+
+const client = withOutputGuard(new Anthropic(), {
+  ...presets.chat,
+  onDegenerate: 'throw',
+});
+
+await client.messages.create({ model, max_tokens, messages });               // guarded
+await client.messages.create({ model, max_tokens, messages, stream: true }); // guarded
+```
+
+Two things are specific to this API:
+
+**Extended thinking is not read as the answer.** `thinking` blocks are the
+model's reasoning, they are often longer than the answer, and they repeat
+themselves as a matter of course while working a problem. Folding them into the
+measured text would raise every repetition score on every thinking response and
+flag the ones that thought hardest — so only `text` blocks are measured, and a
+`thinking` block is not mistaken for a tool call either.
+
+**Both of Anthropic's length stops map to `TRUNCATED`.** `max_tokens` passes
+straight through; `model_context_window_exceeded` is the same event under a
+different name and is normalised in the adapter. `refusal` is deliberately *not*
+truncation — a refusal is a complete response that says no, which is a content
+judgement this package does not make.
+
+> **`messages.stream()` is not guarded**, for the same reason `responses.stream()`
+> isn't: it returns a `MessageStream` — an event emitter with `.on()` and
+> `.finalMessage()` — and guarding only its iteration would leave
+> `.finalMessage()` unchecked. Use `create({ stream: true })`, or run
+> `checkOutput` on `await stream.finalMessage()` yourself. `messages.batches` is
+> unguarded too, and less interestingly: a batch is retrieved later as a file of
+> results, so there is no response at `create` time to inspect.
+
+`@anthropic-ai/sdk` is an **optional peer dependency**, declared
+`>=0.60.0 <1.0.0` and verified at 0.60.0, 0.90.0 and 0.117.1 — each installing
+the packed tarball and running the adapter for real, not just typechecking.
+
+### Google Gemini SDK
+
+Same one wrap, same options — but Gemini splits streaming into its own method
+rather than putting it behind a flag, so both are named:
+
+```ts
+import { GoogleGenAI } from '@google/genai';
+import { withOutputGuard } from 'llm-output-guard/google';
+import { presets } from 'llm-output-guard';
+
+const ai = withOutputGuard(new GoogleGenAI({ apiKey }), {
+  ...presets.chat,
+  onDegenerate: 'throw',
+});
+
+await ai.models.generateContent({ model, contents });        // guarded
+await ai.models.generateContentStream({ model, contents });  // guarded
+```
+
+Three things are specific to this API:
+
+**Cancelling a stream works differently here, and it matters.**
+`generateContentStream` resolves to a bare `AsyncGenerator`. Unlike the OpenAI
+and Anthropic streams it carries no `AbortController`, so there is nothing on it
+to abort — and ending our own iteration would leave the model generating, and
+billing, while we looked away. The adapter therefore puts a signal *into* the
+outgoing request as `config.abortSignal`, which is what reaches `fetch`. Your
+request object is cloned rather than mutated, and a signal you supplied yourself
+keeps working: ours follows yours rather than replacing it.
+
+**Thought summaries are not read as the answer.** Gemini carries thinking in the
+same `text` field as the answer, flagged `thought: true`, so that flag is the
+only thing separating the two. Measured as answer text, reasoning would raise
+every repetition score on every thinking-enabled response — the same reason
+`./anthropic` skips `thinking` blocks.
+
+**Only the first candidate is judged.** `candidateCount` above 1 asks for
+*alternatives* to the same question and only one of them gets used.
+Concatenating them would measure a document nobody receives, and measure it
+wrongly in one direction: two good answers to one question share their subject,
+vocabulary and often their structure, so the more consistent the model was the
+higher its repetition score would climb.
+
+`MAX_TOKENS` maps to `TRUNCATED` — it lands on the length stop the detector
+already knows. Gemini's content stops (`SAFETY`, `RECITATION`, `BLOCKLIST`,
+`PROHIBITED_CONTENT`, `SPII`) deliberately do **not**: each is a decision about
+what the model may say rather than the model losing the thread, and reading one
+as truncation would discard the response and spend a retry earning the identical
+stop again.
+
+> **`chats` is not guarded.** `chats.create()` returns a `Chat` that holds its
+> own history, so guarding it means wrapping a value returned from a method
+> rather than a response — and deciding what a degenerate turn does to history
+> that has already been appended. Left plainly unguarded rather than half-done.
+> Use `models.generateContent`, or run `checkOutput` on the result yourself.
+
+`@google/genai` is an **optional peer dependency**, declared `>=1.0.0 <3.0.0`
+and verified at 1.0.0, 1.9.0, 2.0.0 and 2.19.0 — each installing the packed
+tarball and running the adapter for real, including the cancellation, not just
+typechecking.
+
+### Tool calls and agents
+
+A model that answers by calling a tool returns no assistant text — OpenAI sends
+`content: null` beside `tool_calls`, and the AI SDK sends a `content` array with
+no `text` part. Handed to `checkOutput`, that is an empty string, and an empty
+string scores `EMPTY`.
+
+So **the presence of tool calls means the text, if any, is a preamble rather
+than the answer**, and every adapter judges it as one:
+
+| | On a tool-calling turn |
+|---|---|
+| No text at all | Nothing is judged, and nothing is reported to `onVerdict` |
+| Text beside the call | `REPETITION`, `TAIL_LOOP` and `LOW_ENTROPY` still run |
+| `TOO_SHORT` | Off — "Let me look that up" is sixteen characters and correct |
+| `TRUNCATED` | Off — a preamble ends without terminal punctuation as a matter of course |
+| `INVALID_JSON` | Off — the JSON is in the call arguments, which your provider already validated against the schema |
+
+The redundancy detectors stay on because a model looping in its preamble is
+still a model that is looping. `EMPTY` is not disarmed either: a response with
+neither text nor tool calls still fails, which is the case this package exists
+for.
+
+Nothing is reported to `onVerdict` for a text-free tool call on purpose. Those
+samples are what a `calibrate` run is built from, and a spike of `EMPTY: 1` in
+them would describe your agent's tool use rather than any degeneration.
+
+---
+
+---
+
+[← Back to the README](../README.md) · [Try the playground](https://edwinsatya.github.io/llm-output-guard/)
+
+## Building an agent turn
+
+Every subpath above also exports `toTurn`, which maps that provider's response
+onto the shape `llm-output-guard/agent` reads:
+
+```ts
+import { toTurn } from 'llm-output-guard/openai';
+guard.observe(toTurn(completion));
+```
+
+It carries the same provider knowledge the guard beside it does — thought parts
+excluded on Gemini, thinking blocks on Anthropic, first choice or candidate
+only. **[docs/agent-loops.md](agent-loops.md#building-a-turn-from-your-provider)**
+
+### What is covered
+
+| Provider | Maps | Via |
+|---|---|---|
+| OpenAI — `chat.completions` and `responses` | fully | `./openai` |
+| Anthropic | fully | `./anthropic` |
+| Google Gemini | fully | `./google` |
+| Vercel AI SDK | fully | `./ai-sdk` |
+| Groq, Together, OpenRouter, Fireworks, DeepInfra, vLLM | fully | `./openai` |
+| Ollama — OpenAI-compatible endpoint | fully | `./openai` |
+| Azure OpenAI | fully | `./openai` — it *is* the `openai` SDK |
+| Mistral SDK | fully | `./openai` — see below |
+| AWS Bedrock, Cohere, LangChain, Ollama's native `/api/chat` | **nothing** | not adapted |
+
+**Full, or nothing.** An unsupported envelope maps to an empty turn, which a
+trace drops. There is deliberately no partial outcome, and that is the invariant
+worth understanding: a turn with its text read and its calls missed
+fingerprints by its *preamble prose* instead of its arguments, so an agent that
+reuses one preamble while working through twenty files reads as a total
+collapse. A false positive on a healthy run is the failure this package treats
+as worse than a miss. The table is asserted in `test/openai-alikes.test.ts`.
+
+**Mistral is why that invariant is written down.** Its generated SDK renames
+`tool_calls` to `toolCalls`, and the rename broke more than the mapper: a
+tool-calling turn carries no prose, so an unrecognised call list left
+`content: ''` — which scores `EMPTY: 1`. Every tool call from a Mistral client
+aborted under `withOutputGuard`. Both spellings are now read everywhere the list
+is, which is safe rather than merely convenient: no response in the OpenAI
+family carries both fields, so there is nothing to disambiguate.
+
+## Tool-call arguments
+
+A tool-calling turn is judged by its preamble — the text beside the call — because
+that text is not the answer. Which leaves the answer itself, the arguments,
+unmeasured. Before 1.5 nothing here looked at them at all.
+
+```ts
+const client = withOutputGuard(new OpenAI(), {
+  ...presets.chat,
+  checkToolArguments: true,
+});
+```
+
+Your provider already validates arguments against the schema you declared. That
+covers **types**, not content — and a model that loops does not produce the wrong
+type, it produces a valid string with nothing in it:
+
+```json
+{ "query": "site reliability engineering site reliability engineering …", "limit": 10 }
+```
+
+Schema-valid. Passes the provider. Reaches your tool as a garbage search, or,
+if the tool writes, persists the loop.
+
+**What is measured.** Redundancy only, per string value — `REPETITION` and
+`TAIL_LOOP`, under the same thresholds your guard already uses. The reason codes
+are unchanged so existing handling keeps working; the `message` says the loop was
+found in an argument rather than in the prose.
+
+**What is not, and why.** `LOW_ENTROPY` is off because JSON is legitimately
+repetitive at the character level, the same reason `presets.strictJson` turns it
+off. `TRUNCATED`, `TOO_SHORT` and `INVALID_JSON` are off because the provider
+already guarantees the arguments parse and match your schema — they would be
+answering a question that has been answered. `SCRIPT_MISMATCH` and
+`LANG_MISMATCH` are off because an argument is not prose addressed to a user: a
+search query in another language is ordinary, not degenerate.
+
+Values are measured **individually**, never as a serialised document. Two calls
+against the same schema are legitimately near-identical documents, and an array
+of repeated records inside one argument is the shape that was asked for — the
+same rule `redundancyScope: 'jsonValues'` follows, for the same reason.
+
+**Nothing is measured when there is nothing to measure.** A tool that takes no
+parameters is called with `{}`, and arguments carrying no strings at all — 
+`{"lat":-6.2,"zoom":11}` — have no prose to judge. Both are skipped rather than
+scored, so a no-argument tool does not fail on `EMPTY`.
+
+**Non-streaming responses only.** Arguments arrive as JSON fragments that do not
+parse until the call is complete, so there is nothing meaningful to measure
+mid-stream.
+
+Off by default. Switching it on can only make a response fail that previously
+passed, so it is opt-in like every other behaviour change in this package.
+
+## Prompt echo
+
+`PROMPT_ECHO` catches a model that replays your prompt instead of answering it.
+It needs the prompt, and a guard is configured once when you wrap the client
+while the prompt changes on every call — so this is a switch rather than a
+value, and the adapter reads the prompt out of each request it is already
+forwarding.
+
+```ts
+const client = withOutputGuard(new OpenAI(), {
+  ...presets.chat,
+  checkPromptEcho: true,
+});
+```
+
+Works on all three adapters, and each reads its own request shape:
+`messages` for `chat.completions`, `instructions` plus `input` for `responses`,
+`system` plus `messages` for Anthropic, and the spec's normalised `params.prompt`
+for the AI SDK. Content is read whether it is a plain string or a list of parts;
+non-text parts (images, audio, files) contribute nothing, which is correct — they
+are not text the model could echo back.
+
+**Prior assistant turns are excluded.** The failure being measured is a model
+replaying its *input*, and a model that keeps its terminology consistent across a
+long conversation is doing its job. Counting its own earlier answers would make
+that look worse the longer the conversation ran. Nothing is lost: a model that
+loses the turn boundary and replays the whole transcript replays the system and
+user text too.
+
+**An explicit `prompt` in the options wins.** If you have a fixed system prompt
+and prefer to state it once when wrapping the client, the adapter does not
+override it.
+
+**On a stream the check runs at the end**, not on the mid-stream windows,
+because the score is a share of the whole output and a trailing window measures
+the share of that window. See
+[docs/detectors.md](./detectors.md#returning-the-prompt-instead-of-an-answer)
+for the numbers.
+
+> **Do not enable this on a rewrite, translate, summarise or extract endpoint.**
+> Copying from the input is the job on those, so a correct answer scores high and
+> the detector measures the task rather than a failure.
+
+## Aborting a wrong-language stream early
+
+`SCRIPT_MISMATCH` and `PROMPT_ECHO` are judged at the end of a stream by
+default, which means the whole response is already paid for by the time either
+fires. `earlyDocumentChecks: true` lets them judge the buffer as it arrives:
+
+```ts
+const client = withOutputGuard(new OpenAI(), {
+  ...presets.chat,
+  expectScript: 'latin',
+  earlyDocumentChecks: true,
+  onDegenerate: 'abort',
+});
+```
+
+Measured on a 3,000-character response answered entirely in the wrong script:
+**7 of 30 chunks consumed instead of 30**, with the upstream request cancelled —
+the same transport-level abort the redundancy detectors already trigger.
+
+It is off by default and carries a measured false-positive risk, because the
+buffer is a prefix and a prefix over-reports both detectors. Read
+[docs/streaming.md](./streaming.md#judging-the-whole-response-early) before
+switching it on — the floor, the bar, and the numbers behind both are there.