llm-output-guard 1.9.0 → 1.11.0

package/README.md

@@ -73,9 +73,12 @@ every detector, running on your own pasted output. No API key, no request.
 | `SCRIPT_MISMATCH` | Answered in the wrong alphabet | Share of letters outside expected scripts · opt-in |
 | `LANG_MISMATCH` | Wrong language, same alphabet | Function-word profile · opt-in |
 | `PROMPT_ECHO` | Returned your prompt instead of an answer | Share of output copied from the prompt · opt-in |
+| `AGENT_LOOP` | An agent run circling instead of advancing | Exact periodicity across turns · `llm-output-guard/agent` |
 
 Every detector runs even after one fails, so a verdict shows the whole picture
-rather than whichever check happened to be ordered first. Each returns **0–1, not
+rather than whichever check happened to be ordered first. All but the last read
+one response; `AGENT_LOOP` reads a *sequence* of them and comes from its own
+entry point, so `checkOutput` never returns it. Each returns **0–1, not
 a boolean** — you pick the line.
 
 **Full reference, with the measurements behind every default:
@@ -112,16 +115,23 @@ sequence.
 
 ```ts
 import { createAgentGuard } from 'llm-output-guard/agent';
+import { toTurn } from 'llm-output-guard/openai';
 
 const guard = createAgentGuard();
 
 while (!done) {
-  const response = await model.step();
-  const verdict = guard.observe({ text: response.text, toolCalls: response.toolCalls });
+  const completion = await client.chat.completions.create(params);
+  const verdict = guard.observe(toTurn(completion));
   if (!verdict.ok) break; // the run is circling; stop paying for it
 }
 ```
 
+`toTurn` ships from all four adapter subpaths and reads whatever that provider
+sends — a JSON-string argument from OpenAI, a parsed object from Anthropic,
+thought parts excluded on Gemini. Map a field by hand and get it wrong and
+nothing throws: every turn fingerprints differently, the score pins at 0.000,
+and you have a guard that never runs.
+
 `AGENT_LOOP` reads one axis nothing else here reads. It looks for an **exact**
 repeating cycle of turns, which is what keeps it off the shapes that dominate
 healthy agent traffic: twenty reads of twenty different files, an identical
@@ -132,6 +142,10 @@ A turn carrying tool calls is judged by its **arguments**, never its prose —
 that is the difference between an agent working through a list and an agent
 stuck on one item.
 
+**[Try it on a run →](https://edwinsatya.github.io/llm-output-guard/)** — the
+playground has an *agent run* mode: pick a trace, see which turns form the
+cycle. The healthy traps are the half worth looking at.
+
 **[docs/agent-loops.md](docs/agent-loops.md)**
 
 ## The hard part is not catching loops
@@ -235,6 +249,13 @@ responses you already have, then derive thresholds you can defend:
 npx llm-output-guard check logs/*.txt --json | npx llm-output-guard calibrate --fpr 0.001
 ```
 
+Agent runs calibrate the same way — `--trace` reads one run per line and scores
+`AGENT_LOOP` instead of the per-response detectors:
+
+```bash
+npx llm-output-guard check runs.jsonl --trace --json | npx llm-output-guard calibrate
+```
+
 `check` also works as a CI assertion — it exits 1 when anything is degenerate,
 2 when the input cannot be read.
 
@@ -250,7 +271,7 @@ because only one of those is evidence.
 - **Scores, not booleans.** Detectors report 0–1 and leave the threshold decision to you.
 - **Abstains rather than guesses.** Samples too short to judge score 0.
 - **Hand-rolled LZ77** rather than `node:zlib`, so the package stays runtime-agnostic.
-- **Sub-millisecond**, 0.383 ms at 500 B and 1.014 ms at 32 KB, with one detector accounting for most of it. `npm run bench` reproduces it — **[docs/performance.md](docs/performance.md)**
+- **Sub-millisecond**, 0.383 ms at 500 B and 1.014 ms at 32 KB, with one detector accounting for most of it. A 12-turn agent run costs 0.075 ms. `npm run bench` reproduces both — **[docs/performance.md](docs/performance.md)**
 - **Chinese, Japanese and Thai** are handled where they differ: `TAIL_LOOP` switches to character mode, `REPETITION` is blind and says so — **[docs/script-coverage.md](docs/script-coverage.md)**
 
 ## Stability
@@ -261,7 +282,8 @@ the public surface was frozen export by export in that release.
 **The public API is** everything exported from `llm-output-guard`, plus
 `outputGuard` / `OutputGuardOptions` / `DegenerateAction` from `./ai-sdk`,
 `withOutputGuard` / `OutputGuardOptions` / `DegenerateAction` from `./openai`,
-`./anthropic` and `./google`, and `checkTrace` / `assertTrace` /
+`./anthropic` and `./google`, `toTurn` from each of those four, and
+`checkTrace` / `assertTrace` /
 `createAgentGuard` / `agentLoopScore` / `agentLoopDetail` from `./agent`. Each
 subpath is its own contract, so an option added to one is not
 a promise about the others. Anything else is internal and may move in any

package/dist/agent-types-D310_xck.d.cts

@@ -0,0 +1,97 @@
+/**
+ * The shapes `./agent` reads.
+ *
+ * Structurally typed and provider-neutral, for the same reason the adapters
+ * are: a trace is assembled by the caller from whatever their framework hands
+ * back, and requiring a provider's own type here would make this subpath
+ * depend on a provider.
+ */
+/** One tool call in a turn. */
+interface AgentToolCall {
+    /**
+     * The tool's name. OpenAI spells it `function.name`, Anthropic and the AI SDK
+     * spell it `name`, and the caller maps whichever they have.
+     *
+     * Optional because a streamed call can arrive before its name does. A call
+     * with no name still fingerprints -- by its arguments -- rather than being
+     * dropped, since two nameless calls with identical arguments are still the
+     * same call twice.
+     */
+    name?: string | null;
+    /**
+     * The arguments, in any shape a provider sends: a parsed object, or the raw
+     * JSON string OpenAI uses. Both canonicalise to the same fingerprint, so a
+     * trace assembled from mixed sources still compares.
+     */
+    arguments?: unknown;
+}
+/** One model response in an agent run. */
+interface AgentTurn {
+    /** The prose the model produced, if any. */
+    text?: string | null;
+    /**
+     * The tool calls the model issued, if any.
+     *
+     * **A turn carrying tool calls is judged by them alone**, and its `text` is
+     * read as a preamble and ignored -- the same rule the single-response
+     * adapters already apply. See `internal/turn-fingerprint.ts` for the trap
+     * that rule exists to avoid.
+     */
+    toolCalls?: readonly AgentToolCall[] | null;
+}
+interface AgentCheckOptions {
+    /**
+     * How many trailing turns to inspect. Default 12.
+     *
+     * The same idea as `TAIL_LOOP`'s tail: what matters is whether the agent is
+     * stuck *now*, not whether it repeated itself twenty turns ago and recovered.
+     * A trace shorter than the window is measured whole.
+     *
+     * It also bounds the longest cycle that can be found, at `window / minRepeats`
+     * turns -- 4 at the defaults. Raise both to catch longer orbits, and expect
+     * a loop at the very end of a long trace to score lower, because the score is
+     * coverage of the window rather than a count.
+     */
+    window?: number;
+    /**
+     * Turns required before this will judge at all. Default 4.
+     *
+     * Below it the detector abstains, which is the rule everywhere else in this
+     * package. Two identical turns is a retry, and three is a short poll; neither
+     * is evidence that an agent has stopped advancing.
+     */
+    minTurns?: number;
+    /** How many times a block must repeat to count as a cycle. Default 3. */
+    minRepeats?: number;
+    /**
+     * Longest cycle to look for, in turns. Default 4.
+     *
+     * Capped by `window / minRepeats` regardless, since a block cannot repeat
+     * three times inside a window that does not hold it three times.
+     */
+    maxPeriod?: number;
+    /**
+     * Cycle-coverage threshold. Set null to disable. Default 0.4.
+     *
+     * Measured on this repo's agent corpus: every healthy trace scores **0.000**,
+     * including the traps built to look like loops, and the weakest degenerate
+     * trace scores 0.455. The default sits in that gap, nearer the healthy side
+     * because nothing healthy approaches it.
+     */
+    maxAgentLoop?: number | null;
+    /**
+     * Tools whose calls are dropped before fingerprinting. Empty by default.
+     *
+     * The escape hatch for a tool whose whole job is to be called repeatedly with
+     * identical arguments -- polling a job, sleeping, reading a clock. By shape
+     * those are indistinguishable from a loop, so naming them is the only honest
+     * way to separate them, in the same way `PROMPT_ECHO` cannot be pointed at a
+     * translate endpoint.
+     *
+     * A turn whose calls are *all* ignored drops out of the trace entirely rather
+     * than falling back to its preamble text.
+     */
+    ignoreTools?: readonly string[];
+}
+
+export type { AgentTurn as A, AgentCheckOptions as a, AgentToolCall as b };

package/dist/agent-types-D310_xck.d.ts

@@ -0,0 +1,97 @@
+/**
+ * The shapes `./agent` reads.
+ *
+ * Structurally typed and provider-neutral, for the same reason the adapters
+ * are: a trace is assembled by the caller from whatever their framework hands
+ * back, and requiring a provider's own type here would make this subpath
+ * depend on a provider.
+ */
+/** One tool call in a turn. */
+interface AgentToolCall {
+    /**
+     * The tool's name. OpenAI spells it `function.name`, Anthropic and the AI SDK
+     * spell it `name`, and the caller maps whichever they have.
+     *
+     * Optional because a streamed call can arrive before its name does. A call
+     * with no name still fingerprints -- by its arguments -- rather than being
+     * dropped, since two nameless calls with identical arguments are still the
+     * same call twice.
+     */
+    name?: string | null;
+    /**
+     * The arguments, in any shape a provider sends: a parsed object, or the raw
+     * JSON string OpenAI uses. Both canonicalise to the same fingerprint, so a
+     * trace assembled from mixed sources still compares.
+     */
+    arguments?: unknown;
+}
+/** One model response in an agent run. */
+interface AgentTurn {
+    /** The prose the model produced, if any. */
+    text?: string | null;
+    /**
+     * The tool calls the model issued, if any.
+     *
+     * **A turn carrying tool calls is judged by them alone**, and its `text` is
+     * read as a preamble and ignored -- the same rule the single-response
+     * adapters already apply. See `internal/turn-fingerprint.ts` for the trap
+     * that rule exists to avoid.
+     */
+    toolCalls?: readonly AgentToolCall[] | null;
+}
+interface AgentCheckOptions {
+    /**
+     * How many trailing turns to inspect. Default 12.
+     *
+     * The same idea as `TAIL_LOOP`'s tail: what matters is whether the agent is
+     * stuck *now*, not whether it repeated itself twenty turns ago and recovered.
+     * A trace shorter than the window is measured whole.
+     *
+     * It also bounds the longest cycle that can be found, at `window / minRepeats`
+     * turns -- 4 at the defaults. Raise both to catch longer orbits, and expect
+     * a loop at the very end of a long trace to score lower, because the score is
+     * coverage of the window rather than a count.
+     */
+    window?: number;
+    /**
+     * Turns required before this will judge at all. Default 4.
+     *
+     * Below it the detector abstains, which is the rule everywhere else in this
+     * package. Two identical turns is a retry, and three is a short poll; neither
+     * is evidence that an agent has stopped advancing.
+     */
+    minTurns?: number;
+    /** How many times a block must repeat to count as a cycle. Default 3. */
+    minRepeats?: number;
+    /**
+     * Longest cycle to look for, in turns. Default 4.
+     *
+     * Capped by `window / minRepeats` regardless, since a block cannot repeat
+     * three times inside a window that does not hold it three times.
+     */
+    maxPeriod?: number;
+    /**
+     * Cycle-coverage threshold. Set null to disable. Default 0.4.
+     *
+     * Measured on this repo's agent corpus: every healthy trace scores **0.000**,
+     * including the traps built to look like loops, and the weakest degenerate
+     * trace scores 0.455. The default sits in that gap, nearer the healthy side
+     * because nothing healthy approaches it.
+     */
+    maxAgentLoop?: number | null;
+    /**
+     * Tools whose calls are dropped before fingerprinting. Empty by default.
+     *
+     * The escape hatch for a tool whose whole job is to be called repeatedly with
+     * identical arguments -- polling a job, sleeping, reading a clock. By shape
+     * those are indistinguishable from a loop, so naming them is the only honest
+     * way to separate them, in the same way `PROMPT_ECHO` cannot be pointed at a
+     * translate endpoint.
+     *
+     * A turn whose calls are *all* ignored drops out of the trace entirely rather
+     * than falling back to its preamble text.
+     */
+    ignoreTools?: readonly string[];
+}
+
+export type { AgentTurn as A, AgentCheckOptions as a, AgentToolCall as b };

package/dist/agent.d.cts

@@ -1,102 +1,8 @@
 import { V as Verdict } from './types-CxKV_wpA.cjs';
+import { A as AgentTurn, a as AgentCheckOptions } from './agent-types-D310_xck.cjs';
+export { b as AgentToolCall } from './agent-types-D310_xck.cjs';
 export { D as DegenerateOutputError } from './check-B7lLF8a8.cjs';
 
-/**
- * The shapes `./agent` reads.
- *
- * Structurally typed and provider-neutral, for the same reason the adapters
- * are: a trace is assembled by the caller from whatever their framework hands
- * back, and requiring a provider's own type here would make this subpath
- * depend on a provider.
- */
-/** One tool call in a turn. */
-interface AgentToolCall {
-    /**
-     * The tool's name. OpenAI spells it `function.name`, Anthropic and the AI SDK
-     * spell it `name`, and the caller maps whichever they have.
-     *
-     * Optional because a streamed call can arrive before its name does. A call
-     * with no name still fingerprints -- by its arguments -- rather than being
-     * dropped, since two nameless calls with identical arguments are still the
-     * same call twice.
-     */
-    name?: string | null;
-    /**
-     * The arguments, in any shape a provider sends: a parsed object, or the raw
-     * JSON string OpenAI uses. Both canonicalise to the same fingerprint, so a
-     * trace assembled from mixed sources still compares.
-     */
-    arguments?: unknown;
-}
-/** One model response in an agent run. */
-interface AgentTurn {
-    /** The prose the model produced, if any. */
-    text?: string | null;
-    /**
-     * The tool calls the model issued, if any.
-     *
-     * **A turn carrying tool calls is judged by them alone**, and its `text` is
-     * read as a preamble and ignored -- the same rule the single-response
-     * adapters already apply. See `internal/turn-fingerprint.ts` for the trap
-     * that rule exists to avoid.
-     */
-    toolCalls?: readonly AgentToolCall[] | null;
-}
-interface AgentCheckOptions {
-    /**
-     * How many trailing turns to inspect. Default 12.
-     *
-     * The same idea as `TAIL_LOOP`'s tail: what matters is whether the agent is
-     * stuck *now*, not whether it repeated itself twenty turns ago and recovered.
-     * A trace shorter than the window is measured whole.
-     *
-     * It also bounds the longest cycle that can be found, at `window / minRepeats`
-     * turns -- 4 at the defaults. Raise both to catch longer orbits, and expect
-     * a loop at the very end of a long trace to score lower, because the score is
-     * coverage of the window rather than a count.
-     */
-    window?: number;
-    /**
-     * Turns required before this will judge at all. Default 4.
-     *
-     * Below it the detector abstains, which is the rule everywhere else in this
-     * package. Two identical turns is a retry, and three is a short poll; neither
-     * is evidence that an agent has stopped advancing.
-     */
-    minTurns?: number;
-    /** How many times a block must repeat to count as a cycle. Default 3. */
-    minRepeats?: number;
-    /**
-     * Longest cycle to look for, in turns. Default 4.
-     *
-     * Capped by `window / minRepeats` regardless, since a block cannot repeat
-     * three times inside a window that does not hold it three times.
-     */
-    maxPeriod?: number;
-    /**
-     * Cycle-coverage threshold. Set null to disable. Default 0.4.
-     *
-     * Measured on this repo's agent corpus: every healthy trace scores **0.000**,
-     * including the traps built to look like loops, and the weakest degenerate
-     * trace scores 0.455. The default sits in that gap, nearer the healthy side
-     * because nothing healthy approaches it.
-     */
-    maxAgentLoop?: number | null;
-    /**
-     * Tools whose calls are dropped before fingerprinting. Empty by default.
-     *
-     * The escape hatch for a tool whose whole job is to be called repeatedly with
-     * identical arguments -- polling a job, sleeping, reading a clock. By shape
-     * those are indistinguishable from a loop, so naming them is the only honest
-     * way to separate them, in the same way `PROMPT_ECHO` cannot be pointed at a
-     * translate endpoint.
-     *
-     * A turn whose calls are *all* ignored drops out of the trace entirely rather
-     * than falling back to its preamble text.
-     */
-    ignoreTools?: readonly string[];
-}
-
 /**
  * The failure that every other detector in this package scores 0.000 on.
  *
@@ -238,4 +144,4 @@ interface AgentGuard {
  */
 declare function createAgentGuard(options?: AgentCheckOptions): AgentGuard;
 
-export { type AgentCheckOptions, type AgentGuard, type AgentLoopResult, type AgentToolCall, type AgentTurn, agentLoopDetail, agentLoopScore, assertTrace, checkTrace, createAgentGuard };
+export { AgentCheckOptions, type AgentGuard, type AgentLoopResult, AgentTurn, agentLoopDetail, agentLoopScore, assertTrace, checkTrace, createAgentGuard };

package/dist/agent.d.ts

@@ -1,102 +1,8 @@
 import { V as Verdict } from './types-CxKV_wpA.js';
+import { A as AgentTurn, a as AgentCheckOptions } from './agent-types-D310_xck.js';
+export { b as AgentToolCall } from './agent-types-D310_xck.js';
 export { D as DegenerateOutputError } from './check-DiKuVvZM.js';
 
-/**
- * The shapes `./agent` reads.
- *
- * Structurally typed and provider-neutral, for the same reason the adapters
- * are: a trace is assembled by the caller from whatever their framework hands
- * back, and requiring a provider's own type here would make this subpath
- * depend on a provider.
- */
-/** One tool call in a turn. */
-interface AgentToolCall {
-    /**
-     * The tool's name. OpenAI spells it `function.name`, Anthropic and the AI SDK
-     * spell it `name`, and the caller maps whichever they have.
-     *
-     * Optional because a streamed call can arrive before its name does. A call
-     * with no name still fingerprints -- by its arguments -- rather than being
-     * dropped, since two nameless calls with identical arguments are still the
-     * same call twice.
-     */
-    name?: string | null;
-    /**
-     * The arguments, in any shape a provider sends: a parsed object, or the raw
-     * JSON string OpenAI uses. Both canonicalise to the same fingerprint, so a
-     * trace assembled from mixed sources still compares.
-     */
-    arguments?: unknown;
-}
-/** One model response in an agent run. */
-interface AgentTurn {
-    /** The prose the model produced, if any. */
-    text?: string | null;
-    /**
-     * The tool calls the model issued, if any.
-     *
-     * **A turn carrying tool calls is judged by them alone**, and its `text` is
-     * read as a preamble and ignored -- the same rule the single-response
-     * adapters already apply. See `internal/turn-fingerprint.ts` for the trap
-     * that rule exists to avoid.
-     */
-    toolCalls?: readonly AgentToolCall[] | null;
-}
-interface AgentCheckOptions {
-    /**
-     * How many trailing turns to inspect. Default 12.
-     *
-     * The same idea as `TAIL_LOOP`'s tail: what matters is whether the agent is
-     * stuck *now*, not whether it repeated itself twenty turns ago and recovered.
-     * A trace shorter than the window is measured whole.
-     *
-     * It also bounds the longest cycle that can be found, at `window / minRepeats`
-     * turns -- 4 at the defaults. Raise both to catch longer orbits, and expect
-     * a loop at the very end of a long trace to score lower, because the score is
-     * coverage of the window rather than a count.
-     */
-    window?: number;
-    /**
-     * Turns required before this will judge at all. Default 4.
-     *
-     * Below it the detector abstains, which is the rule everywhere else in this
-     * package. Two identical turns is a retry, and three is a short poll; neither
-     * is evidence that an agent has stopped advancing.
-     */
-    minTurns?: number;
-    /** How many times a block must repeat to count as a cycle. Default 3. */
-    minRepeats?: number;
-    /**
-     * Longest cycle to look for, in turns. Default 4.
-     *
-     * Capped by `window / minRepeats` regardless, since a block cannot repeat
-     * three times inside a window that does not hold it three times.
-     */
-    maxPeriod?: number;
-    /**
-     * Cycle-coverage threshold. Set null to disable. Default 0.4.
-     *
-     * Measured on this repo's agent corpus: every healthy trace scores **0.000**,
-     * including the traps built to look like loops, and the weakest degenerate
-     * trace scores 0.455. The default sits in that gap, nearer the healthy side
-     * because nothing healthy approaches it.
-     */
-    maxAgentLoop?: number | null;
-    /**
-     * Tools whose calls are dropped before fingerprinting. Empty by default.
-     *
-     * The escape hatch for a tool whose whole job is to be called repeatedly with
-     * identical arguments -- polling a job, sleeping, reading a clock. By shape
-     * those are indistinguishable from a loop, so naming them is the only honest
-     * way to separate them, in the same way `PROMPT_ECHO` cannot be pointed at a
-     * translate endpoint.
-     *
-     * A turn whose calls are *all* ignored drops out of the trace entirely rather
-     * than falling back to its preamble text.
-     */
-    ignoreTools?: readonly string[];
-}
-
 /**
  * The failure that every other detector in this package scores 0.000 on.
  *
@@ -238,4 +144,4 @@ interface AgentGuard {
  */
 declare function createAgentGuard(options?: AgentCheckOptions): AgentGuard;
 
-export { type AgentCheckOptions, type AgentGuard, type AgentLoopResult, type AgentToolCall, type AgentTurn, agentLoopDetail, agentLoopScore, assertTrace, checkTrace, createAgentGuard };
+export { AgentCheckOptions, type AgentGuard, type AgentLoopResult, AgentTurn, agentLoopDetail, agentLoopScore, assertTrace, checkTrace, createAgentGuard };

package/dist/ai-sdk.cjs

@@ -748,6 +748,11 @@ function promptFromMessages(messages) {
   return parts.join("\n\n");
 }
 
+// src/internal/as-array.ts
+function asArray(value) {
+  return Array.isArray(value) ? value : [];
+}
+
 // src/ai-sdk.ts
 var isToolPart = (part) => part.type.startsWith("tool-");
 function finishReasonOf(value) {
@@ -863,7 +868,28 @@ function outputGuard(options = {}) {
     }
   };
 }
+function toTurn(result) {
+  if (result == null || typeof result !== "object") return {};
+  const value = result;
+  if (Array.isArray(value.content)) {
+    return {
+      text: value.content.filter((part) => part.type === "text").map((part) => part.text ?? "").join(""),
+      toolCalls: value.content.filter(isToolPart).map((part) => ({
+        name: part.toolName,
+        arguments: part.input ?? part.args
+      }))
+    };
+  }
+  return {
+    text: typeof value.text === "string" ? value.text : "",
+    toolCalls: asArray(value.toolCalls).map((call) => ({
+      name: call.toolName ?? call.name,
+      arguments: call.input ?? call.args
+    }))
+  };
+}
 
 exports.outputGuard = outputGuard;
+exports.toTurn = toTurn;
 //# sourceMappingURL=ai-sdk.cjs.map
 //# sourceMappingURL=ai-sdk.cjs.map