llm-output-guard 1.4.1 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +123 -141
- package/dist/{adapter-options-CQNPNjES.d.cts → adapter-options-B-fgYd--.d.cts} +35 -1
- package/dist/{adapter-options-BUOzVp3T.d.ts → adapter-options-DNkMNS92.d.ts} +35 -1
- package/dist/ai-sdk.cjs +213 -10
- package/dist/ai-sdk.cjs.map +1 -1
- package/dist/ai-sdk.d.cts +22 -5
- package/dist/ai-sdk.d.ts +22 -5
- package/dist/ai-sdk.js +28 -9
- package/dist/ai-sdk.js.map +1 -1
- package/dist/anthropic.cjs +232 -12
- package/dist/anthropic.cjs.map +1 -1
- package/dist/anthropic.d.cts +3 -3
- package/dist/anthropic.d.ts +3 -3
- package/dist/anthropic.js +17 -3
- package/dist/anthropic.js.map +1 -1
- package/dist/bin.cjs +684 -2
- package/dist/bin.cjs.map +1 -1
- package/dist/bin.js +684 -2
- package/dist/bin.js.map +1 -1
- package/dist/{chunk-R6QMUCVR.js → chunk-D3HQLFND.js} +112 -6
- package/dist/chunk-D3HQLFND.js.map +1 -0
- package/dist/{chunk-NHVIUA4Z.js → chunk-FCD42SUY.js} +28 -13
- package/dist/chunk-FCD42SUY.js.map +1 -0
- package/dist/chunk-WVML434P.js +105 -0
- package/dist/chunk-WVML434P.js.map +1 -0
- package/dist/index.cjs +111 -3
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +81 -3
- package/dist/index.d.ts +81 -3
- package/dist/index.js +1 -1
- package/dist/openai.cjs +241 -13
- package/dist/openai.cjs.map +1 -1
- package/dist/openai.d.cts +3 -3
- package/dist/openai.d.ts +3 -3
- package/dist/openai.js +26 -4
- package/dist/openai.js.map +1 -1
- package/dist/{stream-NO-TTGYs.d.cts → stream-BYy6vuGy.d.cts} +64 -3
- package/dist/{stream-NO-TTGYs.d.ts → stream-BYy6vuGy.d.ts} +64 -3
- package/package.json +3 -2
- package/dist/chunk-NHVIUA4Z.js.map +0 -1
- package/dist/chunk-OHF5SN7W.js +0 -17
- package/dist/chunk-OHF5SN7W.js.map +0 -1
- package/dist/chunk-R6QMUCVR.js.map +0 -1
package/README.md
CHANGED
|
@@ -7,12 +7,26 @@
|
|
|
7
7
|
|
|
8
8
|
**Detect LLM responses that failed while returning `200 OK`.**
|
|
9
9
|
|
|
10
|
+
<!--
|
|
11
|
+
Two rows on purpose: what you are installing, then whether it is looked after.
|
|
12
|
+
|
|
13
|
+
Every GitHub-backed badge carries an explicit cacheSeconds. Shields defaults
|
|
14
|
+
these to max-age=120, which has GitHub's camo proxy refetch 720 times a day
|
|
15
|
+
and gives 720 daily chances to catch the upstream API rate limited and cache
|
|
16
|
+
the error for the whole TTL. That is not hypothetical: it happened to the
|
|
17
|
+
downloads badge, which is why it carries one too.
|
|
18
|
+
-->
|
|
10
19
|
[](https://www.npmjs.com/package/llm-output-guard)
|
|
11
20
|
[](https://www.npmjs.com/package/llm-output-guard)
|
|
12
21
|
[](https://bundlejs.com/?q=llm-output-guard)
|
|
13
22
|
[](https://github.com/edwinsatya/llm-output-guard/blob/main/package.json)
|
|
23
|
+
[](https://www.npmjs.com/package/llm-output-guard)
|
|
24
|
+
[](https://nodejs.org)
|
|
25
|
+
|
|
14
26
|
[](https://github.com/edwinsatya/llm-output-guard/actions/workflows/ci.yml)
|
|
15
|
-
[](https://github.com/edwinsatya/llm-output-guard/commits/main)
|
|
28
|
+
[](https://github.com/edwinsatya/llm-output-guard/graphs/commit-activity)
|
|
29
|
+
[](./LICENSE)
|
|
16
30
|
|
|
17
31
|
Your retry layer watches for `429`, `5xx` and timeouts. It cannot see a model that
|
|
18
32
|
looped until `max_tokens`, returned `{}`, stopped mid-sentence, or answered in the
|
|
@@ -56,12 +70,16 @@ every detector, running on your own pasted output. No API key, no request.
|
|
|
56
70
|
| `LOW_ENTROPY` | Character-level collapse, token artifacts | Compression ratio |
|
|
57
71
|
| `TRUNCATED` | Cut off mid-thought | `finish_reason`, unbalanced fences |
|
|
58
72
|
| `INVALID_JSON` | Prose around the payload, wrong types | Parse + key + schema contract |
|
|
59
|
-
| `SCRIPT_MISMATCH` | Answered in the wrong alphabet | Share of letters outside
|
|
60
|
-
| `LANG_MISMATCH` |
|
|
73
|
+
| `SCRIPT_MISMATCH` | Answered in the wrong alphabet | Share of letters outside expected scripts · opt-in |
|
|
74
|
+
| `LANG_MISMATCH` | Wrong language, same alphabet | Function-word profile · opt-in |
|
|
75
|
+
| `PROMPT_ECHO` | Returned your prompt instead of an answer | Share of output copied from the prompt · opt-in |
|
|
61
76
|
|
|
62
77
|
Every detector runs even after one fails, so a verdict shows the whole picture
|
|
63
78
|
rather than whichever check happened to be ordered first. Each returns **0–1, not
|
|
64
|
-
a boolean** — you pick the line.
|
|
79
|
+
a boolean** — you pick the line.
|
|
80
|
+
|
|
81
|
+
**Full reference, with the measurements behind every default:
|
|
82
|
+
[docs/detectors.md](docs/detectors.md)**
|
|
65
83
|
|
|
66
84
|
## Guard your provider in one wrap
|
|
67
85
|
|
|
@@ -77,8 +95,12 @@ Adapters for the **OpenAI SDK** (both `chat.completions` and `responses`),
|
|
|
77
95
|
protocol: Groq, Together, OpenRouter, Fireworks, vLLM, Ollama.
|
|
78
96
|
|
|
79
97
|
On a stream this **cancels the HTTP request** the moment a loop is detectable, so
|
|
80
|
-
you stop paying for the rest of it.
|
|
81
|
-
and
|
|
98
|
+
you stop paying for the rest of it. Two switches worth knowing:
|
|
99
|
+
`checkToolArguments: true` measures the arguments a model passes to a tool, and
|
|
100
|
+
`checkPromptEcho: true` reads the prompt out of each request so a model that
|
|
101
|
+
replays it instead of answering is caught.
|
|
102
|
+
|
|
103
|
+
**[docs/adapters.md](docs/adapters.md)** · **[docs/streaming.md](docs/streaming.md)**
|
|
82
104
|
|
|
83
105
|
## The hard part is not catching loops
|
|
84
106
|
|
|
@@ -86,89 +108,60 @@ A miss is annoying. **A false positive is worse** — a healthy response gets
|
|
|
86
108
|
discarded and retried against a slower provider for nothing.
|
|
87
109
|
|
|
88
110
|
So the corpus carries deliberate traps: markdown tables, repeated-prefix lists,
|
|
89
|
-
code blocks, rhetorical refrains, a Chinese poem refrain
|
|
90
|
-
|
|
111
|
+
code blocks, rhetorical refrains, a Chinese poem refrain, a Chinese answer
|
|
112
|
+
wrapped around a TypeScript block. All of them look degenerate to a naive
|
|
113
|
+
detector, and all of them are fine. Paste one into the
|
|
91
114
|
[playground](https://edwinsatya.github.io/llm-output-guard/) and watch it pass.
|
|
92
115
|
|
|
93
116
|
---
|
|
94
117
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
```ts
|
|
98
|
-
const verdict = checkOutput(raw, {
|
|
99
|
-
...presets.strictJson,
|
|
100
|
-
requiredKeys: ['score', 'notes', 'followUp'],
|
|
101
|
-
});
|
|
102
|
-
|
|
103
|
-
if (verdict.ok) use(verdict.json); // already parsed, fence stripped
|
|
104
|
-
```
|
|
118
|
+
## Common setups
|
|
105
119
|
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
breaks everything downstream that does arithmetic. Pass a **schema** to check
|
|
109
|
-
the shape rather than the spelling:
|
|
120
|
+
**Structured output** — parse, check keys, and validate against a schema you
|
|
121
|
+
already have:
|
|
110
122
|
|
|
111
123
|
```ts
|
|
112
124
|
import { z } from 'zod';
|
|
113
125
|
|
|
114
|
-
const
|
|
115
|
-
|
|
116
|
-
notes: z.string(),
|
|
117
|
-
followUp: z.array(z.string()),
|
|
126
|
+
const verdict = checkOutput(raw, {
|
|
127
|
+
...presets.strictJson,
|
|
128
|
+
schema: z.object({ score: z.number(), notes: z.string() }),
|
|
118
129
|
});
|
|
119
130
|
|
|
120
|
-
const verdict = checkOutput(raw, { ...presets.strictJson, schema: Review });
|
|
121
|
-
|
|
122
131
|
if (verdict.ok) use(verdict.json); // parsed, validated, defaults applied
|
|
123
132
|
```
|
|
124
133
|
|
|
125
|
-
Any [Standard Schema](https://standardschema.dev) validator works —
|
|
126
|
-
Valibot, ArkType
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
On success `verdict.json` is the schema's *output*, so defaults, coercions and
|
|
131
|
-
transforms are applied and the value matches the type you declared. On failure
|
|
132
|
-
you get `INVALID_JSON` with the failing path in the message —
|
|
133
|
-
`score: Expected number, received string`. It is the same reason code as a
|
|
134
|
-
missing key or an unparseable payload because it wants the same handling: retry,
|
|
135
|
-
or fall through to another provider.
|
|
136
|
-
|
|
137
|
-
The two compose, and keys are checked first, so a missing key is still reported
|
|
138
|
-
as a missing key rather than as whatever the schema calls it.
|
|
139
|
-
|
|
140
|
-
> **The schema must validate synchronously.** `checkOutput` is synchronous by
|
|
141
|
-
> design — that is what makes it safe on a hot path — so a schema carrying an
|
|
142
|
-
> async refinement throws a `TypeError` telling you so, rather than silently
|
|
143
|
-
> passing. Everything Zod, Valibot and ArkType produce otherwise is synchronous.
|
|
144
|
-
> This is the one thing in the package that throws about your configuration; it
|
|
145
|
-
> still never throws about a response.
|
|
134
|
+
Any [Standard Schema](https://standardschema.dev) validator works — Zod 4,
|
|
135
|
+
Valibot, ArkType — and the spec is types-only, so this still costs **no
|
|
136
|
+
dependency**. Details, and the one case that throws:
|
|
137
|
+
[docs/detectors.md](docs/detectors.md#structured-output)
|
|
146
138
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
`redundancyScope: 'jsonValues'` is the fix.
|
|
150
|
-
|
|
151
|
-
### Answering in the wrong language
|
|
139
|
+
**Answered in the wrong language** — a model that ignores "answer in English"
|
|
140
|
+
returns fluent Chinese, not broken English:
|
|
152
141
|
|
|
153
142
|
```ts
|
|
154
143
|
checkOutput(raw, { ...presets.chat, expectScript: 'latin' });
|
|
155
144
|
```
|
|
156
145
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
146
|
+
Detectable by counting characters — no word list, decisive from about a dozen
|
|
147
|
+
letters. A wrong-script answer scores **1.000**; a healthy one measured against
|
|
148
|
+
its own script scores **0.000–0.028**. Pass every script the answer may
|
|
149
|
+
legitimately contain (`['han', 'latin']` for Chinese). Code fences and URLs are
|
|
150
|
+
excluded, so a TypeScript block never counts as answering in English.
|
|
151
|
+
[More →](docs/detectors.md#answering-in-the-wrong-language)
|
|
152
|
+
|
|
153
|
+
**Returned your prompt instead of an answer** — invisible to every other
|
|
154
|
+
detector here, because such a response is fluent, well-formed and the right
|
|
155
|
+
length:
|
|
163
156
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
157
|
+
```ts
|
|
158
|
+
checkOutput(raw, { ...presets.chat, prompt });
|
|
159
|
+
// or, from an adapter, which reads the prompt out of the request itself:
|
|
160
|
+
withOutputGuard(new OpenAI(), { ...presets.chat, checkPromptEcho: true });
|
|
161
|
+
```
|
|
169
162
|
|
|
170
|
-
|
|
171
|
-
|
|
163
|
+
Not for rewrite, translate or summarise endpoints, where copying the input is
|
|
164
|
+
the job. [More →](docs/detectors.md#returning-the-prompt-instead-of-an-answer)
|
|
172
165
|
|
|
173
166
|
## The verdict
|
|
174
167
|
|
|
@@ -185,71 +178,63 @@ Same script means no signal: Spanish against English scores 0. That is what
|
|
|
185
178
|
}
|
|
186
179
|
```
|
|
187
180
|
|
|
188
|
-
|
|
181
|
+
`reasons` carries every failing signal, not just the first. `scores` includes the
|
|
182
|
+
passing detectors too — send them to your metrics and you will know your real
|
|
183
|
+
degeneration rate within a day.
|
|
189
184
|
|
|
190
|
-
`modes` says which tokenizer produced a score
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
gives you a number that describes neither.
|
|
185
|
+
`modes` says which tokenizer produced a score. **Log it next to `scores`:**
|
|
186
|
+
`TAIL_LOOP` measures words on spaced scripts and characters on Chinese, Japanese
|
|
187
|
+
and Thai, and pooling two distributions into one histogram gives you a number
|
|
188
|
+
that describes neither.
|
|
195
189
|
|
|
196
190
|
## Presets
|
|
197
191
|
|
|
198
192
|
`chat` · `strictJson` · `longForm` · `lenient`
|
|
199
193
|
|
|
200
|
-
|
|
194
|
+
Starting points calibrated against this repo's fixture corpus — not universal
|
|
195
|
+
truths.
|
|
201
196
|
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
## Calibrating against your own traffic
|
|
197
|
+
## Calibrate against your own traffic
|
|
205
198
|
|
|
206
|
-
The shipped presets are tuned on
|
|
207
|
-
|
|
199
|
+
The shipped presets are tuned on a corpus that is **not your traffic**. Score the
|
|
200
|
+
responses you already have, then derive thresholds you can defend:
|
|
208
201
|
|
|
209
202
|
```bash
|
|
210
|
-
npx llm-output-guard
|
|
203
|
+
npx llm-output-guard check logs/*.txt --json | npx llm-output-guard calibrate --fpr 0.001
|
|
211
204
|
```
|
|
212
205
|
|
|
206
|
+
`check` also works as a CI assertion — it exits 1 when anything is degenerate,
|
|
207
|
+
2 when the input cannot be read.
|
|
208
|
+
|
|
213
209
|
The report tells you when your sample is too small to support the rate you asked
|
|
214
210
|
for, and distinguishes real separation in your data from a false-positive budget —
|
|
215
|
-
because only one of those is evidence.
|
|
216
|
-
**[docs/calibration.md](docs/calibration.md)
|
|
217
|
-
|
|
218
|
-
## Script coverage
|
|
219
|
-
|
|
220
|
-
Korean, Cyrillic, Greek, Arabic and Devanagari separate words and are handled like
|
|
221
|
-
English. **Chinese, Japanese and Thai do not**, so `TAIL_LOOP` switches to
|
|
222
|
-
character mode and reads its own threshold. `REPETITION` is blind on those scripts —
|
|
223
|
-
a known, measured gap. `SCRIPT_MISMATCH` covers all ten scripts and is the one
|
|
224
|
-
detector these are *not* the weak case for. Numbers behind both in
|
|
225
|
-
**[docs/script-coverage.md](docs/script-coverage.md)**.
|
|
211
|
+
because only one of those is evidence.
|
|
212
|
+
**[docs/calibration.md](docs/calibration.md)**
|
|
226
213
|
|
|
227
214
|
## Design notes
|
|
228
215
|
|
|
229
|
-
- **Zero runtime dependencies**, enforced in CI. Node ≥ 18
|
|
230
|
-
- **Hand-rolled LZ77** rather than `node:zlib`, so the package stays runtime-agnostic. It is not a real compressor; it only needs to move monotonically with redundancy.
|
|
216
|
+
- **Zero runtime dependencies**, enforced in CI. Node ≥ 18; works on edge, browser, Deno, Bun.
|
|
231
217
|
- **Pure and synchronous.** No network, no clock, no randomness — safe on a hot path, trivial to test.
|
|
232
218
|
- **Scores, not booleans.** Detectors report 0–1 and leave the threshold decision to you.
|
|
233
219
|
- **Abstains rather than guesses.** Samples too short to judge score 0.
|
|
220
|
+
- **Hand-rolled LZ77** rather than `node:zlib`, so the package stays runtime-agnostic.
|
|
221
|
+
- **Sub-millisecond**, 0.383 ms at 500 B and 1.014 ms at 32 KB, with one detector accounting for most of it. `npm run bench` reproduces it — **[docs/performance.md](docs/performance.md)**
|
|
222
|
+
- **Chinese, Japanese and Thai** are handled where they differ: `TAIL_LOOP` switches to character mode, `REPETITION` is blind and says so — **[docs/script-coverage.md](docs/script-coverage.md)**
|
|
234
223
|
|
|
235
224
|
## Stability
|
|
236
225
|
|
|
237
|
-
What semver means
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
`
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
behaviour they carry. The list is asserted in `test/surface.test.ts`, so an
|
|
250
|
-
export cannot join it by accident.
|
|
251
|
-
|
|
252
|
-
**Threshold and preset values are behaviour, not implementation.** This is the
|
|
226
|
+
What semver means here specifically. These rules bind from **1.0.0** onward, and
|
|
227
|
+
the public surface was frozen export by export in that release.
|
|
228
|
+
|
|
229
|
+
**The public API is** everything exported from `llm-output-guard`, plus
|
|
230
|
+
`outputGuard` / `OutputGuardOptions` / `DegenerateAction` from `./ai-sdk` and
|
|
231
|
+
`withOutputGuard` / `OutputGuardOptions` / `DegenerateAction` from `./openai` and
|
|
232
|
+
`./anthropic`. Each subpath is its own contract, so an option added to one is not
|
|
233
|
+
a promise about the others. Anything else is internal and may move in any
|
|
234
|
+
release. The list is asserted in `test/surface.test.ts`, so an export cannot join
|
|
235
|
+
it by accident.
|
|
236
|
+
|
|
237
|
+
**Threshold and preset values are behaviour, not implementation.** That is the
|
|
253
238
|
interesting case, so it gets a rule of its own:
|
|
254
239
|
|
|
255
240
|
| Change | Release type |
|
|
@@ -259,42 +244,39 @@ interesting case, so it gets a rule of its own:
|
|
|
259
244
|
| Adding a new *option*, defaulted so nothing changes | minor |
|
|
260
245
|
| Adding a new detector that is opt-in | minor |
|
|
261
246
|
| Making an existing detector strictly more accurate on its own axis | minor |
|
|
262
|
-
| Docs, internals, performance, fixing a detector that
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
**
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
version range upgrades into automatically. It was withdrawn from npm within the
|
|
282
|
-
72-hour unpublish window and re-released as 0.5.0, where `^0.4.1` correctly
|
|
283
|
-
resolves away from it. The rule it broke is the one in the table above: threshold
|
|
284
|
-
and preset changes are behaviour changes, and behaviour changes are never
|
|
285
|
-
patches.
|
|
247
|
+
| Docs, internals, performance, fixing a detector that returned a wrong score | patch |
|
|
248
|
+
|
|
249
|
+
A threshold change does not break your build. It changes which of your production
|
|
250
|
+
responses get discarded and retried, which is a larger event than a signature
|
|
251
|
+
change and invisible until your traffic hits it. A number in `presets.chat` is
|
|
252
|
+
part of the contract in the same way a function name is.
|
|
253
|
+
|
|
254
|
+
**Not covered:** the exact scores a detector returns (only their direction and
|
|
255
|
+
the thresholds acting on them), the fixture corpus, `message` strings in
|
|
256
|
+
`Reason`, or the human-readable `calibrate` report. `--json` output *is* covered.
|
|
257
|
+
|
|
258
|
+
**Peer ranges** narrow only in a major, and are verified rather than assumed —
|
|
259
|
+
`npm run check:peers` installs the packed tarball against each end of each
|
|
260
|
+
declared range, then typechecks and runs the adapter.
|
|
261
|
+
|
|
262
|
+
**Why this is written down:** 0.4.2 shipped a peer narrowing, a new subpath, and
|
|
263
|
+
a behaviour change under a **patch** number, which every default version range
|
|
264
|
+
upgrades into automatically. It was unpublished within the 72-hour window and
|
|
265
|
+
re-released as 0.5.0. The rule it broke is the one in the table above.
|
|
286
266
|
|
|
287
267
|
## Limitations
|
|
288
268
|
|
|
289
269
|
- Not a hallucination detector. It measures *shape*, never truth.
|
|
290
|
-
-
|
|
291
|
-
- `
|
|
292
|
-
- `
|
|
293
|
-
-
|
|
294
|
-
- `SCRIPT_MISMATCH`
|
|
295
|
-
-
|
|
296
|
-
-
|
|
297
|
-
-
|
|
270
|
+
- `REPETITION` does not work on Chinese, Japanese or Thai — a known, measured gap, not an oversight.
|
|
271
|
+
- `LANG_MISMATCH` is a function-word heuristic covering `id`/`en`/`es`/`pt`/`it`/`fr`/`de`/`nl`, and is unreliable under 25 words. `expectScript` is the stronger check wherever the alphabets differ.
|
|
272
|
+
- `expectLang: 'es'` does not reliably catch Portuguese, Italian or French: its profile is built from function words all four share. Measured, and unfixable without a threshold change — see **[docs/detectors.md](docs/detectors.md#answering-in-the-wrong-language)**.
|
|
273
|
+
- `PROMPT_ECHO` cannot tell a degenerate echo from a rewrite or translation — the difference is in what you asked for, not in the text.
|
|
274
|
+
- `SCRIPT_MISMATCH` and `PROMPT_ECHO` do not run mid-stream by default: both measure a property of the whole response, and a mid-stream check reads a trailing window. `earlyDocumentChecks: true` opts in, with a measured false-positive risk — see **[docs/streaming.md](docs/streaming.md)**.
|
|
275
|
+
- Tool *arguments* are measured only with `checkToolArguments: true`, non-streaming responses only.
|
|
276
|
+
- `openai`'s `responses.stream()` helper is not wrapped; `create({ stream: true })` is.
|
|
277
|
+
- Truncation from a missing full stop is weak evidence, scored 0.55 and deliberately left below the defaults. Lower `maxTruncation` to ~0.5 to catch it, and expect false positives.
|
|
278
|
+
- A JSON array of repeated identical records reads as a loop and fails from three records up. Set `redundancyScope: 'jsonValues'`.
|
|
279
|
+
- Thresholds are calibrated on the bundled corpus. Yours will differ — and the word and character thresholds need calibrating **separately**.
|
|
298
280
|
|
|
299
281
|
## License
|
|
300
282
|
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { V as Verdict } from './stream-
|
|
1
|
+
import { V as Verdict } from './stream-BYy6vuGy.cjs';
|
|
2
2
|
|
|
3
3
|
/**
|
|
4
4
|
* The option surface every provider adapter shares.
|
|
@@ -62,6 +62,40 @@ interface AdapterGuardOptions {
|
|
|
62
62
|
onVerdict?: (verdict: Verdict, context: {
|
|
63
63
|
streaming: boolean;
|
|
64
64
|
}) => void;
|
|
65
|
+
/**
|
|
66
|
+
* Also measure the arguments the model passed to a tool. Default `false`.
|
|
67
|
+
*
|
|
68
|
+
* A tool-calling turn is judged by its preamble, because the text beside a
|
|
69
|
+
* tool call is not the answer -- and that leaves the answer itself
|
|
70
|
+
* unmeasured. Your provider validates arguments against the schema you gave
|
|
71
|
+
* it, which covers types and not content: `{ "query": "site reliability site
|
|
72
|
+
* reliability site reliability ..." }` is a schema-valid string, and it is
|
|
73
|
+
* still a garbage query.
|
|
74
|
+
*
|
|
75
|
+
* Switching this on measures each string value of each call's arguments for
|
|
76
|
+
* redundancy, and folds the result into the same verdict the preamble
|
|
77
|
+
* produces. Reason codes are unchanged, so existing handling works; the
|
|
78
|
+
* `message` says the loop was found in an argument.
|
|
79
|
+
*
|
|
80
|
+
* **Non-streaming responses only.** Arguments arrive as JSON fragments that
|
|
81
|
+
* do not parse until the call is complete, so there is nothing meaningful to
|
|
82
|
+
* measure mid-stream.
|
|
83
|
+
*/
|
|
84
|
+
checkToolArguments?: boolean;
|
|
85
|
+
/**
|
|
86
|
+
* Check the response against the prompt that produced it, so a model that
|
|
87
|
+
* replays your prompt instead of answering is caught. Default `false`.
|
|
88
|
+
*
|
|
89
|
+
* `PROMPT_ECHO` needs the prompt, and a guard is configured once when you
|
|
90
|
+
* wrap the client while the prompt changes on every call -- so this is a
|
|
91
|
+
* switch rather than a value. Turn it on and the adapter reads the prompt out
|
|
92
|
+
* of each request it is already forwarding.
|
|
93
|
+
*
|
|
94
|
+
* **Do not enable this on a rewrite, translate, summarise or extract
|
|
95
|
+
* endpoint.** Copying from the input is the job on those, so a correct answer
|
|
96
|
+
* scores high and the detector measures the task rather than a failure.
|
|
97
|
+
*/
|
|
98
|
+
checkPromptEcho?: boolean;
|
|
65
99
|
}
|
|
66
100
|
|
|
67
101
|
export type { AdapterGuardOptions as A, DegenerateAction as D };
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { V as Verdict } from './stream-
|
|
1
|
+
import { V as Verdict } from './stream-BYy6vuGy.js';
|
|
2
2
|
|
|
3
3
|
/**
|
|
4
4
|
* The option surface every provider adapter shares.
|
|
@@ -62,6 +62,40 @@ interface AdapterGuardOptions {
|
|
|
62
62
|
onVerdict?: (verdict: Verdict, context: {
|
|
63
63
|
streaming: boolean;
|
|
64
64
|
}) => void;
|
|
65
|
+
/**
|
|
66
|
+
* Also measure the arguments the model passed to a tool. Default `false`.
|
|
67
|
+
*
|
|
68
|
+
* A tool-calling turn is judged by its preamble, because the text beside a
|
|
69
|
+
* tool call is not the answer -- and that leaves the answer itself
|
|
70
|
+
* unmeasured. Your provider validates arguments against the schema you gave
|
|
71
|
+
* it, which covers types and not content: `{ "query": "site reliability site
|
|
72
|
+
* reliability site reliability ..." }` is a schema-valid string, and it is
|
|
73
|
+
* still a garbage query.
|
|
74
|
+
*
|
|
75
|
+
* Switching this on measures each string value of each call's arguments for
|
|
76
|
+
* redundancy, and folds the result into the same verdict the preamble
|
|
77
|
+
* produces. Reason codes are unchanged, so existing handling works; the
|
|
78
|
+
* `message` says the loop was found in an argument.
|
|
79
|
+
*
|
|
80
|
+
* **Non-streaming responses only.** Arguments arrive as JSON fragments that
|
|
81
|
+
* do not parse until the call is complete, so there is nothing meaningful to
|
|
82
|
+
* measure mid-stream.
|
|
83
|
+
*/
|
|
84
|
+
checkToolArguments?: boolean;
|
|
85
|
+
/**
|
|
86
|
+
* Check the response against the prompt that produced it, so a model that
|
|
87
|
+
* replays your prompt instead of answering is caught. Default `false`.
|
|
88
|
+
*
|
|
89
|
+
* `PROMPT_ECHO` needs the prompt, and a guard is configured once when you
|
|
90
|
+
* wrap the client while the prompt changes on every call -- so this is a
|
|
91
|
+
* switch rather than a value. Turn it on and the adapter reads the prompt out
|
|
92
|
+
* of each request it is already forwarding.
|
|
93
|
+
*
|
|
94
|
+
* **Do not enable this on a rewrite, translate, summarise or extract
|
|
95
|
+
* endpoint.** Copying from the input is the job on those, so a correct answer
|
|
96
|
+
* scores high and the detector measures the task rather than a failure.
|
|
97
|
+
*/
|
|
98
|
+
checkPromptEcho?: boolean;
|
|
65
99
|
}
|
|
66
100
|
|
|
67
101
|
export type { AdapterGuardOptions as A, DegenerateAction as D };
|