veriquote 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CITATION.cff +1 -1
- package/README.md +182 -154
- package/dist/judge/chat-judge.d.ts +7 -0
- package/dist/judge/chat-judge.d.ts.map +1 -1
- package/dist/judge/chat-judge.js +2 -0
- package/dist/judge/chat-judge.js.map +1 -1
- package/package.json +4 -2
- package/src/judge/chat-judge.ts +11 -1
package/CITATION.cff
CHANGED
package/README.md
CHANGED
|
@@ -6,14 +6,18 @@
|
|
|
6
6
|
**Make an LLM quote its sources, then check every quote twice: is it really in
|
|
7
7
|
the source, and does it support the claim?**
|
|
8
8
|
|
|
9
|
-
A `[1]` after a sentence looks like evidence
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
citation that had nothing behind it, and
|
|
13
|
-
that
|
|
14
|
-
quote
|
|
15
|
-
possible, with an LLM only where it has to.
|
|
16
|
-
browser.
|
|
9
|
+
A `[1]` after a sentence looks like evidence, but almost nobody checks it. In
|
|
10
|
+
my tests, a fifth of one capable model's "verbatim" quotes were not in the
|
|
11
|
+
source, another model left two thirds of its cited answers with at least one
|
|
12
|
+
citation that had nothing behind it, and an LLM judge happily confirmed a quote
|
|
13
|
+
that was invented. VeriQuote makes the answering model commit to a verbatim
|
|
14
|
+
quote for every citation, then verifies each one: deterministically where
|
|
15
|
+
possible, with an LLM only where it has to. It is written in TypeScript, has no
|
|
16
|
+
dependencies and runs in Node and in the browser.
|
|
17
|
+
|
|
18
|
+
**Try it in the browser:** [rickinto.place/veriquote](https://rickinto.place/veriquote)
|
|
19
|
+
has eight worked examples, a box for your own text, and the benchmark results as
|
|
20
|
+
interactive charts.
|
|
17
21
|
|
|
18
22
|
```console
|
|
19
23
|
$ curl -s https://raw.githubusercontent.com/rickintoplace/veriquote/main/examples/ozone-answer.md \
|
|
@@ -37,20 +41,19 @@ $ curl -s https://raw.githubusercontent.com/rickintoplace/veriquote/main/example
|
|
|
37
41
|
REVISE — 3 failed citation(s), 1 uncited sentence(s)
|
|
38
42
|
```
|
|
39
43
|
|
|
40
|
-
`c2`
|
|
41
|
-
|
|
42
|
-
|
|
44
|
+
Look at `c2` and `c4`. The quote in `c2` is copied correctly, but the claim
|
|
45
|
+
names the wrong person; only the judge can see that. The quote in `c4` is made
|
|
46
|
+
up, and the judge still calls it supported; only the matcher can see that.
|
|
47
|
+
Without an API key the CLI checks the quotes alone. Set
|
|
43
48
|
`VERIQUOTE_JUDGE_API_KEY` and `VERIQUOTE_JUDGE_MODEL` (any OpenAI-compatible
|
|
44
|
-
endpoint)
|
|
45
|
-
|
|
46
|
-
**Try both checks in the browser:** [`demo/index.html`](demo/index.html) — eight
|
|
47
|
-
examples with recorded judge verdicts, or your own text with your own key.
|
|
49
|
+
endpoint) to add the judge.
|
|
48
50
|
|
|
49
51
|
## How it works
|
|
50
52
|
|
|
51
|
-
The answering model gets `veriquote prompt` (or
|
|
52
|
-
in its system prompt. Every cited sentence
|
|
53
|
-
and the answer ends with a plain-text quote
|
|
53
|
+
The answering model gets the output of `veriquote prompt` (or
|
|
54
|
+
`buildCitationInstructions()`) in its system prompt. Every cited sentence then
|
|
55
|
+
ends with source and claim markers, and the answer ends with a plain-text quote
|
|
56
|
+
appendix:
|
|
54
57
|
|
|
55
58
|
```
|
|
56
59
|
Vitamin D supplementation reduced fall risk in older adults.[1]{c1}
|
|
@@ -63,128 +66,148 @@ c2|3|"BMD improved with \"high-dose\" regimens"
|
|
|
63
66
|
END_EVI1
|
|
64
67
|
```
|
|
65
68
|
|
|
66
|
-
|
|
67
|
-
weak models, and the `[n]` markers stay readable if nothing
|
|
68
|
-
|
|
69
|
+
The format is plain text rather than JSON, so it survives streaming, Markdown
|
|
70
|
+
renderers and weak models, and the `[n]` markers stay readable if nothing
|
|
71
|
+
checks them. Each citation then goes through three steps:
|
|
69
72
|
|
|
70
|
-
1. **Parse
|
|
71
|
-
2. **Match
|
|
72
|
-
tolerates whitespace, typography, OCR noise and
|
|
73
|
-
|
|
73
|
+
1. **Parse.** Every cited claim must carry a quote. Missing quotes are reported.
|
|
74
|
+
2. **Match.** Is the quote in the source? Deterministic fuzzy matching that
|
|
75
|
+
tolerates whitespace, typography, OCR noise and elisions, and reports where
|
|
76
|
+
the quote was found.
|
|
77
|
+
3. **Judge.** Does the quote support the claim? An LLM at temperature 0 answers
|
|
74
78
|
`entailed`, `partially_entailed`, `overstated`, `insufficient` or
|
|
75
79
|
`contradicted`, with a support score.
|
|
76
80
|
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
+
A citation's combined score is the lower of its match score and its support
|
|
82
|
+
score. A judge error can therefore never lift a citation above what the matcher
|
|
83
|
+
found, and a failed judge call is reported as an error, never counted as
|
|
84
|
+
support. Quoting does not make a model hallucinate less. It makes every claim
|
|
85
|
+
checkable, which is the point.
|
|
81
86
|
|
|
82
|
-
##
|
|
87
|
+
## Benchmarks
|
|
83
88
|
|
|
84
|
-
|
|
85
|
-
pipeline would hide the failures it exists to separate.
|
|
86
|
-
[`bench/`](bench), including how to reproduce it.
|
|
89
|
+
There are three benchmarks, kept separate on purpose: one blended number for a
|
|
90
|
+
two-stage pipeline would hide exactly the failures it exists to separate.
|
|
91
|
+
Everything is in [`bench/`](bench), including how to reproduce it.
|
|
87
92
|
|
|
88
93
|
<picture>
|
|
89
94
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/tango-dark.svg">
|
|
90
|
-
<img alt="
|
|
95
|
+
<img alt="The matcher blocks all 187 quotes that are not in the source and none of the 586 real quotes whose meaning was changed; the judge blocks 84% of unsupported citations and never sees the source; together they cover both kinds of failure." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/tango-light.svg">
|
|
91
96
|
</picture>
|
|
92
97
|
|
|
93
|
-
The two checks are blind in opposite places. The matcher
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
invented quote "entailed 1.00", and only the matcher notices it is not in the
|
|
98
|
-
source. That is why the combined score is `min(textMatchScore, judgeConfidence)`.
|
|
98
|
+
The two checks are blind in opposite places. The matcher never reads the claim,
|
|
99
|
+
so it cannot tell whether a real quote supports it. The judge never sees the
|
|
100
|
+
source, so an invented quote that fits the claim passes it: in the example
|
|
101
|
+
above, the judge rates the made-up quote in `c4` as "entailed 0.90".
|
|
99
102
|
|
|
100
103
|
### Matcher
|
|
101
104
|
|
|
102
105
|
<picture>
|
|
103
106
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/matcher-dark.svg">
|
|
104
|
-
<img alt="Matcher score ranges
|
|
107
|
+
<img alt="Matcher score ranges for fifteen kinds of quote damage: faithful quotes score between 0.66 and 1.0, quotes that are not in the source between 0.14 and 0.40, and quotes whose meaning was changed score high because they are still near-verbatim." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/matcher-light.svg">
|
|
105
108
|
</picture>
|
|
106
109
|
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
110
|
+
The matcher benchmark needs no API key and no labels, because the right answer
|
|
111
|
+
is known by construction: 2,061 quotes built from five Wikipedia articles, then
|
|
112
|
+
copied faithfully, reformatted, altered or replaced in fifteen different ways. `npm run bench:matcher` reproduces it in
|
|
113
|
+
about two seconds. Faithfully copied quotes never score below 0.660, quotes
|
|
114
|
+
that are not in the source never above 0.396, and the default threshold of 0.4
|
|
115
|
+
sits in that gap. The amber rows are there on purpose: a quote whose meaning was
|
|
116
|
+
changed is still near-verbatim, and catching it is the judge's job.
|
|
117
|
+
|
|
118
|
+
There are two honest limits. Invented prose built from the source's own words
|
|
119
|
+
(an adversarial case) scores high, and so can a hand-written fabrication that
|
|
120
|
+
reuses the source's vocabulary, like `c3` above at 0.48. The CLI therefore
|
|
121
|
+
fails every citation below 0.5 and leaves the rest to the judge.
|
|
116
122
|
|
|
117
123
|
### Judge
|
|
118
124
|
|
|
119
125
|
<picture>
|
|
120
126
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/judges-dark.svg">
|
|
121
|
-
<img alt="Five open judge models against
|
|
127
|
+
<img alt="Five open judge models against the human labels of ALCE: they catch between 75% and 84% of unsupported citations, and agree with the annotators on 76.7% to 80.3% of pairs, around the 77.6% of the TRUE NLI model." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/judges-light.svg">
|
|
122
128
|
</picture>
|
|
123
129
|
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
130
|
+
The judge is measured against the human annotators of ALCE[^alce], with the
|
|
131
|
+
mapping from VeriQuote's classes to ALCE's labels fixed before any model ran.
|
|
132
|
+
*Caught* is the share of citations the annotators marked as unsupported that
|
|
133
|
+
the judge did not call fully supported. On ALCE's yes-or-no question ("does the
|
|
134
|
+
source fully support the claim?"), general-purpose open models agree with the
|
|
135
|
+
annotators about as often as TRUE[^true], the specialised 11B NLI model that
|
|
136
|
+
ALCE uses for its own automatic scores (77.6%). The best of them, `glm-5.3-flash`,
|
|
137
|
+
reaches Cohen's κ 0.53 on the three-way labels (full, partial or no support).
|
|
138
|
+
For comparison, ALCE reports κ 0.525 between its automatic metric and the
|
|
139
|
+
annotators on citation precision.
|
|
140
|
+
|
|
141
|
+
Before trusting any of this, look at the left panel: even the best judge lets
|
|
142
|
+
one unsupported citation in six through as fully supported. No judge should be
|
|
143
|
+
the only check. With only 56 unsupported pairs per run the intervals are wide,
|
|
144
|
+
so neighbouring models are not really separated. Reasoning helps: with it
|
|
145
|
+
switched off, all three hybrid models catch less and agree less, in exchange
|
|
146
|
+
for answering five to ten times faster.
|
|
147
|
+
|
|
148
|
+
### Answering models
|
|
136
149
|
|
|
137
150
|
<picture>
|
|
138
151
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/protocol-dark.svg">
|
|
139
|
-
<img alt="Protocol compliance: gpt-oss-120b complete in
|
|
152
|
+
<img alt="Protocol compliance for eight answering models: qwen3.6, qwen3.5, deepseek-v4-flash and glm-5.3-flash are close to 100% complete and verbatim; gpt-oss-120b is complete in 33.3% of answers; mistral-medium quotes verbatim in 81.0% of citations; llama-3.1-8b manages about half." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/protocol-light.svg">
|
|
140
153
|
</picture>
|
|
141
154
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
155
|
+
Each model answered 18 questions twice, with the citation instructions in its
|
|
156
|
+
system prompt, and `parseAnswer()` decided mechanically whether it complied.
|
|
157
|
+
Every raw answer is in
|
|
158
|
+
[`bench/results/protocol-answers.jsonl`](bench/results/protocol-answers.jsonl).
|
|
159
|
+
The open models recommended here comply almost perfectly. The failures of the
|
|
160
|
+
others are invisible to a reader: `gpt-oss-120b` almost always prints an
|
|
161
|
+
appendix, yet only a third of its answers give every citation a quote, and
|
|
162
|
+
`mistral-medium` is nearly always complete, yet a fifth of its "quotes" are
|
|
163
|
+
paraphrases. Both answers look impeccably cited. On the three questions the
|
|
164
|
+
sources cannot answer, no model attached a citation without a real quote.
|
|
149
165
|
|
|
150
166
|
<details>
|
|
151
167
|
<summary>The numbers behind the figures</summary>
|
|
152
168
|
|
|
153
|
-
| judge model |
|
|
169
|
+
| judge model | caught ↑ | agreement | Cohen's κ |
|
|
154
170
|
| --- | ---: | ---: | ---: |
|
|
155
|
-
| `glm-5.3-flash` |
|
|
156
|
-
| `
|
|
157
|
-
| `qwen3.6-35b-a3b` |
|
|
158
|
-
| `
|
|
159
|
-
| `gpt-oss-120b` |
|
|
160
|
-
|
|
|
171
|
+
| `glm-5.3-flash` | 83.9% | 80.3% | 0.529 |
|
|
172
|
+
| `deepseek-v4-flash` | 81.8% | 78.7% | 0.435 |
|
|
173
|
+
| `qwen3.6-35b-a3b` | 81.8% | 78.1% | 0.454 |
|
|
174
|
+
| `qwen3.5-397b-a17b` | 79.2% | 79.7% | 0.483 |
|
|
175
|
+
| `gpt-oss-120b` | 75.0% | 76.7% | 0.394 |
|
|
176
|
+
| `qwen3.5-397b-a17b`, no reasoning | 78.6% | 77.7% | 0.410 |
|
|
177
|
+
| `qwen3.6-35b-a3b`, no reasoning | 78.6% | 77.3% | 0.397 |
|
|
178
|
+
| `glm-5.3-flash`, no reasoning | 75.6% | 74.0% | 0.373 |
|
|
179
|
+
| TRUE (T5-11B), as reported by ALCE[^alce] | | 77.6% | |
|
|
161
180
|
|
|
162
181
|
| quote family | n | median score | accepted at 0.4 |
|
|
163
182
|
| --- | ---: | ---: | ---: |
|
|
164
183
|
| faithful but reformatted | 1,121 | 1.000 | 100.0% |
|
|
165
184
|
| near-verbatim, meaning changed | 586 | 0.932 | 100.0% |
|
|
166
|
-
|
|
|
185
|
+
| not in the source | 187 | 0.249 | 0.0% |
|
|
167
186
|
|
|
168
187
|
| answering model | appendix | complete | verbatim | warning-free |
|
|
169
188
|
| --- | ---: | ---: | ---: | ---: |
|
|
170
|
-
|
|
|
171
|
-
|
|
|
172
|
-
|
|
|
173
|
-
|
|
|
174
|
-
|
|
175
|
-
|
|
189
|
+
| qwen3.6-35b-a3b | 100.0% | 100.0% | 100.0% | 100.0% |
|
|
190
|
+
| qwen3.5-397b-a17b | 100.0% | 100.0% | 100.0% | 100.0% |
|
|
191
|
+
| deepseek-v4-flash | 100.0% | 100.0% | 99.2% | 93.3% |
|
|
192
|
+
| glm-5.3-flash | 96.6% | 96.6% | 99.6% | 96.6% |
|
|
193
|
+
| gemma-4-31b-it | 80.0% | 80.0% | 96.6% | 80.0% |
|
|
194
|
+
| mistral-medium-3.5-128b | 96.7% | 93.3% | 81.0% | 96.7% |
|
|
195
|
+
| gpt-oss-120b | 96.7% | 33.3% | 91.1% | 33.3% |
|
|
196
|
+
| llama-3.1-8b-instruct | 66.7% | 43.3% | 50.3% | 20.0% |
|
|
197
|
+
|
|
198
|
+
The figures are rendered from `bench/results/*.json` by
|
|
176
199
|
[`bench/figures/render.mjs`](bench/figures/render.mjs).
|
|
177
200
|
</details>
|
|
178
201
|
|
|
179
202
|
## Use from an agent
|
|
180
203
|
|
|
181
|
-
An agent that reads sources and writes conclusions is exactly the case
|
|
182
|
-
built for.
|
|
183
|
-
[Agent Skill](integrations/verify-citations/SKILL.md) that has the agent
|
|
184
|
-
its sourced answer in the checkable format and run `veriquote check` on
|
|
185
|
-
before
|
|
186
|
-
pass with its own
|
|
187
|
-
|
|
204
|
+
An agent that reads sources and writes conclusions is exactly the case
|
|
205
|
+
VeriQuote was built for. [`verify-citations`](integrations/verify-citations) is
|
|
206
|
+
an [Agent Skill](integrations/verify-citations/SKILL.md) that has the agent
|
|
207
|
+
write its sourced answer in the checkable format and run `veriquote check` on
|
|
208
|
+
it before showing it to you. The CLI fetches every cited URL itself, so the
|
|
209
|
+
agent cannot pass with its own truncated or misremembered copy of a page. The
|
|
210
|
+
exit code tells the agent what to do next without parsing anything:
|
|
188
211
|
|
|
189
212
|
```
|
|
190
213
|
exit 0 verdict "pass" every cited claim is grounded -> present the answer
|
|
@@ -192,8 +215,8 @@ exit 2 verdict "revise" problems[] + instructionsForModel -> fix and re-check
|
|
|
192
215
|
exit 1 bad input or unreachable source -> do NOT claim the answer was verified
|
|
193
216
|
```
|
|
194
217
|
|
|
195
|
-
|
|
196
|
-
anything that can run a shell command.
|
|
218
|
+
The skill works in any host that reads Agent Skills, such as Claude Code, and
|
|
219
|
+
the CLI in anything that can run a shell command.
|
|
197
220
|
|
|
198
221
|
## Use as a library
|
|
199
222
|
|
|
@@ -208,7 +231,6 @@ const systemPrompt = `${yourAssistantPrompt}\n\n${buildCitationInstructions()}`;
|
|
|
208
231
|
// Give the sources to the model as numbered blocks [1], [2], …
|
|
209
232
|
```
|
|
210
233
|
|
|
211
|
-
|
|
212
234
|
```ts
|
|
213
235
|
import { ChatCompletionsJudge, verifyAnswer } from 'veriquote';
|
|
214
236
|
|
|
@@ -224,7 +246,7 @@ const report = await verifyAnswer({
|
|
|
224
246
|
{ title: 'Trial A', url: 'https://…', text: extractedFullText1 },
|
|
225
247
|
{ title: 'Trial B', url: 'https://…', text: extractedFullText2 },
|
|
226
248
|
],
|
|
227
|
-
judge, // omit
|
|
249
|
+
judge, // omit to check the quotes only
|
|
228
250
|
});
|
|
229
251
|
|
|
230
252
|
console.log(report.summary);
|
|
@@ -237,93 +259,99 @@ for (const c of report.citations) {
|
|
|
237
259
|
}
|
|
238
260
|
```
|
|
239
261
|
|
|
240
|
-
`report.cleanText` is the answer
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
each citation's match score, judge class and combined score next to its
|
|
245
|
-
footnote, so a reader can see which sentence is load-bearing without opening a
|
|
246
|
-
source.
|
|
262
|
+
`report.cleanText` is the answer without the `{cX}` markers, ready to render.
|
|
263
|
+
Show each citation's match score, judge class and combined score next to its
|
|
264
|
+
footnote, so a reader can see which sentence carries weight without opening a
|
|
265
|
+
single source.
|
|
247
266
|
|
|
248
267
|
| Export | Purpose |
|
|
249
268
|
| --- | --- |
|
|
250
|
-
| `buildCitationInstructions(options?)` | Prompt block for the answering model
|
|
251
|
-
| `verifyAnswer(options)` |
|
|
252
|
-
| `parseAnswer(answer)` |
|
|
253
|
-
| `parseEvi1Appendix` / `stripEvi1Appendix` / `serializeEvi1Appendix` | Low-level
|
|
254
|
-
| `matchQuoteAgainstSource(quote, source, options?)` |
|
|
255
|
-
| `ChatCompletionsJudge` |
|
|
256
|
-
| `EntailmentJudge` (interface) | Bring your own judge
|
|
257
|
-
| `gateReport(report, answer)` | Pass
|
|
269
|
+
| `buildCitationInstructions(options?)` | Prompt block for the answering model; budgets and quote lengths are configurable. |
|
|
270
|
+
| `verifyAnswer(options)` | The full pipeline: parse, match, judge, report. |
|
|
271
|
+
| `parseAnswer(answer)` | Claims, quotes and protocol warnings, without verifying anything. |
|
|
272
|
+
| `parseEvi1Appendix` / `stripEvi1Appendix` / `serializeEvi1Appendix` | Low-level handling of the appendix. |
|
|
273
|
+
| `matchQuoteAgainstSource(quote, source, options?)` | The deterministic matcher on its own. |
|
|
274
|
+
| `ChatCompletionsJudge` | Judge for any OpenAI-compatible API. |
|
|
275
|
+
| `EntailmentJudge` (interface) | Bring your own judge, such as a local NLI model. |
|
|
276
|
+
| `gateReport(report, answer)` | Pass or revise, the list of problems, uncited sentences and a correction prompt. |
|
|
258
277
|
| `fetchSource(url)` / `htmlToText(html)` | Fetch a source independently of the model and extract its text. |
|
|
259
278
|
|
|
260
|
-
All inputs and outputs are plain, serializable data
|
|
261
|
-
[`src/types.ts`](src/types.ts)
|
|
262
|
-
[`docs/DESIGN.md`](docs/DESIGN.md).
|
|
279
|
+
All inputs and outputs are plain, serializable data. See
|
|
280
|
+
[`src/types.ts`](src/types.ts) for the data model and
|
|
281
|
+
[`docs/DESIGN.md`](docs/DESIGN.md) for scoring and thresholds.
|
|
263
282
|
|
|
264
|
-
| Class |
|
|
283
|
+
| Class | Support score | Meaning |
|
|
265
284
|
| --- | --- | --- |
|
|
266
|
-
| `entailed` | 0.9–1.0 |
|
|
267
|
-
| `partially_entailed` | 0.5–0.8 |
|
|
268
|
-
| `overstated` | 0.3–0.6 |
|
|
269
|
-
| `insufficient` | 0.1–0.4 |
|
|
270
|
-
| `contradicted` | 0.0 |
|
|
271
|
-
| `error` |
|
|
285
|
+
| `entailed` | 0.9–1.0 | The quote fully covers the claim. |
|
|
286
|
+
| `partially_entailed` | 0.5–0.8 | The core is supported, details are missing. |
|
|
287
|
+
| `overstated` | 0.3–0.6 | The claim is stronger or more general than the quote. |
|
|
288
|
+
| `insufficient` | 0.1–0.4 | The quote is related but does not confirm the claim. |
|
|
289
|
+
| `contradicted` | 0.0 | The quote says the opposite. |
|
|
290
|
+
| `error` | none | The judge failed for this item; it is reported, never dropped. |
|
|
272
291
|
|
|
273
292
|
## How this differs from the alternatives
|
|
274
293
|
|
|
275
294
|
| | verbatim quote checked | claim↔evidence checked | model-agnostic | runtime |
|
|
276
295
|
| --- | --- | --- | --- | --- |
|
|
277
|
-
| **VeriQuote** | yes, deterministic | yes, pluggable judge | yes |
|
|
278
|
-
| [Anthropic Citations API](https://platform.claude.com/docs/en/build-with-claude/citations) |
|
|
279
|
-
| [LettuceDetect](https://github.com/KRLabsOrg/LettuceDetect) | no
|
|
280
|
-
| [RAGAS](https://github.com/explodinggradients/ragas) and eval frameworks | no | yes, as an offline metric | yes | Python, offline
|
|
296
|
+
| **VeriQuote** | yes, deterministic | yes, pluggable judge | yes | TypeScript, no dependencies, browser and edge |
|
|
297
|
+
| [Anthropic Citations API](https://platform.claude.com/docs/en/build-with-claude/citations) | not needed: spans are extracted, so they are real by construction | no | Claude only | hosted |
|
|
298
|
+
| [LettuceDetect](https://github.com/KRLabsOrg/LettuceDetect) | no quote protocol | yes, span-level model | yes | Python and model weights |
|
|
299
|
+
| [RAGAS](https://github.com/explodinggradients/ragas) and eval frameworks | no | yes, as an offline metric | yes | Python, offline evaluation |
|
|
281
300
|
|
|
282
301
|
**Use the Citations API instead** if you are on Claude and only need to know
|
|
283
|
-
that a span is real
|
|
302
|
+
that a span is real. It guarantees that by construction, which is stronger than
|
|
284
303
|
any matcher. It does not tell you whether the span supports the sentence built
|
|
285
|
-
on it
|
|
304
|
+
on it; for that, pair it with VeriQuote's judge and skip the matcher.
|
|
286
305
|
|
|
287
306
|
**Use LettuceDetect instead** if you want unsupported spans flagged in an
|
|
288
|
-
answer that
|
|
289
|
-
in Python. It solves the
|
|
290
|
-
model commits to in the first place.
|
|
291
|
-
|
|
292
|
-
VeriQuote's own niche is narrow and worth stating plainly: you want the
|
|
293
|
-
answering model pinned to a quote *before* it
|
|
294
|
-
deterministic half of the check to run anywhere including a browser
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
verbatim
|
|
302
|
-
|
|
303
|
-
labels.
|
|
307
|
+
answer that follows no citation protocol at all, and you are happy to run a
|
|
308
|
+
model in Python. It solves the problem after the fact; VeriQuote changes what
|
|
309
|
+
the answering model commits to in the first place.
|
|
310
|
+
|
|
311
|
+
VeriQuote's own niche is narrow, and worth stating plainly: you want the
|
|
312
|
+
answering model pinned to a quote *before* it writes, you want the
|
|
313
|
+
deterministic half of the check to run anywhere, including a browser, and you
|
|
314
|
+
want per-claim results you can show a reader rather than one score on a
|
|
315
|
+
dashboard.
|
|
316
|
+
|
|
317
|
+
The closest related work is a recent study by Zhang et al.[^zhang], which
|
|
318
|
+
evaluates the same design (inline verbatim quotes, tiered verbatim matching and
|
|
319
|
+
an LLM judge) on clinical guidelines. It found that `claude-opus-5` quoted
|
|
320
|
+
verbatim for 98.0% of its claims but fully substantiated only 37.1% of them.
|
|
321
|
+
VeriQuote is the reusable library, CLI and agent skill for that kind of check,
|
|
322
|
+
with its judge measured against human labels.
|
|
304
323
|
|
|
305
324
|
## Security
|
|
306
325
|
|
|
307
326
|
- **Keep your key on the server.** `ChatCompletionsJudge` needs an API key; in
|
|
308
|
-
your own app, call `verifyAnswer` from a backend.
|
|
309
|
-
the browser only with a key the visitor enters.
|
|
327
|
+
your own app, call `verifyAnswer` from a backend. The demo runs the judge in
|
|
328
|
+
the browser only with a key the visitor enters.
|
|
310
329
|
- **Source text is untrusted.** Judge inputs are length-capped, stripped of
|
|
311
|
-
control characters and HTML, and
|
|
312
|
-
|
|
313
|
-
and invented item IDs are rejected. Nothing is ever `eval`ed.
|
|
330
|
+
control characters and HTML, and marked as data in the prompt. The judge's
|
|
331
|
+
output is checked against a closed vocabulary: unknown classes, out-of-range
|
|
332
|
+
scores and invented item IDs are rejected. Nothing is ever `eval`ed.
|
|
314
333
|
|
|
315
334
|
## Reproducibility
|
|
316
335
|
|
|
317
|
-
The matcher is pure: same inputs
|
|
318
|
-
(pass `seed` where the provider supports it), but hosted
|
|
319
|
-
|
|
320
|
-
|
|
336
|
+
The matcher is pure: the same inputs always give the same score. The judge
|
|
337
|
+
runs at temperature 0 (pass `seed` where the provider supports it), but hosted
|
|
338
|
+
models are only deterministic on a best-effort basis. For strict
|
|
339
|
+
reproducibility, pin the model version or put a self-hosted model behind the
|
|
340
|
+
`EntailmentJudge` interface.
|
|
321
341
|
|
|
322
342
|
## Citing
|
|
323
343
|
|
|
324
|
-
If you use VeriQuote in academic work, please cite the
|
|
325
|
-
`CITATION.cff`).
|
|
344
|
+
If you use VeriQuote in academic work, please cite it[^veriquote]; the details
|
|
345
|
+
are also in [`CITATION.cff`](CITATION.cff).
|
|
326
346
|
|
|
327
347
|
## License
|
|
328
348
|
|
|
329
349
|
[MIT](LICENSE)
|
|
350
|
+
|
|
351
|
+
[^alce]: Gao, T., Yen, H., Yu, J., & Chen, D. (2023). Enabling large language models to generate text with citations. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing* (pp. 6465–6488). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.398
|
|
352
|
+
|
|
353
|
+
[^true]: Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022). TRUE: Re-evaluating factual consistency evaluation. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies* (pp. 3905–3920). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.naacl-main.287
|
|
354
|
+
|
|
355
|
+
[^zhang]: Zhang, J., Chen, Y., Commodore-Mensah, Y., & Oberst, M. (2026). *Verifiable by construction: Claim-level evaluation of verbatim citation in clinical question answering* (Version 2) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2609.15964
|
|
356
|
+
|
|
357
|
+
[^veriquote]: Heilmann, E. (2026). *VeriQuote: Deterministic and semantic verification of quote-grounded LLM citations* (Version 0.2.1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.21552379
|
|
@@ -35,6 +35,13 @@ export interface ChatJudgeOptions {
|
|
|
35
35
|
quote?: number;
|
|
36
36
|
context?: number;
|
|
37
37
|
};
|
|
38
|
+
/**
|
|
39
|
+
* Extra fields merged into every request body, for provider-specific switches,
|
|
40
|
+
* e.g. `{ chat_template_kwargs: { enable_thinking: false } }` to turn off
|
|
41
|
+
* reasoning on vLLM-served hybrid models. Cannot override model, messages,
|
|
42
|
+
* temperature or response format.
|
|
43
|
+
*/
|
|
44
|
+
extraBody?: Record<string, unknown>;
|
|
38
45
|
/** Custom fetch (for testing or non-standard runtimes). Default globalThis.fetch. */
|
|
39
46
|
fetch?: typeof globalThis.fetch;
|
|
40
47
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"chat-judge.d.ts","sourceRoot":"","sources":["../../src/judge/chat-judge.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,eAAe,EAAE,eAAe,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAIvG,eAAO,MAAM,kBAAkB,EAAE,SAAS,eAAe,EAMxD,CAAC;AAsBF,MAAM,WAAW,gBAAgB;IAC/B,6EAA6E;IAC7E,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,2EAA2E;IAC3E,KAAK,EAAE,MAAM,CAAC;IACd,iFAAiF;IACjF,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,gEAAgE;IAChE,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC,qCAAqC;IACrC,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,0DAA0D;IAC1D,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,4DAA4D;IAC5D,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,yDAAyD;IACzD,IAAI,CAAC,EAAE;QAAE,EAAE,CAAC,EAAE,MAAM,CAAC;QAAC,KAAK,CAAC,EAAE,MAAM,CAAC;QAAC,KAAK,CAAC,EAAE,MAAM,CAAC;QAAC,OAAO,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;IACzE,qFAAqF;IACrF,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;CACjC;
|
|
1
|
+
{"version":3,"file":"chat-judge.d.ts","sourceRoot":"","sources":["../../src/judge/chat-judge.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,eAAe,EAAE,eAAe,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAIvG,eAAO,MAAM,kBAAkB,EAAE,SAAS,eAAe,EAMxD,CAAC;AAsBF,MAAM,WAAW,gBAAgB;IAC/B,6EAA6E;IAC7E,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,2EAA2E;IAC3E,KAAK,EAAE,MAAM,CAAC;IACd,iFAAiF;IACjF,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,gEAAgE;IAChE,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC,qCAAqC;IACrC,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,0DAA0D;IAC1D,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,4DAA4D;IAC5D,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,yDAAyD;IACzD,IAAI,CAAC,EAAE;QAAE,EAAE,CAAC,EAAE,MAAM,CAAC;QAAC,KAAK,CAAC,EAAE,MAAM,CAAC;QAAC,KAAK,CAAC,EAAE,MAAM,CAAC;QAAC,OAAO,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;IACzE;;;;;OAKG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IACpC,qFAAqF;IACrF,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;CACjC;AAWD,qBAAa,oBAAqB,YAAW,eAAe;IAC1D,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAkB;gBAE3B,OAAO,EAAE,gBAAgB;IAiB/B,KAAK,CACT,KAAK,EAAE,eAAe,EAAE,EACxB,OAAO,CAAC,EAAE;QAAE,MAAM,CAAC,EAAE,WAAW,CAAA;KAAE,GACjC,OAAO,CAAC,gBAAgB,EAAE,CAAC;IAa9B,OAAO,CAAC,QAAQ;YAcF,UAAU;YAaV,gBAAgB;YAchB,OAAO;IAiDrB,OAAO,CAAC,YAAY;CAYrB"}
|
package/dist/judge/chat-judge.js
CHANGED
|
@@ -51,6 +51,7 @@ export class ChatCompletionsJudge {
|
|
|
51
51
|
maxRetries: options.maxRetries ?? 2,
|
|
52
52
|
seed: options.seed,
|
|
53
53
|
caps: { id: 80, claim: 700, quote: 700, context: 1200, ...options.caps },
|
|
54
|
+
extraBody: options.extraBody ?? {},
|
|
54
55
|
fetch: options.fetch ?? globalThis.fetch.bind(globalThis),
|
|
55
56
|
};
|
|
56
57
|
}
|
|
@@ -121,6 +122,7 @@ export class ChatCompletionsJudge {
|
|
|
121
122
|
},
|
|
122
123
|
signal: controller.signal,
|
|
123
124
|
body: JSON.stringify({
|
|
125
|
+
...this.opts.extraBody,
|
|
124
126
|
model: this.opts.model,
|
|
125
127
|
temperature: 0,
|
|
126
128
|
...(this.opts.seed !== undefined ? { seed: this.opts.seed } : {}),
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"chat-judge.js","sourceRoot":"","sources":["../../src/judge/chat-judge.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAGH,OAAO,EAAE,kBAAkB,EAAE,iBAAiB,EAAE,aAAa,EAAE,MAAM,uBAAuB,CAAC;AAC7F,OAAO,EAAE,eAAe,EAAE,MAAM,mBAAmB,CAAC;AAEpD,MAAM,CAAC,MAAM,kBAAkB,GAA+B;IAC5D,UAAU;IACV,oBAAoB;IACpB,YAAY;IACZ,cAAc;IACd,cAAc;CACf,CAAC;AAEF,MAAM,aAAa,GAAG;IACpB,4DAA4D;IAC5D,kCAAkC;IAClC,sJAAsJ;IACtJ,sEAAsE;IACtE,+FAA+F;IAC/F,2FAA2F;CAC5F,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAEZ,MAAM,UAAU,GAAG;IACjB,uIAAuI;IACvI,gEAAgE;IAChE,kEAAkE;IAClE,uFAAuF;IACvF,sGAAsG;IACtG,wFAAwF;IACxF,uEAAuE;IACvE,8GAA8G;CAC/G,CAAC;
|
|
1
|
+
{"version":3,"file":"chat-judge.js","sourceRoot":"","sources":["../../src/judge/chat-judge.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAGH,OAAO,EAAE,kBAAkB,EAAE,iBAAiB,EAAE,aAAa,EAAE,MAAM,uBAAuB,CAAC;AAC7F,OAAO,EAAE,eAAe,EAAE,MAAM,mBAAmB,CAAC;AAEpD,MAAM,CAAC,MAAM,kBAAkB,GAA+B;IAC5D,UAAU;IACV,oBAAoB;IACpB,YAAY;IACZ,cAAc;IACd,cAAc;CACf,CAAC;AAEF,MAAM,aAAa,GAAG;IACpB,4DAA4D;IAC5D,kCAAkC;IAClC,sJAAsJ;IACtJ,sEAAsE;IACtE,+FAA+F;IAC/F,2FAA2F;CAC5F,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAEZ,MAAM,UAAU,GAAG;IACjB,uIAAuI;IACvI,gEAAgE;IAChE,kEAAkE;IAClE,uFAAuF;IACvF,sGAAsG;IACtG,wFAAwF;IACxF,uEAAuE;IACvE,8GAA8G;CAC/G,CAAC;AAyCF,MAAM,OAAO,oBAAoB;IACd,IAAI,CAAkB;IAEvC,YAAY,OAAyB;QACnC,IAAI,CAAC,OAAO,CAAC,KAAK;YAAE,MAAM,IAAI,KAAK,CAAC,4CAA4C,CAAC,CAAC;QAClF,IAAI,CAAC,IAAI,GAAG;YACV,MAAM,EAAE,OAAO,CAAC,MAAM;YACtB,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,OAAO,EAAE,CAAC,OAAO,CAAC,OAAO,IAAI,2BAA2B,CAAC,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;YAC7E,OAAO,EAAE,OAAO,CAAC,OAAO,IAAI,EAAE;YAC9B,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,EAAE;YAClC,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,MAAM;YACtC,UAAU,EAAE,OAAO,CAAC,UAAU,IAAI,CAAC;YACnC,IAAI,EAAE,OAAO,CAAC,IAAI;YAClB,IAAI,EAAE,EAAE,EAAE,EAAE,EAAE,EAAE,KAAK,EAAE,GAAG,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,IAAI,EAAE,GAAG,OAAO,CAAC,IAAI,EAAE;YACxE,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,EAAE;YAClC,KAAK,EAAE,OAAO,CAAC,KAAK,IAAI,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,UAAU,CAAC;SAC1D,CAAC;IACJ,CAAC;IAED,KAAK,CAAC,KAAK,CACT,KAAwB,EACxB,OAAkC;QAElC,IAAI,CAAC,KAAK,CAAC,MAAM;YAAE,OAAO,EAAE,CAAC;QAC7B,MAAM,SAAS,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,CAAC;QACvD,MAAM,OAAO,GAAwB,EAAE,CAAC;QACxC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,SAAS,CAAC,MAAM,EAAE,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,CAAC;YAC/D,OAAO,CAAC,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC;QAC5D,CAAC;QACD,MAAM,OAAO,GAAG,MAAM,OAAO,CAAC,GAAG,CAC/B,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,CAAC,CACxD,CAAC;QACF,OAAO,OAAO,CAAC,IAAI,EAAE,CAAC;IACxB,CAAC;IAEO,QAAQ,CAAC,IAAqB;QACpC,MAAM,KAAK,GAAG,CAAC,CAAS,EAAE,GAAW,EAAE,IAAI,GAAG,KAAK,EAAE,EAAE;YACrD,IAAI,GAAG,GAAG,iBAAiB,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;YAC7C,IAAI,IAAI;gBAAE,GAAG,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;YACnC,OAAO,kBAAkB,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;QAC/C,CAAC,CAAC;QACF,OAAO;YACL,EAAE,EAAE,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;YACrC,KAAK,EAAE,KAAK,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC;YAC9C,KAAK,EAAE,KAAK,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC;YAC9C,OAAO,EAAE,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,IAAI,CAAC;SAC3D,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,UAAU,CACtB,KAAwB,EACxB,MAAoB;QAEpB,IAAI,CAAC;YACH,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC;YAC3D,OAAO,IAAI,CAAC,YAAY,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;QAC3C,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,IAAI,MAAM,EAAE,OAAO;gBAAE,MAAM,CAAC,CAAC;YAC7B,OAAO,KAAK,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,WAAW,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QACpD,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gBAAgB,CAAC,KAAwB,EAAE,MAAoB;QAC3E,IAAI,SAAkB,CAAC;QACvB,KAAK,IAAI,OAAO,GAAG,CAAC,EAAE,OAAO,IAAI,IAAI,CAAC,IAAI,CAAC,UAAU,EAAE,OAAO,EAAE,EAAE,CAAC;YACjE,IAAI,OAAO,GAAG,CAAC;gBAAE,MAAM,KAAK,CAAC,GAAG,GAAG,CAAC,IAAI,CAAC,OAAO,GAAG,CAAC,CAAC,EAAE,MAAM,CAAC,CAAC;YAC/D,IAAI,CAAC;gBACH,OAAO,MAAM,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC;YAC3C,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACX,SAAS,GAAG,CAAC,CAAC;gBACd,IAAI,MAAM,EAAE,OAAO,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC;oBAAE,MAAM,CAAC,CAAC;YAClD,CAAC;QACH,CAAC;QACD,MAAM,SAAS,CAAC;IAClB,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,KAAwB,EAAE,WAAyB;QACvE,MAAM,UAAU,GAAG,IAAI,eAAe,EAAE,CAAC;QACzC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC,eAAe,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;QAClG,MAAM,YAAY,GAAG,GAAG,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,WAAW,EAAE,MAAM,CAAC,CAAC;QACjE,WAAW,EAAE,gBAAgB,CAAC,OAAO,EAAE,YAAY,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;QAErE,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO,mBAAmB,EAAE;gBACzE,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE;oBACP,cAAc,EAAE,kBAAkB;oBAClC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,UAAU,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;oBAC5E,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO;iBACrB;gBACD,MAAM,EAAE,UAAU,CAAC,MAAM;gBACzB,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;oBACnB,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS;oBACtB,KAAK,EAAE,IAAI,CAAC,IAAI,CAAC,KAAK;oBACtB,WAAW,EAAE,CAAC;oBACd,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;oBACjE,eAAe,EAAE,EAAE,IAAI,EAAE,aAAa,EAAE;oBACxC,QAAQ,EAAE;wBACR,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,aAAa,EAAE;wBAC1C;4BACE,IAAI,EAAE,MAAM;4BACZ,OAAO,EAAE,IAAI,CAAC,SAAS,CAAC;gCACtB,IAAI,EAAE,+BAA+B;gCACrC,KAAK,EAAE,UAAU;gCACjB,KAAK,EAAE,KAAK;6BACb,CAAC;yBACH;qBACF;iBACF,CAAC;aACH,CAAC,CAAC;YACH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;gBACZ,MAAM,GAAG,GAAG,IAAI,KAAK,CAAC,cAAc,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC;gBACjD,GAAmC,CAAC,MAAM,GAAG,GAAG,CAAC,MAAM,CAAC;gBACzD,MAAM,GAAG,CAAC;YACZ,CAAC;YACD,MAAM,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAE7B,CAAC;YACF,OAAO,MAAM,CAAC,IAAI,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,IAAI,EAAE,CAAC,CAAC;QAC5D,CAAC;gBAAS,CAAC;YACT,YAAY,CAAC,KAAK,CAAC,CAAC;YACpB,WAAW,EAAE,mBAAmB,CAAC,OAAO,EAAE,YAAY,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC;IAEO,YAAY,CAAC,OAAe,EAAE,KAAwB;QAC5D,MAAM,GAAG,GAAG,eAAe,CAAC,OAAO,CAAC,CAAC;QACrC,MAAM,IAAI,GAAG,IAAI,GAAG,EAA4B,CAAC;QACjD,KAAK,MAAM,IAAI,IAAI,GAAG,EAAE,CAAC;YACvB,MAAM,SAAS,GAAG,cAAc,CAAC,IAAI,CAAC,CAAC;YACvC,0EAA0E;YAC1E,IAAI,SAAS,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;gBAAE,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,EAAE,SAAS,CAAC,MAAM,CAAC,CAAC;QACrF,CAAC;QACD,OAAO,KAAK,CAAC,GAAG,CACd,CAAC,GAAG,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,WAAW,CAAC,cAAc,CAAC,CACzD,CAAC;IACJ,CAAC;CACF;AAED,SAAS,cAAc,CAAC,IAAa;IACnC,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,KAAK,IAAI;QAAE,OAAO,IAAI,CAAC;IAC3D,MAAM,CAAC,GAAG,IAA+B,CAAC;IAC1C,MAAM,EAAE,GAAG,OAAO,CAAC,CAAC,EAAE,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;IAChD,IAAI,CAAC,EAAE;QAAE,OAAO,IAAI,CAAC;IACrB,MAAM,GAAG,GAAI,kBAAwC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;QAC7E,CAAC,CAAE,CAAC,CAAC,KAAyB;QAC9B,CAAC,CAAC,OAAO,CAAC;IACZ,MAAM,UAAU,GACd,OAAO,CAAC,CAAC,UAAU,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,UAAU,CAAC;QAC/D,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,UAAU,CAAC,CAAC;QACxC,CAAC,CAAC,IAAI,CAAC;IACX,MAAM,OAAO,GAAG,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;QACtC,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,kBAAkB,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;QAC/E,CAAC,CAAC,EAAE,CAAC;IACP,OAAO,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,KAAK,EAAE,GAAG,EAAE,UAAU,EAAE,OAAO,EAAE,EAAE,CAAC;AAC7D,CAAC;AAED,SAAS,WAAW,CAAC,MAAc;IACjC,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,MAAM,CAAC,EAAE,CAAC;AACjE,CAAC;AAED,SAAS,WAAW,CAAC,CAAU;IAC7B,MAAM,MAAM,GAAI,CAAgC,EAAE,MAAM,CAAC;IACzD,IAAI,MAAM,KAAK,SAAS;QAAE,OAAO,MAAM,KAAK,GAAG,IAAI,MAAM,IAAI,GAAG,CAAC;IACjE,OAAO,IAAI,CAAC,CAAC,uCAAuC;AACtD,CAAC;AAED,SAAS,SAAS,CAAC,CAAU;IAC3B,OAAO,CAAC,YAAY,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;AACpD,CAAC;AAED,SAAS,KAAK,CAAC,EAAU,EAAE,MAAoB;IAC7C,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;QACrC,MAAM,CAAC,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,OAAO,EAAE,EAAE,EAAE,CAAC,CAAC;QAC1C,MAAM,EAAE,gBAAgB,CACtB,OAAO,EACP,GAAG,EAAE;YACH,YAAY,CAAC,CAAC,CAAC,CAAC;YAChB,MAAM,CAAC,MAAM,CAAC,MAAM,YAAY,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC;QAChF,CAAC,EACD,EAAE,IAAI,EAAE,IAAI,EAAE,CACf,CAAC;IACJ,CAAC,CAAC,CAAC;AACL,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "veriquote",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.1",
|
|
4
4
|
"description": "Deterministic + semantic verification of quote-grounded LLM citations (EVI1 protocol): fuzzy verbatim-quote matching and an LLM entailment judge for transparent, per-claim hallucination detection.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"llm",
|
|
@@ -60,9 +60,11 @@
|
|
|
60
60
|
"bench:probe": "npm run build && node --env-file=.env bench/probe.mjs",
|
|
61
61
|
"bench:protocol": "npm run build && node --env-file=.env bench/protocol/run.mjs --json bench/results/protocol.json",
|
|
62
62
|
"bench:judge:prepare": "node bench/judge/prepare-alce.mjs",
|
|
63
|
-
"bench:judge": "npm run build && node --env-file=.env bench/judge/run.mjs"
|
|
63
|
+
"bench:judge": "npm run build && node --env-file=.env bench/judge/run.mjs",
|
|
64
|
+
"demo:lib": "esbuild src/index.ts --bundle --format=esm --minify --legal-comments=none --outfile=demo/lib/veriquote.js"
|
|
64
65
|
},
|
|
65
66
|
"devDependencies": {
|
|
67
|
+
"esbuild": "^0.25.12",
|
|
66
68
|
"typescript": "^5.5.0",
|
|
67
69
|
"vitest": "^3.0.0"
|
|
68
70
|
}
|
package/src/judge/chat-judge.ts
CHANGED
|
@@ -61,11 +61,19 @@ export interface ChatJudgeOptions {
|
|
|
61
61
|
seed?: number;
|
|
62
62
|
/** Character caps applied to inputs before prompting. */
|
|
63
63
|
caps?: { id?: number; claim?: number; quote?: number; context?: number };
|
|
64
|
+
/**
|
|
65
|
+
* Extra fields merged into every request body, for provider-specific switches,
|
|
66
|
+
* e.g. `{ chat_template_kwargs: { enable_thinking: false } }` to turn off
|
|
67
|
+
* reasoning on vLLM-served hybrid models. Cannot override model, messages,
|
|
68
|
+
* temperature or response format.
|
|
69
|
+
*/
|
|
70
|
+
extraBody?: Record<string, unknown>;
|
|
64
71
|
/** Custom fetch (for testing or non-standard runtimes). Default globalThis.fetch. */
|
|
65
72
|
fetch?: typeof globalThis.fetch;
|
|
66
73
|
}
|
|
67
74
|
|
|
68
|
-
interface ResolvedOptions extends Required<Omit<ChatJudgeOptions, 'apiKey' | 'seed' | 'headers' | 'caps' | 'fetch'>> {
|
|
75
|
+
interface ResolvedOptions extends Required<Omit<ChatJudgeOptions, 'apiKey' | 'seed' | 'headers' | 'caps' | 'fetch' | 'extraBody'>> {
|
|
76
|
+
extraBody: Record<string, unknown>;
|
|
69
77
|
apiKey?: string;
|
|
70
78
|
seed?: number;
|
|
71
79
|
headers: Record<string, string>;
|
|
@@ -88,6 +96,7 @@ export class ChatCompletionsJudge implements EntailmentJudge {
|
|
|
88
96
|
maxRetries: options.maxRetries ?? 2,
|
|
89
97
|
seed: options.seed,
|
|
90
98
|
caps: { id: 80, claim: 700, quote: 700, context: 1200, ...options.caps },
|
|
99
|
+
extraBody: options.extraBody ?? {},
|
|
91
100
|
fetch: options.fetch ?? globalThis.fetch.bind(globalThis),
|
|
92
101
|
};
|
|
93
102
|
}
|
|
@@ -165,6 +174,7 @@ export class ChatCompletionsJudge implements EntailmentJudge {
|
|
|
165
174
|
},
|
|
166
175
|
signal: controller.signal,
|
|
167
176
|
body: JSON.stringify({
|
|
177
|
+
...this.opts.extraBody,
|
|
168
178
|
model: this.opts.model,
|
|
169
179
|
temperature: 0,
|
|
170
180
|
...(this.opts.seed !== undefined ? { seed: this.opts.seed } : {}),
|