veriquote 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CITATION.cff +2 -2
- package/README.md +102 -51
- package/dist/cli/main.d.ts.map +1 -1
- package/dist/cli/main.js +42 -5
- package/dist/cli/main.js.map +1 -1
- package/dist/gate.d.ts +13 -3
- package/dist/gate.d.ts.map +1 -1
- package/dist/gate.js +20 -2
- package/dist/gate.js.map +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -1
- package/dist/index.js.map +1 -1
- package/dist/match/fuzzy.d.ts +10 -2
- package/dist/match/fuzzy.d.ts.map +1 -1
- package/dist/match/fuzzy.js +87 -11
- package/dist/match/fuzzy.js.map +1 -1
- package/dist/protocol/evi1.d.ts +21 -1
- package/dist/protocol/evi1.d.ts.map +1 -1
- package/dist/protocol/evi1.js +169 -28
- package/dist/protocol/evi1.js.map +1 -1
- package/dist/protocol/prompt.d.ts +6 -0
- package/dist/protocol/prompt.d.ts.map +1 -1
- package/dist/protocol/prompt.js +35 -10
- package/dist/protocol/prompt.js.map +1 -1
- package/dist/report.d.ts.map +1 -1
- package/dist/report.js +52 -15
- package/dist/report.js.map +1 -1
- package/dist/source/html.d.ts.map +1 -1
- package/dist/source/html.js +3 -0
- package/dist/source/html.js.map +1 -1
- package/dist/types.d.ts +23 -2
- package/dist/types.d.ts.map +1 -1
- package/package.json +3 -3
- package/src/cli/main.ts +41 -6
- package/src/gate.ts +30 -5
- package/src/index.ts +1 -0
- package/src/match/fuzzy.ts +98 -11
- package/src/protocol/evi1.ts +174 -29
- package/src/protocol/prompt.ts +42 -9
- package/src/report.ts +53 -21
- package/src/source/html.ts +3 -0
- package/src/types.ts +20 -2
package/CITATION.cff
CHANGED
|
@@ -9,12 +9,12 @@ authors:
|
|
|
9
9
|
orcid: "https://orcid.org/0009-0009-5596-736X"
|
|
10
10
|
repository-code: "https://github.com/rickintoplace/veriquote"
|
|
11
11
|
license: MIT
|
|
12
|
-
version: 0.
|
|
12
|
+
version: 0.3.0
|
|
13
13
|
identifiers:
|
|
14
14
|
- type: doi
|
|
15
15
|
value: 10.5281/zenodo.21552379
|
|
16
16
|
description: Concept DOI, resolves to the latest version
|
|
17
|
-
date-released: "2026-09-
|
|
17
|
+
date-released: "2026-09-25"
|
|
18
18
|
keywords:
|
|
19
19
|
- large language models
|
|
20
20
|
- retrieval-augmented generation
|
package/README.md
CHANGED
|
@@ -6,14 +6,21 @@
|
|
|
6
6
|
**Make an LLM quote its sources, then check every quote twice: is it really in
|
|
7
7
|
the source, and does it support the claim?**
|
|
8
8
|
|
|
9
|
-
A `[1]` after a sentence looks like evidence, but
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
9
|
+
A `[1]` after a sentence looks like evidence, but checking it means opening the
|
|
10
|
+
source and finding the passage. VeriQuote makes the answering model attach a
|
|
11
|
+
verbatim quote to every citation and then checks each quote twice: a
|
|
12
|
+
deterministic matcher confirms that the quote is in the source, and an LLM
|
|
13
|
+
judge decides whether the quote supports the claim.
|
|
14
|
+
|
|
15
|
+
Both checks are needed. Strong models copy quotes reliably: the best open
|
|
16
|
+
models in my tests quoted verbatim in 100% of cases, and a recent study
|
|
17
|
+
measured 98% for a frontier model. But a real quote is not a supported claim:
|
|
18
|
+
in that study, only 37% of the model's claims were fully supported by their
|
|
19
|
+
quotes,[^zhang] and in my tests 71% to 97% were, depending on the model.
|
|
20
|
+
Weaker models also leave citations without any quote, and a judge asked
|
|
21
|
+
whether a quote supports a claim is not built to notice that the quote was
|
|
22
|
+
made up. VeriQuote is written in TypeScript, has no dependencies and
|
|
23
|
+
runs in Node and in the browser.
|
|
17
24
|
|
|
18
25
|
**Try it in the browser:** [rickinto.place/veriquote](https://rickinto.place/veriquote)
|
|
19
26
|
has eight worked examples, a box for your own text, and the benchmark results as
|
|
@@ -81,8 +88,8 @@ checks them. Each citation then goes through three steps:
|
|
|
81
88
|
A citation's combined score is the lower of its match score and its support
|
|
82
89
|
score. A judge error can therefore never lift a citation above what the matcher
|
|
83
90
|
found, and a failed judge call is reported as an error, never counted as
|
|
84
|
-
support.
|
|
85
|
-
|
|
91
|
+
support. The quote format is not meant to make a model more accurate; it makes
|
|
92
|
+
every claim checkable.
|
|
86
93
|
|
|
87
94
|
## Benchmarks
|
|
88
95
|
|
|
@@ -92,25 +99,26 @@ Everything is in [`bench/`](bench), including how to reproduce it.
|
|
|
92
99
|
|
|
93
100
|
<picture>
|
|
94
101
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/tango-dark.svg">
|
|
95
|
-
<img alt="The matcher blocks all 187 quotes that are not in the source and none of the 586 real quotes whose meaning was changed; the judge blocks 84% of unsupported citations
|
|
102
|
+
<img alt="The matcher blocks all 187 quotes that are not in the source and none of the 586 real quotes whose meaning was changed; the judge blocks 84% of unsupported citations but does not check whether a quote exists; together they cover both kinds of failure." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/tango-light.svg">
|
|
96
103
|
</picture>
|
|
97
104
|
|
|
98
105
|
The two checks are blind in opposite places. The matcher never reads the claim,
|
|
99
|
-
so it cannot tell whether a real quote supports it. The judge
|
|
100
|
-
|
|
101
|
-
|
|
106
|
+
so it cannot tell whether a real quote supports it. The judge is asked whether
|
|
107
|
+
the quote supports the claim, not whether the quote exists. It sees a little of
|
|
108
|
+
the source as context, but an invented quote that fits the claim can still pass
|
|
109
|
+
it: in the example above, it rates the made-up quote in `c4` as "entailed 0.90".
|
|
102
110
|
|
|
103
111
|
### Matcher
|
|
104
112
|
|
|
105
113
|
<picture>
|
|
106
114
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/matcher-dark.svg">
|
|
107
|
-
<img alt="Matcher score ranges for fifteen kinds of quote damage: faithful quotes score between 0.
|
|
115
|
+
<img alt="Matcher score ranges for fifteen kinds of quote damage: faithful quotes score between 0.85 and 1.0, quotes that are not in the source between 0.14 and 0.40, and quotes whose meaning was changed score high because they are still near-verbatim." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/matcher-light.svg">
|
|
108
116
|
</picture>
|
|
109
117
|
|
|
110
118
|
The matcher benchmark needs no API key and no labels, because the right answer
|
|
111
119
|
is known by construction: 2,061 quotes built from five Wikipedia articles, then
|
|
112
120
|
copied faithfully, reformatted, altered or replaced in fifteen different ways. `npm run bench:matcher` reproduces it in
|
|
113
|
-
about two seconds. Faithfully copied quotes never score below 0.
|
|
121
|
+
about two seconds. Faithfully copied quotes never score below 0.851, quotes
|
|
114
122
|
that are not in the source never above 0.396, and the default threshold of 0.4
|
|
115
123
|
sits in that gap. The amber rows are there on purpose: a quote whose meaning was
|
|
116
124
|
changed is still near-verbatim, and catching it is the judge's job.
|
|
@@ -124,7 +132,7 @@ fails every citation below 0.5 and leaves the rest to the judge.
|
|
|
124
132
|
|
|
125
133
|
<picture>
|
|
126
134
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/judges-dark.svg">
|
|
127
|
-
<img alt="
|
|
135
|
+
<img alt="Open judge models, asked as chat judges or read through their logprobs, and one closed decision model against the human labels of ALCE: they catch between 75% and 89.3% of unsupported citations, and agree with the annotators on 74.0% to 81.3% of pairs, around the 77.6% of the TRUE NLI model." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/judges-light.svg">
|
|
128
136
|
</picture>
|
|
129
137
|
|
|
130
138
|
The judge is measured against the human annotators of ALCE[^alce], with the
|
|
@@ -138,36 +146,70 @@ reaches Cohen's κ 0.53 on the three-way labels (full, partial or no support).
|
|
|
138
146
|
For comparison, ALCE reports κ 0.525 between its automatic metric and the
|
|
139
147
|
annotators on citation precision.
|
|
140
148
|
|
|
149
|
+
`jev-1.13` is a different kind of judge: a closed decision model that returns
|
|
150
|
+
a probability per class instead of text, so it gives no reasons. Through
|
|
151
|
+
OpenRouter it judged the same 240 pairs in 6 seconds for under one cent, and
|
|
152
|
+
let fewer unsupported citations through than any chat judge. On all 2,896 ALCE pairs it
|
|
153
|
+
catches 89.7% (κ 0.530), for $0.09. It is an optional backend, not part of
|
|
154
|
+
VeriQuote ([`bench/lib/decisions-judge.mjs`](bench/lib/decisions-judge.mjs)),
|
|
155
|
+
and ALCE has been public since 2023, so training contamination cannot be ruled
|
|
156
|
+
out for it or for any other judge here.
|
|
157
|
+
|
|
158
|
+
The same idea works with open models. With `--logprobs`, a chat model answers
|
|
159
|
+
with a single letter (fully, partially or not supported), and the judge reads
|
|
160
|
+
the three probabilities from that token's logprobs: no reasoning, one token per
|
|
161
|
+
citation. Read this way, `qwen3.5-397b-a17b` caught 89.3% of the unsupported
|
|
162
|
+
citations (κ 0.526) and `qwen3.6-35b-a3b` 85.7% (κ 0.507), more than either
|
|
163
|
+
model reached as a chat judge with reasoning on. One caveat: this judge's short
|
|
164
|
+
prompt is not the chat judge's prompt, so part of the difference may come from
|
|
165
|
+
the wording rather than the readout. It needs an endpoint that returns logprobs
|
|
166
|
+
and continues a prefilled answer, as vLLM and llama.cpp do
|
|
167
|
+
([`bench/lib/logprobs-judge.mjs`](bench/lib/logprobs-judge.mjs)).
|
|
168
|
+
|
|
141
169
|
Before trusting any of this, look at the left panel: even the best judge lets
|
|
142
|
-
one unsupported citation in
|
|
170
|
+
about one unsupported citation in nine through as fully supported. No judge should be
|
|
143
171
|
the only check. With only 56 unsupported pairs per run the intervals are wide,
|
|
144
|
-
so neighbouring models are not really separated.
|
|
145
|
-
|
|
146
|
-
|
|
172
|
+
so neighbouring models are not really separated. With reasoning switched off,
|
|
173
|
+
all three hybrid models caught less and agreed less while answering five to ten
|
|
174
|
+
times faster; each difference is within the sampling error, but all three point
|
|
175
|
+
the same way. These pairs come from ALCE, where the judge gets a whole passage
|
|
176
|
+
rather than a quote, which makes its task harder than in normal use.
|
|
147
177
|
|
|
148
178
|
### Answering models
|
|
149
179
|
|
|
150
180
|
<picture>
|
|
151
181
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/protocol-dark.svg">
|
|
152
|
-
<img alt="
|
|
182
|
+
<img alt="Four open answering models with the current citation instructions: all quote verbatim in 100% of cases; fully supported citations range from 71.4% to 97.4% depending on model and judge, highest for qwen3.8-27b and glm-5.3-flash, lowest for qwen3.6-35b-a3b." src="https://raw.githubusercontent.com/rickintoplace/veriquote/main/bench/figures/protocol-light.svg">
|
|
153
183
|
</picture>
|
|
154
184
|
|
|
155
|
-
|
|
156
|
-
system prompt
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
185
|
+
Four good open models answered 18 questions twice with the citation
|
|
186
|
+
instructions in their system prompt. Every quote was matched against its
|
|
187
|
+
source, and two judges rated whether the quotes cover every detail of the claim
|
|
188
|
+
(*fully supported*). All quotes were verbatim. The share of fully supported
|
|
189
|
+
citations depends more on the model than on the judge.
|
|
190
|
+
|
|
191
|
+
The instructions were revised on the strength of this benchmark: they now ask
|
|
192
|
+
that the quotes of a sentence contain every number, population, condition and
|
|
193
|
+
hedge it states, and otherwise drop the detail. On the same tasks and models,
|
|
194
|
+
that raised full support by 7.6 points (95% interval 1.5–14.2) with the chat
|
|
195
|
+
judge and by 14.1 points (7.6–21.5) with the decision model, while the answers
|
|
196
|
+
stated as many cited claims as before. Writing the evidence before the answer
|
|
197
|
+
(`evidenceFirst`) added nothing reliable (+3.6 and −0.3 points, both intervals
|
|
198
|
+
spanning zero), made one model drop quotes, and hides the answer until the
|
|
199
|
+
evidence is written, so it is not the default. On the three questions the
|
|
164
200
|
sources cannot answer, no model attached a citation without a real quote.
|
|
201
|
+
Every raw answer is in `bench/results/`, and
|
|
202
|
+
[`bench/README.md`](bench/README.md#2-protocol-compliance) has the full tables,
|
|
203
|
+
including weaker models under the previous instructions.
|
|
165
204
|
|
|
166
205
|
<details>
|
|
167
206
|
<summary>The numbers behind the figures</summary>
|
|
168
207
|
|
|
169
208
|
| judge model | caught ↑ | agreement | Cohen's κ |
|
|
170
209
|
| --- | ---: | ---: | ---: |
|
|
210
|
+
| `qwen3.5-397b-a17b` (logprobs) | 89.3% | 81.3% | 0.526 |
|
|
211
|
+
| `jev-1.13` (decision model) | 87.5% | 79.6% | 0.497 |
|
|
212
|
+
| `qwen3.6-35b-a3b` (logprobs) | 85.7% | 80.0% | 0.507 |
|
|
171
213
|
| `glm-5.3-flash` | 83.9% | 80.3% | 0.529 |
|
|
172
214
|
| `deepseek-v4-flash` | 81.8% | 78.7% | 0.435 |
|
|
173
215
|
| `qwen3.6-35b-a3b` | 81.8% | 78.1% | 0.454 |
|
|
@@ -184,16 +226,15 @@ sources cannot answer, no model attached a citation without a real quote.
|
|
|
184
226
|
| near-verbatim, meaning changed | 586 | 0.932 | 100.0% |
|
|
185
227
|
| not in the source | 187 | 0.249 | 0.0% |
|
|
186
228
|
|
|
187
|
-
| answering model |
|
|
229
|
+
| answering model | verbatim | complete | fully supported (deepseek-v4-flash) | fully supported (jev-1.13) |
|
|
188
230
|
| --- | ---: | ---: | ---: | ---: |
|
|
189
|
-
| qwen3.
|
|
190
|
-
|
|
|
191
|
-
| deepseek-v4-flash | 100.0% | 100.0% |
|
|
192
|
-
|
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
| llama-3.1-8b-instruct | 66.7% | 43.3% | 50.3% | 20.0% |
|
|
231
|
+
| qwen3.8-27b | 100.0% | 100.0% | 97.4% | 97.4% |
|
|
232
|
+
| glm-5.3-flash | 100.0% | 100.0% | 95.1% | 93.0% |
|
|
233
|
+
| deepseek-v4-flash | 100.0% | 100.0% | 88.0% | 90.0% |
|
|
234
|
+
| qwen3.6-35b-a3b | 100.0% | 96.7% | 76.9% | 71.4% |
|
|
235
|
+
|
|
236
|
+
`deepseek-v4-flash` also judges its own answers here; the decision model is the
|
|
237
|
+
independent check, and it ranks the models the same way.
|
|
197
238
|
|
|
198
239
|
The figures are rendered from `bench/results/*.json` by
|
|
199
240
|
[`bench/figures/render.mjs`](bench/figures/render.mjs).
|
|
@@ -203,20 +244,29 @@ The figures are rendered from `bench/results/*.json` by
|
|
|
203
244
|
|
|
204
245
|
An agent that reads sources and writes conclusions is exactly the case
|
|
205
246
|
VeriQuote was built for. [`verify-citations`](integrations/verify-citations) is
|
|
206
|
-
an [Agent Skill](integrations/verify-citations/SKILL.md)
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
247
|
+
an [Agent Skill](integrations/verify-citations/SKILL.md) for Claude Code and
|
|
248
|
+
other hosts that read Agent Skills. With it, a research answer goes like this:
|
|
249
|
+
|
|
250
|
+
1. The agent reads each source with `veriquote source <url>`, which prints the
|
|
251
|
+
exact text the check will use. Web-fetch tools often summarize or reformat
|
|
252
|
+
pages, so quotes copied from them may not match.
|
|
253
|
+
2. It writes the answer in the checkable format and runs
|
|
254
|
+
`veriquote check answer.md --source <url> … --json`. The CLI fetches every
|
|
255
|
+
URL itself, so a truncated or misremembered copy of a page cannot pass.
|
|
256
|
+
3. The exit code says what to do next:
|
|
211
257
|
|
|
212
258
|
```
|
|
213
|
-
exit 0 verdict "pass"
|
|
214
|
-
exit 2 verdict "revise"
|
|
215
|
-
exit 1
|
|
259
|
+
exit 0 verdict "pass" present cleanAnswer, the sources and one line on the check
|
|
260
|
+
exit 2 verdict "revise" follow instructionsForModel, then check again
|
|
261
|
+
exit 1 verdict "unverified" the judge failed; check again before claiming anything
|
|
262
|
+
exit 1 (no verdict) bad input or unreachable source
|
|
216
263
|
```
|
|
217
264
|
|
|
218
|
-
The
|
|
219
|
-
|
|
265
|
+
The user sees only the final answer, with `[n]` markers, the list of sources
|
|
266
|
+
and a line such as "Checked with VeriQuote: all 6 quotes were found in the
|
|
267
|
+
sources and judged to support their claims." The quote appendix, the JSON and
|
|
268
|
+
any revision rounds stay with the agent. The CLI works in anything that can run
|
|
269
|
+
a shell command.
|
|
220
270
|
|
|
221
271
|
## Use as a library
|
|
222
272
|
|
|
@@ -270,6 +320,7 @@ single source.
|
|
|
270
320
|
| `verifyAnswer(options)` | The full pipeline: parse, match, judge, report. |
|
|
271
321
|
| `parseAnswer(answer)` | Claims, quotes and protocol warnings, without verifying anything. |
|
|
272
322
|
| `parseEvi1Appendix` / `stripEvi1Appendix` / `serializeEvi1Appendix` | Low-level handling of the appendix. |
|
|
323
|
+
| `stripForDisplay(text)` | The answer as a reader sees it, also mid-stream: no appendix, no `{cX}` markers, no half-streamed `EVI1` or `{c`. |
|
|
273
324
|
| `matchQuoteAgainstSource(quote, source, options?)` | The deterministic matcher on its own. |
|
|
274
325
|
| `ChatCompletionsJudge` | Judge for any OpenAI-compatible API. |
|
|
275
326
|
| `EntailmentJudge` (interface) | Bring your own judge, such as a local NLI model. |
|
|
@@ -354,4 +405,4 @@ are also in [`CITATION.cff`](CITATION.cff).
|
|
|
354
405
|
|
|
355
406
|
[^zhang]: Zhang, J., Chen, Y., Commodore-Mensah, Y., & Oberst, M. (2026). *Verifiable by construction: Claim-level evaluation of verbatim citation in clinical question answering* (Version 2) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2609.15964
|
|
356
407
|
|
|
357
|
-
[^veriquote]: Heilmann, E. (2026). *VeriQuote: Deterministic and semantic verification of quote-grounded LLM citations* (Version 0.
|
|
408
|
+
[^veriquote]: Heilmann, E. (2026). *VeriQuote: Deterministic and semantic verification of quote-grounded LLM citations* (Version 0.3.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.21552379
|
package/dist/cli/main.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"main.d.ts","sourceRoot":"","sources":["../../src/cli/main.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAsC,MAAM,aAAa,CAAC;AAOvF,MAAM,WAAW,KAAK;IACpB,IAAI,EAAE,MAAM,EAAE,CAAC;IACf,GAAG,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC;IACxC,MAAM,EAAE,CAAC,CAAC,EAAE,MAAM,KAAK,IAAI,CAAC;IAC5B,MAAM,EAAE,CAAC,CAAC,EAAE,MAAM,KAAK,IAAI,CAAC;IAC5B,QAAQ,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,OAAO,CAAC,MAAM,CAAC,CAAC;IAC5C,SAAS,EAAE,MAAM,OAAO,CAAC,MAAM,CAAC,CAAC;IACjC,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;IAChC,OAAO,EAAE,MAAM,CAAC;IAChB,uCAAuC;IACvC,KAAK,EAAE,OAAO,CAAC;IACf,oDAAoD;IACpD,KAAK,CAAC,EAAE,eAAe,CAAC;CACzB;AAED,eAAO,MAAM,IAAI;;;;CAA4C,CAAC;
|
|
1
|
+
{"version":3,"file":"main.d.ts","sourceRoot":"","sources":["../../src/cli/main.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAsC,MAAM,aAAa,CAAC;AAOvF,MAAM,WAAW,KAAK;IACpB,IAAI,EAAE,MAAM,EAAE,CAAC;IACf,GAAG,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC;IACxC,MAAM,EAAE,CAAC,CAAC,EAAE,MAAM,KAAK,IAAI,CAAC;IAC5B,MAAM,EAAE,CAAC,CAAC,EAAE,MAAM,KAAK,IAAI,CAAC;IAC5B,QAAQ,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,OAAO,CAAC,MAAM,CAAC,CAAC;IAC5C,SAAS,EAAE,MAAM,OAAO,CAAC,MAAM,CAAC,CAAC;IACjC,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;IAChC,OAAO,EAAE,MAAM,CAAC;IAChB,uCAAuC;IACvC,KAAK,EAAE,OAAO,CAAC;IACf,oDAAoD;IACpD,KAAK,CAAC,EAAE,eAAe,CAAC;CACzB;AAED,eAAO,MAAM,IAAI;;;;CAA4C,CAAC;AAmG9D,wBAAsB,IAAI,CAAC,EAAE,EAAE,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,CAsCrD"}
|
package/dist/cli/main.js
CHANGED
|
@@ -12,6 +12,7 @@ const HELP = `veriquote — check that an LLM answer's quotes exist in its sourc
|
|
|
12
12
|
|
|
13
13
|
Usage
|
|
14
14
|
veriquote prompt print the citation instructions for the answering model
|
|
15
|
+
veriquote source <url|file> print the text a source is checked against (quote from this)
|
|
15
16
|
veriquote check <answer> --source <url|file> [--source …]
|
|
16
17
|
veriquote check --job <file|-> JSON job: {"answer": "...", "sources": [{"text": "..."}]}
|
|
17
18
|
|
|
@@ -34,7 +35,7 @@ Judge (semantic support check), from the environment:
|
|
|
34
35
|
VERIQUOTE_JUDGE_MODEL required when a key is set
|
|
35
36
|
VERIQUOTE_JUDGE_BASE_URL default https://openrouter.ai/api/v1
|
|
36
37
|
|
|
37
|
-
Exit status: 0 pass · 2 revise · 1 error`;
|
|
38
|
+
Exit status: 0 pass · 2 revise · 1 error, or the judge failed (unverified)`;
|
|
38
39
|
class UsageError extends Error {
|
|
39
40
|
}
|
|
40
41
|
function parseArgs(argv) {
|
|
@@ -117,6 +118,13 @@ export async function main(io) {
|
|
|
117
118
|
return EXIT.pass;
|
|
118
119
|
case 'check':
|
|
119
120
|
return await check(args, io);
|
|
121
|
+
case 'source': {
|
|
122
|
+
if (args.positional.length !== 1)
|
|
123
|
+
throw new UsageError('source needs exactly one URL or file');
|
|
124
|
+
const src = await loadSource(args.positional[0], io);
|
|
125
|
+
io.stdout(`${src.text}\n`);
|
|
126
|
+
return EXIT.pass;
|
|
127
|
+
}
|
|
120
128
|
default:
|
|
121
129
|
throw new UsageError(`unknown command "${args.command}"`);
|
|
122
130
|
}
|
|
@@ -162,9 +170,21 @@ async function check(args, io) {
|
|
|
162
170
|
io.stdout(`${JSON.stringify({
|
|
163
171
|
verdict: gate.verdict,
|
|
164
172
|
judge: judgeSetup.model ?? null,
|
|
173
|
+
// The answer as the user should see it: [n] markers kept, {cX} markers and the appendix removed.
|
|
174
|
+
cleanAnswer: report.cleanText,
|
|
165
175
|
summary: report.summary,
|
|
166
176
|
problems: gate.problems,
|
|
167
177
|
uncited: gate.uncited,
|
|
178
|
+
unjudged: gate.unjudged,
|
|
179
|
+
citations: report.citations.map((c) => ({
|
|
180
|
+
claimId: c.claimId,
|
|
181
|
+
sourceIndex: c.sourceIndex,
|
|
182
|
+
match: { method: c.textMatch.method, score: round2(c.textMatch.score) },
|
|
183
|
+
judge: c.entailment
|
|
184
|
+
? { class: c.entailment.class, support: c.entailment.confidence === null ? null : round2(c.entailment.confidence) }
|
|
185
|
+
: null,
|
|
186
|
+
score: c.score === null ? null : round2(c.score),
|
|
187
|
+
})),
|
|
168
188
|
warnings: report.warnings,
|
|
169
189
|
sources: sources.map((s, i) => ({ index: i + 1, url: s.url, title: s.title, chars: s.text.length })),
|
|
170
190
|
instructionsForModel: gate.instructionsForModel,
|
|
@@ -173,8 +193,9 @@ async function check(args, io) {
|
|
|
173
193
|
else {
|
|
174
194
|
io.stdout(renderHuman(report, gate, sources, judgeSetup.model, io.color, args.minScore ?? 0.5));
|
|
175
195
|
}
|
|
176
|
-
return gate.verdict === 'pass' ? EXIT.pass : EXIT.revise;
|
|
196
|
+
return gate.verdict === 'pass' ? EXIT.pass : gate.verdict === 'revise' ? EXIT.revise : EXIT.error;
|
|
177
197
|
}
|
|
198
|
+
const round2 = (x) => Math.round(x * 100) / 100;
|
|
178
199
|
async function loadSource(ref, io) {
|
|
179
200
|
if (/^https?:\/\//i.test(ref)) {
|
|
180
201
|
return fetchSource(ref, {
|
|
@@ -206,6 +227,12 @@ function judgeFromEnv(args, io) {
|
|
|
206
227
|
};
|
|
207
228
|
}
|
|
208
229
|
// ------------------------------------------------------------------ output
|
|
230
|
+
const MATCH_LABEL = {
|
|
231
|
+
exact: 'verbatim',
|
|
232
|
+
normalized: 'verbatim',
|
|
233
|
+
elided: 'verbatim with omission',
|
|
234
|
+
fuzzy: 'fuzzy',
|
|
235
|
+
};
|
|
209
236
|
function renderHuman(report, gate, sources, model, color, minScore) {
|
|
210
237
|
const c = (code, s) => (color ? `\x1b[${code}m${s}\x1b[0m` : s);
|
|
211
238
|
const green = (s) => c('32', s);
|
|
@@ -220,7 +247,7 @@ function renderHuman(report, gate, sources, model, color, minScore) {
|
|
|
220
247
|
const mark = bad ? red('✗') : cit.score === null ? yellow('?') : green('✓');
|
|
221
248
|
const match = cit.textMatch.method === 'not_found'
|
|
222
249
|
? red(`quote not in source (best ${cit.textMatch.score.toFixed(2)})`)
|
|
223
|
-
: `${cit.textMatch.method
|
|
250
|
+
: `${MATCH_LABEL[cit.textMatch.method]} ${cit.textMatch.score.toFixed(2)}`;
|
|
224
251
|
const judged = cit.entailment
|
|
225
252
|
? cit.entailment.class === 'error'
|
|
226
253
|
? yellow('judge error')
|
|
@@ -228,8 +255,13 @@ function renderHuman(report, gate, sources, model, color, minScore) {
|
|
|
228
255
|
: '';
|
|
229
256
|
out.push(`${mark} ${cit.claimId} [${cit.sourceIndex}] ${[match, judged].filter(Boolean).join(' · ')}`);
|
|
230
257
|
out.push(` ${truncate(cit.claimText || '(no claim text)', 160)}`);
|
|
258
|
+
const hides = gate.problems.find((p) => p.claimId === cit.claimId && p.sourceIndex === cit.sourceIndex)?.type === 'ellipsis_hides_qualifier';
|
|
231
259
|
if (bad && cit.textMatch.method === 'not_found')
|
|
232
260
|
out.push(dim(' the quoted text does not occur in the source'));
|
|
261
|
+
else if (hides) {
|
|
262
|
+
const cues = (cit.textMatch.omittedCues ?? []).map((c) => `"${c}"`).join(', ');
|
|
263
|
+
out.push(dim(` the ellipsis leaves out ${cues}; run with a judge to check the full passage`));
|
|
264
|
+
}
|
|
233
265
|
else if (bad && cit.entailment?.reasons.length)
|
|
234
266
|
out.push(dim(` ${cit.entailment.reasons.map((r) => r.trim().replace(/[.;]+$/, '')).join('; ')}`));
|
|
235
267
|
else if (bad && cit.score !== null && cit.textMatch.method !== 'not_found') {
|
|
@@ -242,14 +274,19 @@ function renderHuman(report, gate, sources, model, color, minScore) {
|
|
|
242
274
|
out.push(dim(`warning: ${w}`));
|
|
243
275
|
out.push('');
|
|
244
276
|
if (gate.verdict === 'pass') {
|
|
245
|
-
out.push(green(report.citations.length ? 'PASS' : 'PASS
|
|
277
|
+
out.push(green(report.citations.length ? 'PASS' : 'PASS, but the answer cites nothing'));
|
|
278
|
+
}
|
|
279
|
+
else if (gate.verdict === 'unverified') {
|
|
280
|
+
out.push(yellow(`UNVERIFIED: the judge failed on ${gate.unjudged.length} citation(s), so their support was not checked. Run the check again.`));
|
|
246
281
|
}
|
|
247
282
|
else {
|
|
248
283
|
const parts = [
|
|
249
284
|
gate.problems.length && `${gate.problems.length} failed citation(s)`,
|
|
250
285
|
gate.uncited.length && `${gate.uncited.length} uncited sentence(s)`,
|
|
251
286
|
].filter(Boolean);
|
|
252
|
-
out.push(red(`REVISE
|
|
287
|
+
out.push(red(`REVISE: ${parts.join(', ')}`));
|
|
288
|
+
if (gate.unjudged.length)
|
|
289
|
+
out.push(yellow(`The judge also failed on ${gate.unjudged.length} citation(s).`));
|
|
253
290
|
}
|
|
254
291
|
if (!model)
|
|
255
292
|
out.push(dim('Only checked that quotes exist in their sources. Set VERIQUOTE_JUDGE_API_KEY to check support.'));
|
package/dist/cli/main.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"main.js","sourceRoot":"","sources":["../../src/cli/main.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAGH,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,UAAU,EAAmB,MAAM,YAAY,CAAC;AACzD,OAAO,EAAE,yBAAyB,EAAE,MAAM,uBAAuB,CAAC;AAClE,OAAO,EAAE,oBAAoB,EAAE,MAAM,wBAAwB,CAAC;AAC9D,OAAO,EAAE,WAAW,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AAiBjE,MAAM,CAAC,MAAM,IAAI,GAAG,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAW,CAAC;AAE9D,MAAM,IAAI,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;yCA0B4B,CAAC;AAgB1C,MAAM,UAAW,SAAQ,KAAK;CAAG;AAEjC,SAAS,SAAS,CAAC,IAAc;IAC/B,MAAM,CAAC,GAAS,EAAE,UAAU,EAAE,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;IACvG,MAAM,KAAK,GAAG,CAAC,CAAS,EAAE,IAAY,EAAU,EAAE;QAChD,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QACtB,IAAI,CAAC,KAAK,SAAS;YAAE,MAAM,IAAI,UAAU,CAAC,GAAG,IAAI,gBAAgB,CAAC,CAAC;QACnE,OAAO,CAAC,CAAC;IACX,CAAC,CAAC;IACF,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QACpB,QAAQ,GAAG,EAAE,CAAC;YACZ,KAAK,IAAI,CAAC;YACV,KAAK,UAAU;gBACb,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC,CAAC;gBAChC,MAAM;YACR,KAAK,OAAO;gBACV,CAAC,CAAC,GAAG,GAAG,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;gBACxB,MAAM;YACR,KAAK,QAAQ;gBACX,CAAC,CAAC,IAAI,GAAG,IAAI,CAAC;gBACd,MAAM;YACR,KAAK,YAAY;gBACf,CAAC,CAAC,KAAK,GAAG,KAAK,CAAC;gBAChB,MAAM;YACR,KAAK,aAAa,CAAC,CAAC,CAAC;gBACnB,MAAM,CAAC,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC,CAAC;gBAClC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;oBAAE,MAAM,IAAI,UAAU,CAAC,qCAAqC,CAAC,CAAC;gBACrF,CAAC,CAAC,QAAQ,GAAG,CAAC,CAAC;gBACf,MAAM;YACR,CAAC;YACD,KAAK,eAAe;gBAClB,CAAC,CAAC,UAAU,GAAG,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;gBAC/B,MAAM;YACR,KAAK,kBAAkB;gBACrB,CAAC,CAAC,YAAY,GAAG,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;gBACjC,MAAM;YACR,KAAK,IAAI,CAAC;YACV,KAAK,QAAQ;gBACX,CAAC,CAAC,IAAI,GAAG,IAAI,CAAC;gBACd,MAAM;YACR,KAAK,IAAI,CAAC;YACV,KAAK,WAAW;gBACd,CAAC,CAAC,OAAO,GAAG,IAAI,CAAC;gBACjB,MAAM;YACR;gBACE,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,GAAG,KAAK,GAAG;oBAAE,MAAM,IAAI,UAAU,CAAC,kBAAkB,GAAG,EAAE,CAAC,CAAC;gBACtF,IAAI,CAAC,CAAC,OAAO,KAAK,SAAS;oBAAE,CAAC,CAAC,OAAO,GAAG,GAAG,CAAC;;oBACxC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAChC,CAAC;IACH,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,IAAI,CAAC,EAAS;IAClC,IAAI,IAAU,CAAC;IACf,IAAI,CAAC;QACH,IAAI,GAAG,SAAS,CAAC,EAAE,CAAC,IAAI,CAAC,CAAC;IAC5B,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,EAAE,CAAC,MAAM,CAAC,cAAe,GAAa,CAAC,OAAO,uCAAuC,CAAC,CAAC;QACvF,OAAO,IAAI,CAAC,KAAK,CAAC;IACpB,CAAC;IACD,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;QACjB,EAAE,CAAC,MAAM,CAAC,GAAG,EAAE,CAAC,OAAO,IAAI,CAAC,CAAC;QAC7B,OAAO,IAAI,CAAC,IAAI,CAAC;IACnB,CAAC;IACD,IAAI,IAAI,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;QAC/B,EAAE,CAAC,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,CAAC;QACvB,OAAO,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC;IAC5C,CAAC;IAED,IAAI,CAAC;QACH,QAAQ,IAAI,CAAC,OAAO,EAAE,CAAC;YACrB,KAAK,QAAQ;gBACX,EAAE,CAAC,MAAM,CAAC,GAAG,yBAAyB,EAAE,IAAI,CAAC,CAAC;gBAC9C,OAAO,IAAI,CAAC,IAAI,CAAC;YACnB,KAAK,OAAO;gBACV,OAAO,MAAM,KAAK,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;YAC/B;gBACE,MAAM,IAAI,UAAU,CAAC,oBAAoB,IAAI,CAAC,OAAO,GAAG,CAAC,CAAC;QAC9D,CAAC;IACH,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,GAAG,YAAY,UAAU,CAAC,CAAC,CAAC,qCAAqC,CAAC,CAAC,CAAC,EAAE,CAAC;QACpF,EAAE,CAAC,MAAM,CAAC,cAAe,GAAa,CAAC,OAAO,GAAG,IAAI,IAAI,CAAC,CAAC;QAC3D,OAAO,IAAI,CAAC,KAAK,CAAC;IACpB,CAAC;AACH,CAAC;AAED,KAAK,UAAU,KAAK,CAAC,IAAU,EAAE,EAAS;IACxC,MAAM,IAAI,GAAG,CAAC,IAAY,EAAE,EAAE,CAAC,CAAC,IAAI,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC;IACnF,IAAI,MAAc,CAAC;IACnB,IAAI,OAAyB,CAAC;IAE9B,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;QACb,IAAI,IAAI,CAAC,UAAU,CAAC,MAAM,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YAClD,MAAM,IAAI,UAAU,CAAC,+CAA+C,CAAC,CAAC;QACxE,CAAC;QACD,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAA4C,CAAC;QACxF,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC;QAClC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC;YAAE,MAAM,IAAI,UAAU,CAAC,8BAA8B,CAAC,CAAC;QACtF,OAAO,GAAG,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;YACjC,IAAI,OAAO,CAAC,EAAE,IAAI,KAAK,QAAQ;gBAAE,MAAM,IAAI,UAAU,CAAC,eAAe,CAAC,yBAAyB,CAAC,CAAC;YACjG,OAAO,CAAmB,CAAC;QAC7B,CAAC,CAAC,CAAC;IACL,CAAC;SAAM,CAAC;QACN,IAAI,IAAI,CAAC,UAAU,CAAC,MAAM,KAAK,CAAC;YAAE,MAAM,IAAI,UAAU,CAAC,4CAA4C,CAAC,CAAC;QACrG,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM;YAAE,MAAM,IAAI,UAAU,CAAC,+CAA+C,CAAC,CAAC;QAChG,MAAM,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC;QACxC,OAAO,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC;IAC1E,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE;QAAE,MAAM,IAAI,UAAU,CAAC,qBAAqB,CAAC,CAAC;IAEhE,MAAM,UAAU,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,YAAY,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,SAAS,EAAE,KAAK,EAAE,SAAS,EAAE,CAAC;IAChG,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,KAAK,EAAE,UAAU,CAAC,KAAK,EAAE,CAAC,CAAC;IAChF,MAAM,IAAI,GAAG,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,IAAI,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;IAExG,IAAI,IAAI,CAAC,IAAI,EAAE,CAAC;QACd,EAAE,CAAC,MAAM,CACP,GAAG,IAAI,CAAC,SAAS,CACf;YACE,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,KAAK,EAAE,UAAU,CAAC,KAAK,IAAI,IAAI;YAC/B,OAAO,EAAE,MAAM,CAAC,OAAO;YACvB,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,QAAQ,EAAE,MAAM,CAAC,QAAQ;YACzB,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,GAAG,CAAC,EAAE,GAAG,EAAE,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC;YACpG,oBAAoB,EAAE,IAAI,CAAC,oBAAoB;SAChD,EACD,IAAI,EACJ,CAAC,CACF,IAAI,CACN,CAAC;IACJ,CAAC;SAAM,CAAC;QACN,EAAE,CAAC,MAAM,CAAC,WAAW,CAAC,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,UAAU,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,EAAE,IAAI,CAAC,QAAQ,IAAI,GAAG,CAAC,CAAC,CAAC;IAClG,CAAC;IACD,OAAO,IAAI,CAAC,OAAO,KAAK,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;AAC3D,CAAC;AAED,KAAK,UAAU,UAAU,CAAC,GAAW,EAAE,EAAS;IAC9C,IAAI,eAAe,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QAC9B,OAAO,WAAW,CAAC,GAAG,EAAE;YACtB,KAAK,EAAE,EAAE,CAAC,KAAK;YACf,SAAS,EAAE,aAAa,EAAE,CAAC,OAAO,gDAAgD;SACnF,CAAC,CAAC;IACL,CAAC;IACD,IAAI,SAAS,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACxB,MAAM,IAAI,KAAK,CAAC,GAAG,GAAG,2FAA2F,CAAC,CAAC;IACrH,CAAC;IACD,MAAM,IAAI,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;IACpC,MAAM,MAAM,GAAG,cAAc,CAAC,IAAI,EAAE,aAAa,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC;IAC1F,OAAO,EAAE,GAAG,MAAM,EAAE,EAAE,EAAE,GAAG,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,GAAG,EAAE,CAAC;AAC5D,CAAC;AAED,SAAS,YAAY,CAAC,IAAU,EAAE,EAAS;IACzC,IAAI,EAAE,CAAC,KAAK;QAAE,OAAO,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,EAAE,KAAK,EAAE,IAAI,CAAC,UAAU,IAAI,QAAQ,EAAE,CAAC;IAC7E,MAAM,MAAM,GAAG,EAAE,CAAC,GAAG,CAAC,uBAAuB,IAAI,EAAE,CAAC,GAAG,CAAC,kBAAkB,CAAC;IAC3E,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IACvB,MAAM,KAAK,GAAG,IAAI,CAAC,UAAU,IAAI,EAAE,CAAC,GAAG,CAAC,qBAAqB,CAAC;IAC9D,IAAI,CAAC,KAAK,EAAE,CAAC;QACX,MAAM,IAAI,UAAU,CAAC,oFAAoF,CAAC,CAAC;IAC7G,CAAC;IACD,MAAM,OAAO,GACX,IAAI,CAAC,YAAY,IAAI,EAAE,CAAC,GAAG,CAAC,wBAAwB,IAAI,EAAE,CAAC,GAAG,CAAC,kBAAkB,IAAI,8BAA8B,CAAC;IACtH,OAAO;QACL,KAAK;QACL,KAAK,EAAE,IAAI,oBAAoB,CAAC,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,EAAE,CAAC;KAChG,CAAC;AACJ,CAAC;AAED,4EAA4E;AAE5E,SAAS,WAAW,CAClB,MAA0B,EAC1B,IAAgB,EAChB,OAAyB,EACzB,KAAyB,EACzB,KAAc,EACd,QAAgB;IAEhB,MAAM,CAAC,GAAG,CAAC,IAAY,EAAE,CAAS,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,QAAQ,IAAI,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAChF,MAAM,KAAK,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IACxC,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IACtC,MAAM,MAAM,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IACzC,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC;IACrC,MAAM,GAAG,GAAa,EAAE,CAAC;IAEzB,GAAG,CAAC,IAAI,CACN,GAAG,CAAC,GAAG,MAAM,CAAC,SAAS,CAAC,MAAM,kBAAkB,OAAO,CAAC,MAAM,uBAAuB,KAAK,IAAI,mBAAmB,EAAE,CAAC,EACpH,EAAE,CACH,CAAC;IAEF,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,OAAO,IAAI,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;IAClF,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,SAAS,EAAE,CAAC;QACnC,MAAM,GAAG,GAAG,MAAM,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC,OAAO,IAAI,GAAG,CAAC,WAAW,EAAE,CAAC,CAAC;QAC5D,MAAM,IAAI,GAAG,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,KAAK,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;QAC5E,MAAM,KAAK,GACT,GAAG,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW;YAClC,CAAC,CAAC,GAAG,CAAC,6BAA6B,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC;YACrE,CAAC,CAAC,GAAG,GAAG,CAAC,SAAS,CAAC,MAAM,KAAK,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,IAAI,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;QACrG,MAAM,MAAM,GAAG,GAAG,CAAC,UAAU;YAC3B,CAAC,CAAC,GAAG,CAAC,UAAU,CAAC,KAAK,KAAK,OAAO;gBAChC,CAAC,CAAC,MAAM,CAAC,aAAa,CAAC;gBACvB,CAAC,CAAC,GAAG,GAAG,CAAC,UAAU,CAAC,KAAK,IAAI,GAAG,CAAC,UAAU,CAAC,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,EAAE;YACnF,CAAC,CAAC,EAAE,CAAC;QACP,GAAG,CAAC,IAAI,CAAC,GAAG,IAAI,IAAI,GAAG,CAAC,OAAO,KAAK,GAAG,CAAC,WAAW,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACxG,GAAG,CAAC,IAAI,CAAC,OAAO,QAAQ,CAAC,GAAG,CAAC,SAAS,IAAI,iBAAiB,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;QACrE,IAAI,GAAG,IAAI,GAAG,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW;YAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,kDAAkD,CAAC,CAAC,CAAC;aAC9G,IAAI,GAAG,IAAI,GAAG,CAAC,UAAU,EAAE,OAAO,CAAC,MAAM;YAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,OAAO,GAAG,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,OAAO,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;aAChJ,IAAI,GAAG,IAAI,GAAG,CAAC,KAAK,KAAK,IAAI,IAAI,GAAG,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW,EAAE,CAAC;YAC3E,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,sBAAsB,GAAG,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,yBAAyB,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAC1G,CAAC;IACH,CAAC;IACD,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,OAAO;QAAE,GAAG,CAAC,IAAI,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,aAAa,QAAQ,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;IACtF,KAAK,MAAM,CAAC,IAAI,MAAM,CAAC,QAAQ;QAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,YAAY,CAAC,EAAE,CAAC,CAAC,CAAC;IAEhE,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACb,IAAI,IAAI,CAAC,OAAO,KAAK,MAAM,EAAE,CAAC;QAC5B,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,qCAAqC,CAAC,CAAC,CAAC;IAC5F,CAAC;SAAM,CAAC;QACN,MAAM,KAAK,GAAG;YACZ,IAAI,CAAC,QAAQ,CAAC,MAAM,IAAI,GAAG,IAAI,CAAC,QAAQ,CAAC,MAAM,qBAAqB;YACpE,IAAI,CAAC,OAAO,CAAC,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,sBAAsB;SACpE,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QAClB,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,YAAY,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;IAChD,CAAC;IACD,IAAI,CAAC,KAAK;QAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,gGAAgG,CAAC,CAAC,CAAC;IAC5H,OAAO,GAAG,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC;AAC/B,CAAC;AAED,SAAS,QAAQ,CAAC,CAAS,EAAE,CAAS;IACpC,MAAM,IAAI,GAAG,CAAC,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IAC3C,OAAO,IAAI,CAAC,MAAM,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC;AAC9D,CAAC"}
|
|
1
|
+
{"version":3,"file":"main.js","sourceRoot":"","sources":["../../src/cli/main.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAGH,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,UAAU,EAAmB,MAAM,YAAY,CAAC;AACzD,OAAO,EAAE,yBAAyB,EAAE,MAAM,uBAAuB,CAAC;AAClE,OAAO,EAAE,oBAAoB,EAAE,MAAM,wBAAwB,CAAC;AAC9D,OAAO,EAAE,WAAW,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AAiBjE,MAAM,CAAC,MAAM,IAAI,GAAG,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAW,CAAC;AAE9D,MAAM,IAAI,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;2EA2B8D,CAAC;AAgB5E,MAAM,UAAW,SAAQ,KAAK;CAAG;AAEjC,SAAS,SAAS,CAAC,IAAc;IAC/B,MAAM,CAAC,GAAS,EAAE,UAAU,EAAE,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;IACvG,MAAM,KAAK,GAAG,CAAC,CAAS,EAAE,IAAY,EAAU,EAAE;QAChD,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QACtB,IAAI,CAAC,KAAK,SAAS;YAAE,MAAM,IAAI,UAAU,CAAC,GAAG,IAAI,gBAAgB,CAAC,CAAC;QACnE,OAAO,CAAC,CAAC;IACX,CAAC,CAAC;IACF,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QACpB,QAAQ,GAAG,EAAE,CAAC;YACZ,KAAK,IAAI,CAAC;YACV,KAAK,UAAU;gBACb,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC,CAAC;gBAChC,MAAM;YACR,KAAK,OAAO;gBACV,CAAC,CAAC,GAAG,GAAG,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;gBACxB,MAAM;YACR,KAAK,QAAQ;gBACX,CAAC,CAAC,IAAI,GAAG,IAAI,CAAC;gBACd,MAAM;YACR,KAAK,YAAY;gBACf,CAAC,CAAC,KAAK,GAAG,KAAK,CAAC;gBAChB,MAAM;YACR,KAAK,aAAa,CAAC,CAAC,CAAC;gBACnB,MAAM,CAAC,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC,CAAC;gBAClC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;oBAAE,MAAM,IAAI,UAAU,CAAC,qCAAqC,CAAC,CAAC;gBACrF,CAAC,CAAC,QAAQ,GAAG,CAAC,CAAC;gBACf,MAAM;YACR,CAAC;YACD,KAAK,eAAe;gBAClB,CAAC,CAAC,UAAU,GAAG,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;gBAC/B,MAAM;YACR,KAAK,kBAAkB;gBACrB,CAAC,CAAC,YAAY,GAAG,KAAK,CAAC,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;gBACjC,MAAM;YACR,KAAK,IAAI,CAAC;YACV,KAAK,QAAQ;gBACX,CAAC,CAAC,IAAI,GAAG,IAAI,CAAC;gBACd,MAAM;YACR,KAAK,IAAI,CAAC;YACV,KAAK,WAAW;gBACd,CAAC,CAAC,OAAO,GAAG,IAAI,CAAC;gBACjB,MAAM;YACR;gBACE,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,GAAG,KAAK,GAAG;oBAAE,MAAM,IAAI,UAAU,CAAC,kBAAkB,GAAG,EAAE,CAAC,CAAC;gBACtF,IAAI,CAAC,CAAC,OAAO,KAAK,SAAS;oBAAE,CAAC,CAAC,OAAO,GAAG,GAAG,CAAC;;oBACxC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAChC,CAAC;IACH,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,IAAI,CAAC,EAAS;IAClC,IAAI,IAAU,CAAC;IACf,IAAI,CAAC;QACH,IAAI,GAAG,SAAS,CAAC,EAAE,CAAC,IAAI,CAAC,CAAC;IAC5B,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,EAAE,CAAC,MAAM,CAAC,cAAe,GAAa,CAAC,OAAO,uCAAuC,CAAC,CAAC;QACvF,OAAO,IAAI,CAAC,KAAK,CAAC;IACpB,CAAC;IACD,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;QACjB,EAAE,CAAC,MAAM,CAAC,GAAG,EAAE,CAAC,OAAO,IAAI,CAAC,CAAC;QAC7B,OAAO,IAAI,CAAC,IAAI,CAAC;IACnB,CAAC;IACD,IAAI,IAAI,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;QAC/B,EAAE,CAAC,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,CAAC;QACvB,OAAO,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC;IAC5C,CAAC;IAED,IAAI,CAAC;QACH,QAAQ,IAAI,CAAC,OAAO,EAAE,CAAC;YACrB,KAAK,QAAQ;gBACX,EAAE,CAAC,MAAM,CAAC,GAAG,yBAAyB,EAAE,IAAI,CAAC,CAAC;gBAC9C,OAAO,IAAI,CAAC,IAAI,CAAC;YACnB,KAAK,OAAO;gBACV,OAAO,MAAM,KAAK,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;YAC/B,KAAK,QAAQ,CAAC,CAAC,CAAC;gBACd,IAAI,IAAI,CAAC,UAAU,CAAC,MAAM,KAAK,CAAC;oBAAE,MAAM,IAAI,UAAU,CAAC,sCAAsC,CAAC,CAAC;gBAC/F,MAAM,GAAG,GAAG,MAAM,UAAU,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;gBACrD,EAAE,CAAC,MAAM,CAAC,GAAG,GAAG,CAAC,IAAI,IAAI,CAAC,CAAC;gBAC3B,OAAO,IAAI,CAAC,IAAI,CAAC;YACnB,CAAC;YACD;gBACE,MAAM,IAAI,UAAU,CAAC,oBAAoB,IAAI,CAAC,OAAO,GAAG,CAAC,CAAC;QAC9D,CAAC;IACH,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,GAAG,YAAY,UAAU,CAAC,CAAC,CAAC,qCAAqC,CAAC,CAAC,CAAC,EAAE,CAAC;QACpF,EAAE,CAAC,MAAM,CAAC,cAAe,GAAa,CAAC,OAAO,GAAG,IAAI,IAAI,CAAC,CAAC;QAC3D,OAAO,IAAI,CAAC,KAAK,CAAC;IACpB,CAAC;AACH,CAAC;AAED,KAAK,UAAU,KAAK,CAAC,IAAU,EAAE,EAAS;IACxC,MAAM,IAAI,GAAG,CAAC,IAAY,EAAE,EAAE,CAAC,CAAC,IAAI,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC;IACnF,IAAI,MAAc,CAAC;IACnB,IAAI,OAAyB,CAAC;IAE9B,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;QACb,IAAI,IAAI,CAAC,UAAU,CAAC,MAAM,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YAClD,MAAM,IAAI,UAAU,CAAC,+CAA+C,CAAC,CAAC;QACxE,CAAC;QACD,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAA4C,CAAC;QACxF,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC;QAClC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC;YAAE,MAAM,IAAI,UAAU,CAAC,8BAA8B,CAAC,CAAC;QACtF,OAAO,GAAG,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;YACjC,IAAI,OAAO,CAAC,EAAE,IAAI,KAAK,QAAQ;gBAAE,MAAM,IAAI,UAAU,CAAC,eAAe,CAAC,yBAAyB,CAAC,CAAC;YACjG,OAAO,CAAmB,CAAC;QAC7B,CAAC,CAAC,CAAC;IACL,CAAC;SAAM,CAAC;QACN,IAAI,IAAI,CAAC,UAAU,CAAC,MAAM,KAAK,CAAC;YAAE,MAAM,IAAI,UAAU,CAAC,4CAA4C,CAAC,CAAC;QACrG,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM;YAAE,MAAM,IAAI,UAAU,CAAC,+CAA+C,CAAC,CAAC;QAChG,MAAM,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC;QACxC,OAAO,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC;IAC1E,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE;QAAE,MAAM,IAAI,UAAU,CAAC,qBAAqB,CAAC,CAAC;IAEhE,MAAM,UAAU,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,YAAY,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,SAAS,EAAE,KAAK,EAAE,SAAS,EAAE,CAAC;IAChG,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,KAAK,EAAE,UAAU,CAAC,KAAK,EAAE,CAAC,CAAC;IAChF,MAAM,IAAI,GAAG,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,IAAI,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;IAExG,IAAI,IAAI,CAAC,IAAI,EAAE,CAAC;QACd,EAAE,CAAC,MAAM,CACP,GAAG,IAAI,CAAC,SAAS,CACf;YACE,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,KAAK,EAAE,UAAU,CAAC,KAAK,IAAI,IAAI;YAC/B,iGAAiG;YACjG,WAAW,EAAE,MAAM,CAAC,SAAS;YAC7B,OAAO,EAAE,MAAM,CAAC,OAAO;YACvB,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,SAAS,EAAE,MAAM,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;gBACtC,OAAO,EAAE,CAAC,CAAC,OAAO;gBAClB,WAAW,EAAE,CAAC,CAAC,WAAW;gBAC1B,KAAK,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC,SAAS,CAAC,MAAM,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE;gBACvE,KAAK,EAAE,CAAC,CAAC,UAAU;oBACjB,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,UAAU,CAAC,KAAK,EAAE,OAAO,EAAE,CAAC,CAAC,UAAU,CAAC,UAAU,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,UAAU,CAAC,UAAU,CAAC,EAAE;oBACnH,CAAC,CAAC,IAAI;gBACR,KAAK,EAAE,CAAC,CAAC,KAAK,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC;aACjD,CAAC,CAAC;YACH,QAAQ,EAAE,MAAM,CAAC,QAAQ;YACzB,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,GAAG,CAAC,EAAE,GAAG,EAAE,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC;YACpG,oBAAoB,EAAE,IAAI,CAAC,oBAAoB;SAChD,EACD,IAAI,EACJ,CAAC,CACF,IAAI,CACN,CAAC;IACJ,CAAC;SAAM,CAAC;QACN,EAAE,CAAC,MAAM,CAAC,WAAW,CAAC,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,UAAU,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,EAAE,IAAI,CAAC,QAAQ,IAAI,GAAG,CAAC,CAAC,CAAC;IAClG,CAAC;IACD,OAAO,IAAI,CAAC,OAAO,KAAK,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC;AACpG,CAAC;AAED,MAAM,MAAM,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;AAExD,KAAK,UAAU,UAAU,CAAC,GAAW,EAAE,EAAS;IAC9C,IAAI,eAAe,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QAC9B,OAAO,WAAW,CAAC,GAAG,EAAE;YACtB,KAAK,EAAE,EAAE,CAAC,KAAK;YACf,SAAS,EAAE,aAAa,EAAE,CAAC,OAAO,gDAAgD;SACnF,CAAC,CAAC;IACL,CAAC;IACD,IAAI,SAAS,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACxB,MAAM,IAAI,KAAK,CAAC,GAAG,GAAG,2FAA2F,CAAC,CAAC;IACrH,CAAC;IACD,MAAM,IAAI,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;IACpC,MAAM,MAAM,GAAG,cAAc,CAAC,IAAI,EAAE,aAAa,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC;IAC1F,OAAO,EAAE,GAAG,MAAM,EAAE,EAAE,EAAE,GAAG,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,GAAG,EAAE,CAAC;AAC5D,CAAC;AAED,SAAS,YAAY,CAAC,IAAU,EAAE,EAAS;IACzC,IAAI,EAAE,CAAC,KAAK;QAAE,OAAO,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,EAAE,KAAK,EAAE,IAAI,CAAC,UAAU,IAAI,QAAQ,EAAE,CAAC;IAC7E,MAAM,MAAM,GAAG,EAAE,CAAC,GAAG,CAAC,uBAAuB,IAAI,EAAE,CAAC,GAAG,CAAC,kBAAkB,CAAC;IAC3E,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IACvB,MAAM,KAAK,GAAG,IAAI,CAAC,UAAU,IAAI,EAAE,CAAC,GAAG,CAAC,qBAAqB,CAAC;IAC9D,IAAI,CAAC,KAAK,EAAE,CAAC;QACX,MAAM,IAAI,UAAU,CAAC,oFAAoF,CAAC,CAAC;IAC7G,CAAC;IACD,MAAM,OAAO,GACX,IAAI,CAAC,YAAY,IAAI,EAAE,CAAC,GAAG,CAAC,wBAAwB,IAAI,EAAE,CAAC,GAAG,CAAC,kBAAkB,IAAI,8BAA8B,CAAC;IACtH,OAAO;QACL,KAAK;QACL,KAAK,EAAE,IAAI,oBAAoB,CAAC,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,EAAE,CAAC;KAChG,CAAC;AACJ,CAAC;AAED,4EAA4E;AAE5E,MAAM,WAAW,GAA2B;IAC1C,KAAK,EAAE,UAAU;IACjB,UAAU,EAAE,UAAU;IACtB,MAAM,EAAE,wBAAwB;IAChC,KAAK,EAAE,OAAO;CACf,CAAC;AAEF,SAAS,WAAW,CAClB,MAA0B,EAC1B,IAAgB,EAChB,OAAyB,EACzB,KAAyB,EACzB,KAAc,EACd,QAAgB;IAEhB,MAAM,CAAC,GAAG,CAAC,IAAY,EAAE,CAAS,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,QAAQ,IAAI,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAChF,MAAM,KAAK,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IACxC,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IACtC,MAAM,MAAM,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IACzC,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC;IACrC,MAAM,GAAG,GAAa,EAAE,CAAC;IAEzB,GAAG,CAAC,IAAI,CACN,GAAG,CAAC,GAAG,MAAM,CAAC,SAAS,CAAC,MAAM,kBAAkB,OAAO,CAAC,MAAM,uBAAuB,KAAK,IAAI,mBAAmB,EAAE,CAAC,EACpH,EAAE,CACH,CAAC;IAEF,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,OAAO,IAAI,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;IAClF,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,SAAS,EAAE,CAAC;QACnC,MAAM,GAAG,GAAG,MAAM,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC,OAAO,IAAI,GAAG,CAAC,WAAW,EAAE,CAAC,CAAC;QAC5D,MAAM,IAAI,GAAG,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,KAAK,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;QAC5E,MAAM,KAAK,GACT,GAAG,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW;YAClC,CAAC,CAAC,GAAG,CAAC,6BAA6B,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC;YACrE,CAAC,CAAC,GAAG,WAAW,CAAC,GAAG,CAAC,SAAS,CAAC,MAAM,CAAC,IAAI,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;QAC/E,MAAM,MAAM,GAAG,GAAG,CAAC,UAAU;YAC3B,CAAC,CAAC,GAAG,CAAC,UAAU,CAAC,KAAK,KAAK,OAAO;gBAChC,CAAC,CAAC,MAAM,CAAC,aAAa,CAAC;gBACvB,CAAC,CAAC,GAAG,GAAG,CAAC,UAAU,CAAC,KAAK,IAAI,GAAG,CAAC,UAAU,CAAC,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,EAAE;YACnF,CAAC,CAAC,EAAE,CAAC;QACP,GAAG,CAAC,IAAI,CAAC,GAAG,IAAI,IAAI,GAAG,CAAC,OAAO,KAAK,GAAG,CAAC,WAAW,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACxG,GAAG,CAAC,IAAI,CAAC,OAAO,QAAQ,CAAC,GAAG,CAAC,SAAS,IAAI,iBAAiB,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;QACrE,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,KAAK,GAAG,CAAC,OAAO,IAAI,CAAC,CAAC,WAAW,KAAK,GAAG,CAAC,WAAW,CAAC,EAAE,IAAI,KAAK,0BAA0B,CAAC;QAC7I,IAAI,GAAG,IAAI,GAAG,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW;YAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,kDAAkD,CAAC,CAAC,CAAC;aAC9G,IAAI,KAAK,EAAE,CAAC;YACf,MAAM,IAAI,GAAG,CAAC,GAAG,CAAC,SAAS,CAAC,WAAW,IAAI,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YAC/E,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,+BAA+B,IAAI,8CAA8C,CAAC,CAAC,CAAC;QACnG,CAAC;aAAM,IAAI,GAAG,IAAI,GAAG,CAAC,UAAU,EAAE,OAAO,CAAC,MAAM;YAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,OAAO,GAAG,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,OAAO,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;aAClJ,IAAI,GAAG,IAAI,GAAG,CAAC,KAAK,KAAK,IAAI,IAAI,GAAG,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW,EAAE,CAAC;YAC3E,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,sBAAsB,GAAG,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,yBAAyB,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAC1G,CAAC;IACH,CAAC;IACD,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,OAAO;QAAE,GAAG,CAAC,IAAI,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,aAAa,QAAQ,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;IACtF,KAAK,MAAM,CAAC,IAAI,MAAM,CAAC,QAAQ;QAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,YAAY,CAAC,EAAE,CAAC,CAAC,CAAC;IAEhE,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACb,IAAI,IAAI,CAAC,OAAO,KAAK,MAAM,EAAE,CAAC;QAC5B,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,oCAAoC,CAAC,CAAC,CAAC;IAC3F,CAAC;SAAM,IAAI,IAAI,CAAC,OAAO,KAAK,YAAY,EAAE,CAAC;QACzC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,mCAAmC,IAAI,CAAC,QAAQ,CAAC,MAAM,sEAAsE,CAAC,CAAC,CAAC;IAClJ,CAAC;SAAM,CAAC;QACN,MAAM,KAAK,GAAG;YACZ,IAAI,CAAC,QAAQ,CAAC,MAAM,IAAI,GAAG,IAAI,CAAC,QAAQ,CAAC,MAAM,qBAAqB;YACpE,IAAI,CAAC,OAAO,CAAC,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,sBAAsB;SACpE,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QAClB,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,WAAW,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;QAC7C,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM;YAAE,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,4BAA4B,IAAI,CAAC,QAAQ,CAAC,MAAM,eAAe,CAAC,CAAC,CAAC;IAC9G,CAAC;IACD,IAAI,CAAC,KAAK;QAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,gGAAgG,CAAC,CAAC,CAAC;IAC5H,OAAO,GAAG,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC;AAC/B,CAAC;AAED,SAAS,QAAQ,CAAC,CAAS,EAAE,CAAS;IACpC,MAAM,IAAI,GAAG,CAAC,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IAC3C,OAAO,IAAI,CAAC,MAAM,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC;AAC9D,CAAC"}
|
package/dist/gate.d.ts
CHANGED
|
@@ -11,7 +11,7 @@ export interface GateThresholds {
|
|
|
11
11
|
/** Uncited sentences shorter than this are ignored. Default 40. */
|
|
12
12
|
minUncitedChars?: number;
|
|
13
13
|
}
|
|
14
|
-
export type ProblemType = 'quote_not_in_source' | 'contradicted_by_source' | 'overstated' | 'weakly_supported';
|
|
14
|
+
export type ProblemType = 'quote_not_in_source' | 'contradicted_by_source' | 'overstated' | 'weakly_supported' | 'ellipsis_hides_qualifier';
|
|
15
15
|
export interface GateProblem {
|
|
16
16
|
claimId: string;
|
|
17
17
|
sourceIndex: number;
|
|
@@ -30,11 +30,21 @@ export interface GateProblem {
|
|
|
30
30
|
score: number | null;
|
|
31
31
|
}
|
|
32
32
|
export interface GateResult {
|
|
33
|
-
|
|
33
|
+
/**
|
|
34
|
+
* `unverified`: nothing failed, but the judge could not rate some citations,
|
|
35
|
+
* so support was not checked for them. Never report that as a pass.
|
|
36
|
+
*/
|
|
37
|
+
verdict: 'pass' | 'revise' | 'unverified';
|
|
34
38
|
problems: GateProblem[];
|
|
35
39
|
/** Factual-looking sentences without any `[n]` marker. Heuristic. */
|
|
36
40
|
uncited: string[];
|
|
37
|
-
/**
|
|
41
|
+
/** Citations the judge failed on (only when a judge ran). */
|
|
42
|
+
unjudged: {
|
|
43
|
+
claimId: string;
|
|
44
|
+
sourceIndex: number;
|
|
45
|
+
reason: string;
|
|
46
|
+
}[];
|
|
47
|
+
/** Ready-to-send correction prompt; `null` unless the verdict is `revise`. */
|
|
38
48
|
instructionsForModel: string | null;
|
|
39
49
|
}
|
|
40
50
|
/** Apply the gate to a report produced by `verifyAnswer` for `answer`. */
|
package/dist/gate.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"gate.d.ts","sourceRoot":"","sources":["../src/gate.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,KAAK,EAAwB,eAAe,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAC;AAG5F,MAAM,WAAW,cAAc;IAC7B,oDAAoD;IACpD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,mEAAmE;IACnE,eAAe,CAAC,EAAE,MAAM,CAAC;CAC1B;AAED,MAAM,MAAM,WAAW,GACnB,qBAAqB,GACrB,wBAAwB,GACxB,YAAY,GACZ,kBAAkB,CAAC;
|
|
1
|
+
{"version":3,"file":"gate.d.ts","sourceRoot":"","sources":["../src/gate.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,KAAK,EAAwB,eAAe,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAC;AAG5F,MAAM,WAAW,cAAc;IAC7B,oDAAoD;IACpD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,mEAAmE;IACnE,eAAe,CAAC,EAAE,MAAM,CAAC;CAC1B;AAED,MAAM,MAAM,WAAW,GACnB,qBAAqB,GACrB,wBAAwB,GACxB,YAAY,GACZ,kBAAkB,GAClB,0BAA0B,CAAC;AAE/B,MAAM,WAAW,WAAW;IAC1B,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;IAClB,KAAK,EAAE,MAAM,CAAC;IACd,IAAI,EAAE,WAAW,CAAC;IAClB,SAAS,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAC7C,UAAU,EAAE;QAAE,KAAK,EAAE,eAAe,GAAG,OAAO,CAAC;QAAC,UAAU,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,OAAO,EAAE,MAAM,EAAE,CAAA;KAAE,GAAG,IAAI,CAAC;IACtG,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;CACtB;AAED,MAAM,WAAW,UAAU;IACzB;;;OAGG;IACH,OAAO,EAAE,MAAM,GAAG,QAAQ,GAAG,YAAY,CAAC;IAC1C,QAAQ,EAAE,WAAW,EAAE,CAAC;IACxB,qEAAqE;IACrE,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,6DAA6D;IAC7D,QAAQ,EAAE;QAAE,OAAO,EAAE,MAAM,CAAC;QAAC,WAAW,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,EAAE,CAAC;IACrE,8EAA8E;IAC9E,oBAAoB,EAAE,MAAM,GAAG,IAAI,CAAC;CACrC;AAID,0EAA0E;AAC1E,wBAAgB,UAAU,CACxB,MAAM,EAAE,kBAAkB,EAC1B,MAAM,EAAE,MAAM,EACd,UAAU,GAAE,cAAmB,GAC9B,UAAU,CAgCZ"}
|
package/dist/gate.js
CHANGED
|
@@ -28,26 +28,43 @@ export function gateReport(report, answer, thresholds = {}) {
|
|
|
28
28
|
score: c.score === null ? null : round(c.score),
|
|
29
29
|
}));
|
|
30
30
|
const uncited = findUncitedSentences(stripEvi1Appendix(answer), t.minUncitedChars);
|
|
31
|
-
const
|
|
31
|
+
const unjudged = report.citations
|
|
32
|
+
.filter((c) => c.entailment?.class === 'error')
|
|
33
|
+
.map((c) => ({ claimId: c.claimId, sourceIndex: c.sourceIndex, reason: c.entailment?.reasons.join('; ') || 'judge error' }));
|
|
34
|
+
const verdict = problems.length || uncited.length ? 'revise' : unjudged.length ? 'unverified' : 'pass';
|
|
32
35
|
return {
|
|
33
36
|
verdict,
|
|
34
37
|
problems,
|
|
35
38
|
uncited,
|
|
36
|
-
|
|
39
|
+
unjudged,
|
|
40
|
+
instructionsForModel: verdict === 'revise' ? correctionInstructions(problems, uncited) : null,
|
|
37
41
|
};
|
|
38
42
|
}
|
|
39
43
|
function isProblem(c, minScore) {
|
|
40
44
|
if (c.textMatch.method === 'not_found')
|
|
41
45
|
return true;
|
|
46
|
+
if (hidesQualifier(c))
|
|
47
|
+
return true;
|
|
42
48
|
const cls = c.entailment?.class;
|
|
43
49
|
if (cls === 'contradicted' || cls === 'overstated')
|
|
44
50
|
return true;
|
|
45
51
|
// A null score means the judge errored; an outage is not evidence against the claim.
|
|
46
52
|
return c.score !== null && c.score < minScore;
|
|
47
53
|
}
|
|
54
|
+
/**
|
|
55
|
+
* The ellipsis leaves out a word such as "not" or "except", and no judge has
|
|
56
|
+
* read the full passage. A judge verdict settles it either way ("not only …
|
|
57
|
+
* but also" is harmless), so this only fires without one.
|
|
58
|
+
*/
|
|
59
|
+
function hidesQualifier(c) {
|
|
60
|
+
const judged = c.entailment && c.entailment.class !== 'error';
|
|
61
|
+
return !judged && (c.textMatch.omittedCues?.length ?? 0) > 0;
|
|
62
|
+
}
|
|
48
63
|
function problemType(c) {
|
|
49
64
|
if (c.textMatch.method === 'not_found')
|
|
50
65
|
return 'quote_not_in_source';
|
|
66
|
+
if (hidesQualifier(c))
|
|
67
|
+
return 'ellipsis_hides_qualifier';
|
|
51
68
|
if (c.entailment?.class === 'contradicted')
|
|
52
69
|
return 'contradicted_by_source';
|
|
53
70
|
if (c.entailment?.class === 'overstated')
|
|
@@ -93,6 +110,7 @@ const FIX = {
|
|
|
93
110
|
contradicted_by_source: 'the cited source contradicts this claim — remove the claim or correct it to match the source',
|
|
94
111
|
overstated: 'this claim is stronger or more general than the evidence — weaken it to exactly what the quote supports',
|
|
95
112
|
weakly_supported: 'this claim is only weakly supported — cite a better passage or soften the wording',
|
|
113
|
+
ellipsis_hides_qualifier: 'the ellipsis in the quote leaves out a word that may reverse or limit the statement — quote the passage without the omission, or split it into separate quotes',
|
|
96
114
|
};
|
|
97
115
|
function correctionInstructions(problems, uncited) {
|
|
98
116
|
const lines = [
|
package/dist/gate.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"gate.js","sourceRoot":"","sources":["../src/gate.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAGH,OAAO,EAAE,iBAAiB,EAAE,MAAM,oBAAoB,CAAC;
|
|
1
|
+
{"version":3,"file":"gate.js","sourceRoot":"","sources":["../src/gate.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAGH,OAAO,EAAE,iBAAiB,EAAE,MAAM,oBAAoB,CAAC;AA0CvD,MAAM,QAAQ,GAA6B,EAAE,QAAQ,EAAE,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE,CAAC;AAElF,0EAA0E;AAC1E,MAAM,UAAU,UAAU,CACxB,MAA0B,EAC1B,MAAc,EACd,aAA6B,EAAE;IAE/B,MAAM,CAAC,GAAG,EAAE,GAAG,QAAQ,EAAE,GAAG,UAAU,EAAE,CAAC;IACzC,MAAM,QAAQ,GAAG,MAAM,CAAC,SAAS;SAC9B,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,SAAS,CAAC,CAAC,EAAE,CAAC,CAAC,QAAQ,CAAC,CAAC;SACvC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QACX,OAAO,EAAE,CAAC,CAAC,OAAO;QAClB,WAAW,EAAE,CAAC,CAAC,WAAW;QAC1B,SAAS,EAAE,CAAC,CAAC,SAAS;QACtB,KAAK,EAAE,CAAC,CAAC,KAAK;QACd,IAAI,EAAE,WAAW,CAAC,CAAC,CAAC;QACpB,SAAS,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC,SAAS,CAAC,MAAM,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE;QAC1E,UAAU,EAAE,CAAC,CAAC,UAAU;YACtB,CAAC,CAAC;gBACE,KAAK,EAAE,CAAC,CAAC,UAAU,CAAC,KAAK;gBACzB,UAAU,EAAE,CAAC,CAAC,UAAU,CAAC,UAAU,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,UAAU,CAAC,UAAU,CAAC;gBACpF,OAAO,EAAE,CAAC,CAAC,UAAU,CAAC,OAAO;aAC9B;YACH,CAAC,CAAC,IAAI;QACR,KAAK,EAAE,CAAC,CAAC,KAAK,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC;KAChD,CAAC,CAAC,CAAC;IACN,MAAM,OAAO,GAAG,oBAAoB,CAAC,iBAAiB,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,eAAe,CAAC,CAAC;IACnF,MAAM,QAAQ,GAAG,MAAM,CAAC,SAAS;SAC9B,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,UAAU,EAAE,KAAK,KAAK,OAAO,CAAC;SAC9C,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,OAAO,EAAE,WAAW,EAAE,CAAC,CAAC,WAAW,EAAE,MAAM,EAAE,CAAC,CAAC,UAAU,EAAE,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,aAAa,EAAE,CAAC,CAAC,CAAC;IAC/H,MAAM,OAAO,GAAG,QAAQ,CAAC,MAAM,IAAI,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,MAAM,CAAC;IACvG,OAAO;QACL,OAAO;QACP,QAAQ;QACR,OAAO;QACP,QAAQ;QACR,oBAAoB,EAAE,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,sBAAsB,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC,IAAI;KAC9F,CAAC;AACJ,CAAC;AAED,SAAS,SAAS,CAAC,CAAuB,EAAE,QAAgB;IAC1D,IAAI,CAAC,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW;QAAE,OAAO,IAAI,CAAC;IACpD,IAAI,cAAc,CAAC,CAAC,CAAC;QAAE,OAAO,IAAI,CAAC;IACnC,MAAM,GAAG,GAAG,CAAC,CAAC,UAAU,EAAE,KAAK,CAAC;IAChC,IAAI,GAAG,KAAK,cAAc,IAAI,GAAG,KAAK,YAAY;QAAE,OAAO,IAAI,CAAC;IAChE,qFAAqF;IACrF,OAAO,CAAC,CAAC,KAAK,KAAK,IAAI,IAAI,CAAC,CAAC,KAAK,GAAG,QAAQ,CAAC;AAChD,CAAC;AAED;;;;GAIG;AACH,SAAS,cAAc,CAAC,CAAuB;IAC7C,MAAM,MAAM,GAAG,CAAC,CAAC,UAAU,IAAI,CAAC,CAAC,UAAU,CAAC,KAAK,KAAK,OAAO,CAAC;IAC9D,OAAO,CAAC,MAAM,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC,WAAW,EAAE,MAAM,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;AAC/D,CAAC;AAED,SAAS,WAAW,CAAC,CAAuB;IAC1C,IAAI,CAAC,CAAC,SAAS,CAAC,MAAM,KAAK,WAAW;QAAE,OAAO,qBAAqB,CAAC;IACrE,IAAI,cAAc,CAAC,CAAC,CAAC;QAAE,OAAO,0BAA0B,CAAC;IACzD,IAAI,CAAC,CAAC,UAAU,EAAE,KAAK,KAAK,cAAc;QAAE,OAAO,wBAAwB,CAAC;IAC5E,IAAI,CAAC,CAAC,UAAU,EAAE,KAAK,KAAK,YAAY;QAAE,OAAO,YAAY,CAAC;IAC9D,OAAO,kBAAkB,CAAC;AAC5B,CAAC;AAED;;;;GAIG;AACH,SAAS,oBAAoB,CAAC,IAAY,EAAE,QAAgB;IAC1D,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,MAAM,OAAO,IAAI,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;QACvC,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC;QAC5B,IAAI,CAAC,IAAI,IAAI,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC;YAAE,SAAS;QAC9C,KAAK,MAAM,QAAQ,IAAI,cAAc,CAAC,IAAI,CAAC,EAAE,CAAC;YAC5C,MAAM,CAAC,GAAG,QAAQ,CAAC,IAAI,EAAE,CAAC;YAC1B,IAAI,CAAC,CAAC,MAAM,GAAG,QAAQ,IAAI,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC;gBAAE,SAAS;YACrD,IAAI,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC,CAAC;gBAAE,SAAS;YACzD,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,qFAAqF;AACrF,MAAM,YAAY,GAAG,uHAAuH,CAAC;AAE7I,SAAS,cAAc,CAAC,IAAY;IAClC,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,mCAAmC,CAAC,EAAE,CAAC;QACpE,IAAI,GAAG,CAAC,MAAM,IAAI,YAAY,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;YAAE,GAAG,CAAC,GAAG,CAAC,MAAM,GAAG,CAAC,CAAC,IAAI,IAAI,KAAK,EAAE,CAAC;;YACxF,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IACvB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,MAAM,GAAG,GAAgC;IACvC,mBAAmB,EACjB,mHAAmH;IACrH,sBAAsB,EAAE,8FAA8F;IACtH,UAAU,EAAE,yGAAyG;IACrH,gBAAgB,EAAE,mFAAmF;IACrG,wBAAwB,EACtB,gKAAgK;CACnK,CAAC;AAEF,SAAS,sBAAsB,CAAC,QAAuB,EAAE,OAAiB;IACxE,MAAM,KAAK,GAAG;QACZ,oIAAoI;KACrI,CAAC;IACF,KAAK,MAAM,CAAC,IAAI,QAAQ,EAAE,CAAC;QACzB,KAAK,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC,OAAO,MAAM,QAAQ,CAAC,CAAC,CAAC,SAAS,EAAE,GAAG,CAAC,OAAO,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IACxF,CAAC;IACD,KAAK,MAAM,CAAC,IAAI,OAAO,EAAE,CAAC;QACxB,KAAK,CAAC,IAAI,CAAC,iCAAiC,QAAQ,CAAC,CAAC,EAAE,GAAG,CAAC,yDAAyD,CAAC,CAAC;IACzH,CAAC;IACD,KAAK,CAAC,IAAI,CAAC,0EAA0E,CAAC,CAAC;IACvF,OAAO,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC1B,CAAC;AAED,SAAS,QAAQ,CAAC,CAAS,EAAE,CAAS;IACpC,OAAO,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC;AACrD,CAAC;AAED,SAAS,KAAK,CAAC,CAAS;IACtB,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;AACnC,CAAC"}
|
package/dist/index.d.ts
CHANGED
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
* which claims are verbatim-backed, supported, overstated, or unsupported.
|
|
12
12
|
*/
|
|
13
13
|
export * from './types.js';
|
|
14
|
-
export { EVI1_START, EVI1_END, parseAnswer, parseEvi1Appendix, stripEvi1Appendix, serializeEvi1Appendix, extractClaims, stripClaimMarkers, } from './protocol/evi1.js';
|
|
14
|
+
export { EVI1_START, EVI1_END, parseAnswer, parseEvi1Appendix, stripEvi1Appendix, stripForDisplay, serializeEvi1Appendix, extractClaims, stripClaimMarkers, } from './protocol/evi1.js';
|
|
15
15
|
export { buildCitationInstructions, type CitationPromptOptions } from './protocol/prompt.js';
|
|
16
16
|
export { matchQuoteAgainstSource, matchQuoteAgainstText, trigramCounts, diceSimilarity, bestFuzzyWindow, type MatchOptions, } from './match/fuzzy.js';
|
|
17
17
|
export { normalizeForMatch, type NormalizedText } from './match/normalize.js';
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,cAAc,YAAY,CAAC;AAE3B,OAAO,EACL,UAAU,EACV,QAAQ,EACR,WAAW,EACX,iBAAiB,EACjB,iBAAiB,EACjB,qBAAqB,EACrB,aAAa,EACb,iBAAiB,GAClB,MAAM,oBAAoB,CAAC;AAE5B,OAAO,EAAE,yBAAyB,EAAE,KAAK,qBAAqB,EAAE,MAAM,sBAAsB,CAAC;AAE7F,OAAO,EACL,uBAAuB,EACvB,qBAAqB,EACrB,aAAa,EACb,cAAc,EACd,eAAe,EACf,KAAK,YAAY,GAClB,MAAM,kBAAkB,CAAC;AAC1B,OAAO,EAAE,iBAAiB,EAAE,KAAK,cAAc,EAAE,MAAM,sBAAsB,CAAC;AAE9E,OAAO,EACL,oBAAoB,EACpB,kBAAkB,EAClB,KAAK,gBAAgB,GACtB,MAAM,uBAAuB,CAAC;AAE/B,OAAO,EAAE,YAAY,EAAE,KAAK,aAAa,EAAE,MAAM,aAAa,CAAC;AAE/D,OAAO,EAAE,UAAU,EAAE,KAAK,UAAU,EAAE,KAAK,WAAW,EAAE,KAAK,cAAc,EAAE,KAAK,WAAW,EAAE,MAAM,WAAW,CAAC;AACjH,OAAO,EAAE,WAAW,EAAE,cAAc,EAAE,KAAK,kBAAkB,EAAE,MAAM,mBAAmB,CAAC;AACzF,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,cAAc,YAAY,CAAC;AAE3B,OAAO,EACL,UAAU,EACV,QAAQ,EACR,WAAW,EACX,iBAAiB,EACjB,iBAAiB,EACjB,eAAe,EACf,qBAAqB,EACrB,aAAa,EACb,iBAAiB,GAClB,MAAM,oBAAoB,CAAC;AAE5B,OAAO,EAAE,yBAAyB,EAAE,KAAK,qBAAqB,EAAE,MAAM,sBAAsB,CAAC;AAE7F,OAAO,EACL,uBAAuB,EACvB,qBAAqB,EACrB,aAAa,EACb,cAAc,EACd,eAAe,EACf,KAAK,YAAY,GAClB,MAAM,kBAAkB,CAAC;AAC1B,OAAO,EAAE,iBAAiB,EAAE,KAAK,cAAc,EAAE,MAAM,sBAAsB,CAAC;AAE9E,OAAO,EACL,oBAAoB,EACpB,kBAAkB,EAClB,KAAK,gBAAgB,GACtB,MAAM,uBAAuB,CAAC;AAE/B,OAAO,EAAE,YAAY,EAAE,KAAK,aAAa,EAAE,MAAM,aAAa,CAAC;AAE/D,OAAO,EAAE,UAAU,EAAE,KAAK,UAAU,EAAE,KAAK,WAAW,EAAE,KAAK,cAAc,EAAE,KAAK,WAAW,EAAE,MAAM,WAAW,CAAC;AACjH,OAAO,EAAE,WAAW,EAAE,cAAc,EAAE,KAAK,kBAAkB,EAAE,MAAM,mBAAmB,CAAC;AACzF,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC"}
|
package/dist/index.js
CHANGED
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
* which claims are verbatim-backed, supported, overstated, or unsupported.
|
|
12
12
|
*/
|
|
13
13
|
export * from './types.js';
|
|
14
|
-
export { EVI1_START, EVI1_END, parseAnswer, parseEvi1Appendix, stripEvi1Appendix, serializeEvi1Appendix, extractClaims, stripClaimMarkers, } from './protocol/evi1.js';
|
|
14
|
+
export { EVI1_START, EVI1_END, parseAnswer, parseEvi1Appendix, stripEvi1Appendix, stripForDisplay, serializeEvi1Appendix, extractClaims, stripClaimMarkers, } from './protocol/evi1.js';
|
|
15
15
|
export { buildCitationInstructions } from './protocol/prompt.js';
|
|
16
16
|
export { matchQuoteAgainstSource, matchQuoteAgainstText, trigramCounts, diceSimilarity, bestFuzzyWindow, } from './match/fuzzy.js';
|
|
17
17
|
export { normalizeForMatch } from './match/normalize.js';
|