@framers/agentos-ext-citation-check 0.2.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +130 -8
- package/dist/claims.d.ts +386 -0
- package/dist/claims.d.ts.map +1 -0
- package/dist/claims.js +792 -0
- package/dist/claims.js.map +1 -0
- package/dist/index.d.ts +6 -3
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +4 -1
- package/dist/index.js.map +1 -1
- package/dist/meta.d.ts +6 -1
- package/dist/meta.d.ts.map +1 -1
- package/dist/meta.js +14 -3
- package/dist/meta.js.map +1 -1
- package/dist/page.d.ts +23 -3
- package/dist/page.d.ts.map +1 -1
- package/dist/page.js +182 -10
- package/dist/page.js.map +1 -1
- package/dist/web-source.d.ts +36 -8
- package/dist/web-source.d.ts.map +1 -1
- package/dist/web-source.js +74 -16
- package/dist/web-source.js.map +1 -1
- package/manifest.json +2 -2
- package/package.json +8 -7
package/README.md
CHANGED
|
@@ -39,7 +39,7 @@ and exits 1. The output is the markdown for a document's "Citation check log" se
|
|
|
39
39
|
npm install @framers/agentos-ext-citation-check
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
-
It needs Node.js 22 or later, as `@framers/agentos` does. It reads pages with [public-sources](https://www.npmjs.com/package/@framers/agentos-ext-public-sources) (`htmlToBlocks` in page mode, `citationSources
|
|
42
|
+
It needs Node.js 22 or later, as `@framers/agentos` does. It reads pages with [public-sources](https://www.npmjs.com/package/@framers/agentos-ext-public-sources) (`htmlToBlocks` in page mode, `citationSources`, the `SourceDocument` a page answers as and `kindOfHost`, from 0.3.0), the [web scraper](https://www.npmjs.com/package/@framers/agentos-ext-web-scraper) (a browser's request headers and the fallback tiers) the `values` entry of [grounding-guard](https://www.npmjs.com/package/@framers/agentos-ext-grounding-guard) (the sentence rule and the word rules) and [entities](https://www.npmjs.com/package/entities) (the character references in a page's title, names and addresses). `@framers/agentos` 0.13.120 or later, the first release with the checked egress the tool reads pages through, is a peer, loaded only to grade with embeddings or to read pages through checked egress. Grounding-guard lists `@huggingface/transformers` as an optional dependency for its NLI model; `npm install --omit=optional` leaves it out when you do not use NLI.
|
|
43
43
|
|
|
44
44
|
## Input
|
|
45
45
|
|
|
@@ -144,9 +144,13 @@ const page = await readPage('https://example.org/harbour', {
|
|
|
144
144
|
if (page.document !== null) console.log(page.document.title, page.document.licence.name, page.document.publisher);
|
|
145
145
|
```
|
|
146
146
|
|
|
147
|
-
`fetchTransport(fetch, userAgent?)` and `egressTransport(profile, bounds, userAgent?)` build the request a page goes through. A transport takes `(url, signal, rules?)` and answers `{ status, finalUrl, header, read, readPrefix, discard }`; a host can pass one of its own.
|
|
147
|
+
`fetchTransport(fetch, userAgent?)` and `egressTransport(profile, bounds, userAgent?)` build the request a page goes through. A transport takes `(url, signal, rules?, limits?)` and answers `{ status, finalUrl, header, read, readPrefix, discard }`; a host can pass one of its own. `limits.maxBytes` is the most of the body the caller will read: the checked egress, which otherwise downloads the body before `read`, then reads the answer's headers first, skips a redirect's body and refuses any other body past that as it arrives, so an icon's read stops at `iconMaxBytes` whatever the page's own cap; a transport that streams the body in `read(cap)` needs nothing more.
|
|
148
148
|
|
|
149
|
-
The document's `url` and `key` are the address after redirects. Its `title` is the page's `<title>`, its character references read, its white space collapsed and cut to 255 characters, or the host when the page has none. Its `licence` is the one the page states (below), or `{ name: 'not read', url: '' }` (`UNREAD_LICENCE`) when the reader finds none. `publisher` is the site's name from `og:site_name`, else from JSON-LD `publisher.name`, cut to 255 characters and left out when the page names none. `pageDate` is the page's own date from `article:modified_time`, else JSON-LD `dateModified`, else the `Last-Modified` header, left out when none parses. The markup is read as served, in one pass in document order: a value a script writes later is not seen, a tag or a JSON-LD block inside a comment is passed over, the content of a script, a style, a template, a `noscript` and a `textarea` is not read for tags, and the time grows with the page alone however its tags are left open. Of the links and anchors the pass keeps only those a licence is read from, at most 1,000 of each kind, and reads the text of a `rel="license"` anchor alone, so a page of a million other anchors holds none of them.
|
|
149
|
+
The document's `url` and `key` are the address after redirects. Its `title` is the page's `<title>`, its character references read, its white space collapsed and cut to 255 characters, or the host when the page has none. Its `licence` is the one the page states (below), or `{ name: 'not read', url: '' }` (`UNREAD_LICENCE`) when the reader finds none. `publisher` is the site's name from `og:site_name`, else from JSON-LD `publisher.name`, cut to 255 characters and left out when the page names none. `pageDate` is the page's own date from `article:modified_time`, else JSON-LD `dateModified`, else the `Last-Modified` header, left out when none parses. The markup is read as served, in one pass in document order: a value a script writes later is not seen, a tag or a JSON-LD block inside a comment is passed over, the content of a script, a style, a template, a `noscript` and a `textarea` is not read for tags, and the time grows with the page alone however its tags are left open. Of the links and anchors the pass keeps only those a licence is read from, at most 1,000 of each kind, and the first 32 `<link>`s whose `rel` names `icon`, and reads the text of a `rel="license"` anchor alone, so a page of a million other anchors holds none of them.
|
|
150
|
+
|
|
151
|
+
#### The page's icon
|
|
152
|
+
|
|
153
|
+
With `icon: true`, `readPage` reads the page's icon once its text is read, with one more request, to the page's own origin after its redirects: the first `<link rel="icon">` (or `rel="shortcut icon"`) whose address is at that origin, an SVG passed over (by its `type`, `image/svg+xml`, or an address ending in `.svg`), since it could not be kept, else that origin's `/favicon.ico`. A declared icon at another host is never requested. The request goes through the same transport, under `allowUrl` and `robots`, every redirect held to the page's origin, under a deadline of its own, `timeoutMs`, as each request has, so the icon can add up to `timeoutMs` to the page's read; at most `iconMaxBytes` (65,536) are read. A PNG, ICO, JPEG or WebP, told by its first bytes and served under an image type, a generic binary type or none, whose data address holds at most 8,192 characters, is kept as `icon` on the document and on each passage (`data:image/png;base64,...`). Any other answer, a refusal or a failure leaves `icon` out and the page is read as it is. No icon is read when `icon` is left out.
|
|
150
154
|
|
|
151
155
|
#### The host's rule and robots.txt, on every hop
|
|
152
156
|
|
|
@@ -190,12 +194,13 @@ The request and its answer's body share one deadline, `timeoutMs`; Firecrawl is
|
|
|
190
194
|
import { WebPageSourceProvider } from '@framers/agentos-ext-citation-check';
|
|
191
195
|
|
|
192
196
|
const web = new WebPageSourceProvider({
|
|
193
|
-
search: async (phrase, signal) => mySearch(phrase, signal), //
|
|
197
|
+
search: async (phrase, signal) => mySearch(phrase, signal), // pages, best first: an address, or { url, foundThrough }
|
|
194
198
|
userAgent: 'ExampleReader/1.0 (https://example.org)',
|
|
195
199
|
robots: { token: 'ExampleReader', timeoutMs: 2_000, maxBytes: 512_000, cacheMs: 600_000 },
|
|
196
200
|
allowUrl: (url, phrase) => myHostRule(url, phrase), // false refuses the address before it is requested
|
|
197
201
|
timeoutMs: 5_000,
|
|
198
202
|
maxBytes: 2_000_000,
|
|
203
|
+
icon: true, // each page's icon, from its own origin
|
|
199
204
|
});
|
|
200
205
|
|
|
201
206
|
const found = await web.search('harbour opening year', { limit: 3, onSend: () => quota.take() });
|
|
@@ -209,7 +214,9 @@ if (found.kind === 'hits') {
|
|
|
209
214
|
|
|
210
215
|
`search(phrase, { limit, signal, onSend })` throws a cancellation that came first, then awaits `onSend` once and runs the host's search, and answers its addresses as hits: `key` the address without its fragment, `title` the address, each page once, at most `limit` (3 when left out), an address that is not http or https passed over. It answers `limited` when `onSend` throws a limiter's refusal, and `failed` when `onSend` throws anything else or the search throws.
|
|
211
216
|
|
|
212
|
-
|
|
217
|
+
The host's search answers each page as its address, or as `{ url, foundThrough }` (`WebSearchAnswer`) with the provider words that found it, in the order the search named its providers. web-search's `multiSearch` names them `providers`, so its `results` can be answered as they come: `providers` is read when `foundThrough` is left out. Words that are not source words (public-sources' `isSourceWord`) are left out, and a page found at two forms of its address that share a key keeps the finders of both, each once, in the order they came.
|
|
218
|
+
|
|
219
|
+
`read(hit, { signal })` reads the page and answers `{ kind: 'document', document }`, with `site` the host without a leading "www.", `revision` null, `readAt` the clock's, the title, licence, publisher and date as read, `kind` by public-sources' `kindOfHost` from the site (`kindOverrides`, the host's own reading by host name, first), for a hit the search found with providers named `foundThrough` and `provider` (the first of them), and with `icon: true` the page's `icon` as `readPage` reads it; or `{ kind: 'skipped', status }` for a page that was not read, with its HTTP status when one came. `allowUrl(url, phrase)` is called on every address and redirect target with the phrase whose search answered the hit read, held by the hit object itself, so pass the hits as `search` answered them: a hit object this provider's search did not answer, a copy of one included, reads with the empty string. robots.txt is read with `robots`, or with `DEFAULT_ROBOTS` (the `*` group, 2,000 ms, 512,000 bytes, ten minutes kept) when left out, and not at all with `robots: null`. Pages go through AgentOS's checked egress (`egress`, `hosted` by default) unless `transport` is given, with no scraper fallback, since the scraper's requests bypass both; `fallback`, such as `FirecrawlPageReader`, reads a page with no text. The provider queues nothing: the host reads at most as many pages at once as it calls `read` for. The caller's own cancellation is thrown.
|
|
213
220
|
|
|
214
221
|
### Grading with embeddings
|
|
215
222
|
|
|
@@ -225,6 +232,120 @@ await checkCitations(claims, { embedFn: managerEmbeddings(embeddingManager, { mo
|
|
|
225
232
|
|
|
226
233
|
The claims of one page that are not on it word for word are graded in one `CitationVerifier.verify` call against the page's sentences (`maxPassages`, 2,000, in page order), which embeds them in one batch. `supportThreshold` and `unverifiableThreshold` pass through to the verifier. An `nliFn`, such as grounding-guard's `createNliFunction()`, lets the verifier turn a `supported` claim its sentence contradicts into `contradicted`; it downloads and runs its model locally. When the embedding function fails, the page's claims are graded by shared words, and each check's `note` says why.
|
|
227
234
|
|
|
235
|
+
### Checking a spoken claim
|
|
236
|
+
|
|
237
|
+
`ClaimChecker` checks one claim heard in a room against sentences from any source: an encyclopedia article, a web page, a paper's abstract, a person's own notes or documents. The host gives the passages (from public-sources' providers, `WebPageSourceProvider`, its own library), its embeddings call and a judge; the checker decides what may be shown.
|
|
238
|
+
|
|
239
|
+
```ts
|
|
240
|
+
import { embedText } from '@framers/agentos';
|
|
241
|
+
import { ClaimChecker, claimJudge, type ClaimPassage } from '@framers/agentos-ext-citation-check';
|
|
242
|
+
import { ValueAgreementChecker } from '@framers/agentos-ext-grounding-guard';
|
|
243
|
+
|
|
244
|
+
const checker = new ClaimChecker({
|
|
245
|
+
// One batch: a vector a text, in order, and the tokens the call used.
|
|
246
|
+
embed: async (texts, signal) => {
|
|
247
|
+
const { embeddings, usage } = await embedText({ provider: 'openai', input: texts, apiKey, abortSignal: signal });
|
|
248
|
+
return { vectors: embeddings, usage: { in: usage.promptTokens } };
|
|
249
|
+
},
|
|
250
|
+
judge: claimJudge({ model: 'openai:gpt-6-luna', apiKey, deadlineMs: 8_000, maxOutputTokens: 400 }),
|
|
251
|
+
checker: new ValueAgreementChecker({ names: true }),
|
|
252
|
+
});
|
|
253
|
+
|
|
254
|
+
const passages: ClaimPassage[] = [
|
|
255
|
+
{
|
|
256
|
+
content: 'The Treaty of Versailles was a peace treaty signed on 28 June 1919.',
|
|
257
|
+
title: 'Treaty of Versailles',
|
|
258
|
+
url: 'https://en.wikipedia.org/wiki/Treaty_of_Versailles',
|
|
259
|
+
group: 'wikipedia:Treaty_of_Versailles', // one source: two passages of one group are of one article
|
|
260
|
+
kind: 'encyclopedia',
|
|
261
|
+
},
|
|
262
|
+
];
|
|
263
|
+
|
|
264
|
+
const outcome = await checker.check(
|
|
265
|
+
{
|
|
266
|
+
heard: 'The Treaty of Versailles was signed in 1918', // the words heard that hold the claim
|
|
267
|
+
heardSentence: 'The Treaty of Versailles was signed in 1918.', // the whole sentence they stand in
|
|
268
|
+
statement: 'The Treaty of Versailles was signed in 1918.', // the claim rewritten to stand alone
|
|
269
|
+
},
|
|
270
|
+
passages,
|
|
271
|
+
{ signal: AbortSignal.timeout(10_000) },
|
|
272
|
+
);
|
|
273
|
+
// outcome.kind 'decided': outcome.decision.relation 'differs', the pair 1918 beside 1919 (offsets into
|
|
274
|
+
// outcome.decision.texts[0], the sentence as read), outcome.named the passage,
|
|
275
|
+
// outcome.quote 'signed on 28 June 1919', outcome.verdict the verifier's ClaimVerdict, outcome.usage the tokens
|
|
276
|
+
// outcome.kind 'ended': outcome.reason 'no_passage' | 'limited' | 'no_best' | 'judge_none' | 'quote' | 'undecided'
|
|
277
|
+
// outcome.kind 'known': a verdict key the host passed in `known`, with nothing sent
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
`check(claim, given, { signal, found, known, beforeSend, verdictContext })` runs these steps in order, and stops at the first that ends it:
|
|
281
|
+
|
|
282
|
+
1. The grounding guard's `passages` keeps the sentences of `given` about the claim (two shared stems and a comparable value, or the claim's names for a stated fact without a value). None ends `no_passage`, with nothing sent anywhere. The first `maxPassages` (24) kept are compared; a sentence two sources share word for word, such as a page that copies an encyclopedia, is compared once, as the first of them, so the judge reads the title and kind of the passage the outcome names, and every passage found still meets step 8.
|
|
283
|
+
2. With `verdictSecret`, the verdict key is made (below). A key in `known` ends the check `known`, with nothing sent.
|
|
284
|
+
3. `beforeSend()` is awaited once, before anything is sent; unless it answers `true` the check ends `limited`. A host counts its allowance here, so a known repeat costs none.
|
|
285
|
+
4. One sentence compared is named with no embeddings call (AgentOS's `CitationVerifier` with `pickOnly`), and the judge decides.
|
|
286
|
+
5. Otherwise the claim is embedded in one call with every compared sentence that carries no passage `vector` and has no stored public vector. Sentence vectors and passage vectors are not on one scale, so each class is ranked by its own cosine, and of the two classes' best the one earlier among the compared sentences (more stems shared with what was heard) is named. A sentence whose cosine is 0 or less is never named; none ends `no_best`. The cosine decides nothing else.
|
|
287
|
+
6. The judge is asked once about that sentence, under `claimJudgeSystem(kind)` with `claimJudgePrompt`: `matches`, `differs` or `none`, with the words it compared quoted. `none` ends `judge_none`.
|
|
288
|
+
7. The quote must hold 1 to 20 words (`CLAIM_QUOTE_MAX_WORDS`) and stand in the sentence word for word, cutting no word or number (the grounding guard's `verifyQuotes`, typographic quotation marks read as plain ones). Else it ends `quote`.
|
|
289
|
+
8. The grounding guard's `decide` takes the judge's relation as a proposal and decides by code over every passage found, the ones in `found` among them: a "differs" needs every content word heard and a heard value paired with a different one, and a passage found that agrees with every heard value stops it. Nothing decided ends `undecided`.
|
|
290
|
+
|
|
291
|
+
`found` is every passage about the claim the host holds, the ones it must not send included, such as a sentence that holds personal data: they are never embedded, judged or sent, and take part in step 8 alone. Left out, it is `given`'s passages about the claim.
|
|
292
|
+
|
|
293
|
+
#### Vectors a host holds, and the verdict key
|
|
294
|
+
|
|
295
|
+
A check embeds no text whose vector the host already holds:
|
|
296
|
+
|
|
297
|
+
```ts
|
|
298
|
+
const checker = new ClaimChecker({
|
|
299
|
+
embed, judge, checker: valueChecker,
|
|
300
|
+
grader: { judgeModel: 'openai:gpt-6-luna', embedding: { model: 'text-embedding-3-small', dimensions: 1536 } },
|
|
301
|
+
// Public sentences' vectors by the hash of their exact input, the tenant fixed by the host, such as AgentOS's
|
|
302
|
+
// SqlEmbeddingStore bound to it; clean is the host's redaction test.
|
|
303
|
+
publicVectors: {
|
|
304
|
+
get: (hashes) => store.get({ tenant }, model, hashes),
|
|
305
|
+
put: (rows) => store.put({ tenant }, model, rows),
|
|
306
|
+
clean: (text) => !redact(text).found,
|
|
307
|
+
},
|
|
308
|
+
verdictSecret: process.env.CLAIM_VERDICT_SECRET,
|
|
309
|
+
});
|
|
310
|
+
const outcome = await checker.check(claim, passages, {
|
|
311
|
+
signal,
|
|
312
|
+
known: keysHeldForThisRun, // a verdict the host already holds ends 'known', with nothing sent
|
|
313
|
+
beforeSend: () => allowance.take(), // awaited once, before the first send
|
|
314
|
+
verdictContext: `${accountId}:${runId}`,
|
|
315
|
+
});
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
- A passage's `vector` (`{ embedding, level: 'passage', passageId, inputHash }`) is the vector of the passage a sentence stands in, made by the check's embedding model: the sentence is ranked by it and never sent to the embeddings. With `grader`, a vector of another size than its embedding's is left out and the sentence is embedded; without it, a vector of another size than the claim's throws a `RangeError`. An embeddings answer holding a vector of another size, or a value that is not a finite number, rejects the check before the judge is asked.
|
|
319
|
+
- A sentence marked `publicKey`, only for a verified public source, whose text `clean` passes, is looked up in `publicVectors` by the SHA-256 of the model, NUL, the dimensions, NUL and the text (AgentOS's `embeddingInputHash`). A store that rejects or answers after `vectorReadMs` (200) counts as empty, and `usage.vectorHits` counts the vectors read. The vectors the check made for such sentences are stored after the verdict, a failed or late write changing nothing; an abort while they are written rejects the check at once with the signal's reason. Any other sentence is embedded and never stored.
|
|
320
|
+
- The verdict key is an HMAC-SHA-256, under a key derived by HKDF from `verdictSecret` with `verdictContext` (an account and a run, so a key never matches across them), over everything that decides the verdict: this pack's and the grounding guard's versions, the judge's model, the embedding model and size, `maxPassages`, the grounding guard checker's rules as `grader.rules` names them (`'names:true'`, for example), the claim, each compared sentence with its text, title, kind, group, `ref` and passage vector (its id, its `inputHash` and a hash of the vector), and each sentence found with its text, group, `ref` and whether it is held back. A changed sentence, a 25th agreeing sentence found, a held-back sentence changed, a new release of either pack or another context makes a new key. Every outcome past `no_passage` carries `key`.
|
|
321
|
+
|
|
322
|
+
`publicVectors` and `verdictSecret` need `grader`; the constructor throws without it.
|
|
323
|
+
|
|
324
|
+
The judge's instruction names the source by the named passage's `kind`, an open word:
|
|
325
|
+
|
|
326
|
+
| `kind` | "one sentence from …" |
|
|
327
|
+
| --- | --- |
|
|
328
|
+
| `encyclopedia` | an encyclopedia article |
|
|
329
|
+
| `official`, `news`, `page`, `forum` | a web page |
|
|
330
|
+
| `paper` | a paper's abstract |
|
|
331
|
+
| `note`, `document` | the person's own notes or documents |
|
|
332
|
+
| any other word | a source |
|
|
333
|
+
|
|
334
|
+
Every instruction opens with "You compare one sentence heard in a room with one sentence from", so a stand-in for the model in a host's tests can tell the judge's call apart from others.
|
|
335
|
+
|
|
336
|
+
`claimJudge({ model, apiKey, baseUrl?, deadlineMs, maxOutputTokens, effort? })` makes the judge on AgentOS's `generateObject` with this pack's schema (`{ relation, quote }`): one request (`maxAttempts: 1`), no second call for an answer outside the schema (`maxRetries: 0`), no other provider (`fallbackProviders: []`), ended at `deadlineMs` or by the check's signal, the body's read included. A host passes any function of the same shape instead, `({ system, prompt, signal }) => Promise<{ value: { relation, quote }, usage: { in, out } }>`, such as a fake in its tests. The pack reads no key from the environment.
|
|
337
|
+
|
|
338
|
+
Passages that arrive after the first decision, from slower sources, are decided by code alone, with no model call:
|
|
339
|
+
|
|
340
|
+
- `agree(first, claim, more)` answers the passages of `more` that agree with a first decision of `matches` or `differs` on the values: one a source other than the first's (by `group`, its best-ranked passage), holding every content stem heard, decided the same relation by the grounding guard, on the same heard value, its value written as the first's.
|
|
341
|
+
- `revisit(first, claim, found)` reads the first decision again over every passage found by then, the check's own found ones among them, and answers it, or `null` when a later passage stops it. A host that holds a "differs" until its sources have answered shows it only when `revisit` still decides it.
|
|
342
|
+
|
|
343
|
+
In both, every passage of `more` and `found` takes part in the veto, as `check`'s `found` does, whatever its length: the 40-word and 600-character limits hold only for a passage that may be sent or join.
|
|
344
|
+
|
|
345
|
+
Each call reads every passage once, as it is when the call begins (its text, group, title, kind and address), and never again: a host that redacts a passage in place between calls sends only the redacted text from then on, and a change made while a call runs reaches only later calls. `agree` and `revisit` read the first check's passages as they are when they are called, so a passage the host moved to another source is read in that source. A decision's `sentences` are the host's own passage objects, and its `texts` hold each one's text as the call read it, at the same index: `marks` and `pairs` index `texts`, whatever the host has done to a passage since. `texts` is the host's copy: `agree` reads the sentence of the decision it is given as the call that answered it read it (`check`'s, or `revisit`'s when the host puts that decision in the outcome's place), kept apart with the decision's `pairs`, so a host may redact or reformat `texts`, in place or in a shallow copy of the decision, without changing what `agree` answers. A decision whose `pairs` the host has replaced is read from its `texts`.
|
|
346
|
+
|
|
347
|
+
`usage` counts the judge's tokens in and out, the embeddings' tokens and the public vectors read apart; `judged` is the judge calls made (0 or 1). An abort rejects `check` with the signal's reason at once, whatever is pending; a failed embeddings call or judge rejects with its own error.
|
|
348
|
+
|
|
228
349
|
## The tool
|
|
229
350
|
|
|
230
351
|
The pack's one tool, `citation_verify`, takes `{ document, format?, allowUnreachable? }` and answers `{ fails, log, report }`. A document the parser cannot read, or arguments of the wrong type, answer `success: false` with the reason.
|
|
@@ -235,7 +356,7 @@ import { createExtensionPack } from '@framers/agentos-ext-citation-check';
|
|
|
235
356
|
const pack = createExtensionPack({ options: { embeddings: 'openai' }, getSecret: (id) => secrets[id] });
|
|
236
357
|
```
|
|
237
358
|
|
|
238
|
-
The tool reads pages through AgentOS's checked egress with the hosted policy, so a document cannot point it at a private address
|
|
359
|
+
The tool reads pages through AgentOS's checked egress with the hosted policy, so a document cannot point it at a private address. The pack needs AgentOS 0.19.4 or later: checked egress came in 0.13.120, `claimJudge` needs `generateObject`'s `maxAttempts`, which came in 0.13.158, and `ClaimChecker` needs `CitationVerifier`'s source vectors and `pickOnly`, which came in 0.19.4. Pack options: `embeddings` (`none` or `openai`, the key from the `openai.apiKey` secret or `OPENAI_API_KEY`), `embeddingModel`, `egress` (`hosted` or `local`), and from code `fetch`, `embedFn` and `nliFn`, with `timeoutMs`, `maxBytes`, `maxPages` and `priority`.
|
|
239
360
|
|
|
240
361
|
## API
|
|
241
362
|
|
|
@@ -252,9 +373,10 @@ The tool reads pages through AgentOS's checked egress with the hosted policy, so
|
|
|
252
373
|
| `readPage(url, options)`, `fetchTransport(fetch, userAgent)`, `egressTransport(profile, bounds, userAgent)`, `PageRefusal`, `UNREAD_LICENCE` | The page reader, its two transports, a transport's refusal, the licence of a page that states none |
|
|
253
374
|
| `readRobots(url, options)`, `robotsAllows(file, token, url)`, `parseRobots(text)`, `ROBOTS_PARSING_FLOOR` | robots.txt read through a transport, its answer for an address, its parser, RFC 9309's parsing floor |
|
|
254
375
|
| `readLicence(html, baseUrl)` | The licence a page's markup states |
|
|
255
|
-
| `WebPageSourceProvider`, `DEFAULT_ROBOTS` | The open web as a public source over the host's search,
|
|
376
|
+
| `WebPageSourceProvider`, `DEFAULT_ROBOTS`, `WebSearchAnswer` | The open web as a public source over the host's search, its robots.txt rule when none is given, and what the host's search answers for one page |
|
|
256
377
|
| `FirecrawlPageReader`, `FIRECRAWL_SCRAPE_URL` | The reader of a page with no text through Firecrawl's scrape endpoint, and that endpoint |
|
|
257
|
-
| `
|
|
378
|
+
| `ClaimChecker` (`check`, `agree`, `revisit`), `claimJudge(options)`, `claimJudgeSystem(kind)`, `claimJudgePrompt(heard, statement, title, sentence)`, `CLAIM_QUOTE_MAX_WORDS` | A claim heard in a room checked against passages from any source; its judge on AgentOS's `generateObject`; the judge's instruction by kind and its prompt; the most words a quote holds |
|
|
379
|
+
| `CitedClaim`, `CitationCheck`, `CitationVerdict`, `CitationPassage`, `PageStatus`, `PageFailure`, `CitationReport`, `CitationSummary`, `CitationCheckOptions`, `CitationLogger`, `EmbedFn`, `NliFn`, `PageReaderOptions`, `PageRead`, `PageFallbackReader`, `FirecrawlPageReaderOptions`, `SpokenClaim`, `ClaimPassage`, `ClaimJudge`, `ClaimJudgement`, `ClaimCheckerOptions`, `ClaimCheckOutcome`, `ClaimDecision`, `ClaimUsage`, `Transport`, `TransportAnswer`, `TransportRules`, `TransportLimits`, `RobotsRule`, `RobotsFile`, `WebPageSourceOptions` and the rest | The types |
|
|
258
380
|
|
|
259
381
|
## License
|
|
260
382
|
|
package/dist/claims.d.ts
ADDED
|
@@ -0,0 +1,386 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @file claims.ts
|
|
3
|
+
* @description A spoken claim checked against passages from any source: an encyclopedia article, a web page, a paper's
|
|
4
|
+
* abstract, a person's own notes or documents. The grounding guard's checker keeps the sentences about the claim;
|
|
5
|
+
* AgentOS's `CitationVerifier` names the best of them by cosine; one judge call, whose instruction, schema and quote
|
|
6
|
+
* rule are this pack's, says whether that sentence matches the claim, differs from it, or neither; the judge's quote is
|
|
7
|
+
* held to the sentence word for word (`verifyQuotes`); and the grounding guard decides by code over every passage
|
|
8
|
+
* found, the ones never sent to a model among them. Passages that arrive after the first decision are decided by code
|
|
9
|
+
* alone against it (`agree`), and a held "differs" is read again over every passage found by then (`revisit`).
|
|
10
|
+
*
|
|
11
|
+
* The host passes its embeddings call and its judge (`claimJudge` builds one on AgentOS's `generateObject`); the pack
|
|
12
|
+
* reads no key from the environment.
|
|
13
|
+
*
|
|
14
|
+
* @module @framers/agentos-ext-citation-check/claims
|
|
15
|
+
*/
|
|
16
|
+
import type { ClaimVerdict } from '@framers/agentos';
|
|
17
|
+
import type { Decision, ValueAgreementChecker } from '@framers/agentos-ext-grounding-guard';
|
|
18
|
+
/** The most words a judge's quote may hold. */
|
|
19
|
+
export declare const CLAIM_QUOTE_MAX_WORDS = 20;
|
|
20
|
+
/** A claim heard in a room: the words that hold it, the whole sentence they stand in, and the claim rewritten to stand alone. */
|
|
21
|
+
export interface SpokenClaim {
|
|
22
|
+
/** The words heard that hold the claim, as they stand in `heardSentence`. */
|
|
23
|
+
heard: string;
|
|
24
|
+
/** The whole sentence the words stand in, whose negations all count. */
|
|
25
|
+
heardSentence: string;
|
|
26
|
+
/** The claim rewritten to stand alone, which the verifier compares with the passages. */
|
|
27
|
+
statement: string;
|
|
28
|
+
}
|
|
29
|
+
/**
|
|
30
|
+
* One sentence of a source, with the source it belongs to and what kind of source it is. Two passages of one `group`
|
|
31
|
+
* are of one source (an article, a page, a note).
|
|
32
|
+
*/
|
|
33
|
+
export interface ClaimPassage {
|
|
34
|
+
/** The sentence, its own characters. */
|
|
35
|
+
content: string;
|
|
36
|
+
/** The source's title, which the judge reads above the sentence; empty when it should not be sent. */
|
|
37
|
+
title: string;
|
|
38
|
+
/** The source's address, when it has one. */
|
|
39
|
+
url?: string;
|
|
40
|
+
/** The source the sentence belongs to. */
|
|
41
|
+
group: string;
|
|
42
|
+
/**
|
|
43
|
+
* What kind of source it is, an open word: `encyclopedia`, `official`, `news`, `page`, `forum`, `paper`, `note`,
|
|
44
|
+
* `document`, or any other. It chooses the judge's instruction ({@link claimJudgeSystem}).
|
|
45
|
+
*/
|
|
46
|
+
kind: string;
|
|
47
|
+
/**
|
|
48
|
+
* The vector of the passage the sentence stands in, made by the check's embedding model: the sentence is ranked by
|
|
49
|
+
* it and never sent to the embeddings. Sentences of one passage share it.
|
|
50
|
+
*/
|
|
51
|
+
vector?: ClaimPassageVector;
|
|
52
|
+
/**
|
|
53
|
+
* True only for a sentence of a verified public source, whose vector may be read from and written to
|
|
54
|
+
* {@link ClaimCheckerOptions.publicVectors} when the host's `clean` test passes it.
|
|
55
|
+
*/
|
|
56
|
+
publicKey?: boolean;
|
|
57
|
+
/** The host's reference for citing the sentence, which the verdict key reads. */
|
|
58
|
+
ref?: string;
|
|
59
|
+
}
|
|
60
|
+
/** A passage's vector a host holds, with the passage's id and the hash of the input it was made from. */
|
|
61
|
+
export interface ClaimPassageVector {
|
|
62
|
+
embedding: readonly number[];
|
|
63
|
+
level: 'passage';
|
|
64
|
+
passageId: string;
|
|
65
|
+
inputHash: string;
|
|
66
|
+
}
|
|
67
|
+
/**
|
|
68
|
+
* A store of public sentences' vectors by the hash of their exact input, its tenant and model fixed by the host, such
|
|
69
|
+
* as AgentOS's `SqlEmbeddingStore` with them bound, and the host's test of what may be stored.
|
|
70
|
+
*/
|
|
71
|
+
export interface ClaimPublicVectors {
|
|
72
|
+
/** The stored vectors of the hashes given, by hash; a hash with none is left out. */
|
|
73
|
+
get(hashes: readonly string[]): Promise<ReadonlyMap<string, readonly number[]>>;
|
|
74
|
+
/** Stores each vector under its hash. */
|
|
75
|
+
put(rows: ReadonlyArray<{
|
|
76
|
+
hash: string;
|
|
77
|
+
vector: readonly number[];
|
|
78
|
+
}>): Promise<unknown>;
|
|
79
|
+
/** The host's redaction test: true when the sentence holds nothing that may not be stored or read by its hash. */
|
|
80
|
+
clean(text: string): boolean | Promise<boolean>;
|
|
81
|
+
}
|
|
82
|
+
/** What grades a check, beside this pack's and the grounding guard's versions: the judge's model and the embeddings'. */
|
|
83
|
+
export interface ClaimGrader {
|
|
84
|
+
judgeModel: string;
|
|
85
|
+
embedding: {
|
|
86
|
+
model: string;
|
|
87
|
+
dimensions: number;
|
|
88
|
+
};
|
|
89
|
+
/**
|
|
90
|
+
* The host's name for the grounding guard checker's rules (its options, such as `names`), which the verdict key
|
|
91
|
+
* reads, so a check under other rules never matches a key made under these.
|
|
92
|
+
*/
|
|
93
|
+
rules?: string;
|
|
94
|
+
}
|
|
95
|
+
/** The tokens a check used: the judge's in and out, and the embeddings' in; and the public vectors read from the store. */
|
|
96
|
+
export interface ClaimUsage {
|
|
97
|
+
judgeIn: number;
|
|
98
|
+
judgeOut: number;
|
|
99
|
+
embedIn: number;
|
|
100
|
+
vectorHits: number;
|
|
101
|
+
}
|
|
102
|
+
/** What a judge answers about one sentence: the relation, and the words of the sentence it compared. */
|
|
103
|
+
export interface ClaimJudgement {
|
|
104
|
+
relation: 'matches' | 'differs' | 'none';
|
|
105
|
+
/** Copied from the sentence, at most {@link CLAIM_QUOTE_MAX_WORDS} words; empty for `none`. */
|
|
106
|
+
quote: string;
|
|
107
|
+
}
|
|
108
|
+
/** The judge: one model call under `system` with `prompt`, ended by `signal`, answering the judgement and its tokens. */
|
|
109
|
+
export type ClaimJudge = (request: {
|
|
110
|
+
system: string;
|
|
111
|
+
prompt: string;
|
|
112
|
+
signal: AbortSignal;
|
|
113
|
+
}) => Promise<{
|
|
114
|
+
value: ClaimJudgement;
|
|
115
|
+
usage: {
|
|
116
|
+
in: number;
|
|
117
|
+
out: number;
|
|
118
|
+
};
|
|
119
|
+
}>;
|
|
120
|
+
/** What a {@link ClaimChecker} is made with. */
|
|
121
|
+
export interface ClaimCheckerOptions {
|
|
122
|
+
/** The host's embeddings call: one batch, one vector a text in order, and the tokens it used. */
|
|
123
|
+
embed: (texts: string[], signal: AbortSignal) => Promise<{
|
|
124
|
+
vectors: number[][];
|
|
125
|
+
usage: {
|
|
126
|
+
in: number;
|
|
127
|
+
};
|
|
128
|
+
}>;
|
|
129
|
+
/** The judge, such as {@link claimJudge}'s. */
|
|
130
|
+
judge: ClaimJudge;
|
|
131
|
+
/**
|
|
132
|
+
* The grounding guard's `ValueAgreementChecker`, with `names` on for a stated fact that holds no value. `stems` is
|
|
133
|
+
* read by {@link ClaimChecker.agree}.
|
|
134
|
+
*/
|
|
135
|
+
checker: Pick<ValueAgreementChecker, 'passages' | 'decide' | 'stems'>;
|
|
136
|
+
/** The most passages the verifier compares; 24 when left out. */
|
|
137
|
+
maxPassages?: number;
|
|
138
|
+
/** A store of public sentences' vectors; it needs `grader`, whose embedding names the hash. */
|
|
139
|
+
publicVectors?: ClaimPublicVectors;
|
|
140
|
+
/** What grades a check, which the verdict key and the stored vectors' hashes read. */
|
|
141
|
+
grader?: ClaimGrader;
|
|
142
|
+
/** The host's secret, from which each context's verdict key is derived; it needs `grader`. */
|
|
143
|
+
verdictSecret?: string | Uint8Array;
|
|
144
|
+
/** How long a read of `publicVectors` may take before it counts as empty, in milliseconds; 200 when left out. */
|
|
145
|
+
vectorReadMs?: number;
|
|
146
|
+
}
|
|
147
|
+
/** The grounding guard's decision, its sentences the host's own passages. */
|
|
148
|
+
export type ClaimDecision<P extends ClaimPassage = ClaimPassage> = Omit<Decision, 'sentences'> & {
|
|
149
|
+
/** The passages the relation rests on: one for "matches" and "differs", two for "disagree". */
|
|
150
|
+
sentences: P[];
|
|
151
|
+
/**
|
|
152
|
+
* Each passage's text as the call read it, at the same index as `sentences`: the text `marks` and `pairs` index, and
|
|
153
|
+
* the text that was compared, whatever the host has done to the passage since.
|
|
154
|
+
*/
|
|
155
|
+
texts: string[];
|
|
156
|
+
};
|
|
157
|
+
/**
|
|
158
|
+
* What a check came to: a decision, with the passage the verifier named, the judge's quote, the verifier's verdict, the
|
|
159
|
+
* judge calls made and the tokens used; a verdict the host already holds; or why nothing may be shown.
|
|
160
|
+
*
|
|
161
|
+
* - `no_passage`: no passage given is about the claim; nothing was sent to a model;
|
|
162
|
+
* - `limited`: `beforeSend` did not answer true; nothing was sent;
|
|
163
|
+
* - `no_best`: the verifier named no passage (no cosine above 0);
|
|
164
|
+
* - `judge_none`: the judge answered `none`;
|
|
165
|
+
* - `quote`: the judge's quote is empty, longer than {@link CLAIM_QUOTE_MAX_WORDS} words, or does not stand word for
|
|
166
|
+
* word in the sentence;
|
|
167
|
+
* - `undecided`: the grounding guard's rule decided nothing, a passage found that agrees with every heard value among
|
|
168
|
+
* the reasons.
|
|
169
|
+
*
|
|
170
|
+
* With `verdictSecret`, every outcome past `no_passage` carries `key`, the verdict key; `known` answers a key the host
|
|
171
|
+
* passed in `known`, with nothing sent.
|
|
172
|
+
*/
|
|
173
|
+
export type ClaimCheckOutcome<P extends ClaimPassage = ClaimPassage> = {
|
|
174
|
+
kind: 'decided';
|
|
175
|
+
decision: ClaimDecision<P>;
|
|
176
|
+
named: P;
|
|
177
|
+
quote: string;
|
|
178
|
+
verdict: ClaimVerdict;
|
|
179
|
+
judged: number;
|
|
180
|
+
usage: ClaimUsage;
|
|
181
|
+
key?: string;
|
|
182
|
+
} | {
|
|
183
|
+
kind: 'ended';
|
|
184
|
+
reason: 'no_passage' | 'limited' | 'no_best' | 'judge_none' | 'quote' | 'undecided';
|
|
185
|
+
judged: number;
|
|
186
|
+
usage: ClaimUsage;
|
|
187
|
+
key?: string;
|
|
188
|
+
} | {
|
|
189
|
+
kind: 'known';
|
|
190
|
+
key: string;
|
|
191
|
+
judged: 0;
|
|
192
|
+
usage: ClaimUsage;
|
|
193
|
+
};
|
|
194
|
+
/**
|
|
195
|
+
* The judge's instruction for a sentence from a source of this kind: whether the sentence states the same fact as the
|
|
196
|
+
* claim (the same value, holder, place, maker or date), a different one, or neither, with the words it compared quoted
|
|
197
|
+
* from the sentence. Every instruction opens with "You compare one sentence heard in a room with one sentence from",
|
|
198
|
+
* then what the sentence is from: "an encyclopedia article" for `encyclopedia`; "a web page" for `official`, `news`,
|
|
199
|
+
* `page` and `forum`; "a paper's abstract" for `paper`; "the person's own notes or documents" for `note` and
|
|
200
|
+
* `document`; "a source" for any other word.
|
|
201
|
+
*
|
|
202
|
+
* @param kind - The named passage's kind ({@link ClaimPassage.kind}).
|
|
203
|
+
*/
|
|
204
|
+
export declare function claimJudgeSystem(kind: string): string;
|
|
205
|
+
/**
|
|
206
|
+
* The judge's prompt: the heard sentence, the claim, and the one sentence with its source's title above it.
|
|
207
|
+
*
|
|
208
|
+
* @param heard - The whole sentence heard.
|
|
209
|
+
* @param statement - The claim rewritten to stand alone.
|
|
210
|
+
* @param title - The source's title; empty when it should not be sent.
|
|
211
|
+
* @param sentence - The source's sentence.
|
|
212
|
+
*/
|
|
213
|
+
export declare function claimJudgePrompt(heard: string, statement: string, title: string, sentence: string): string;
|
|
214
|
+
/**
|
|
215
|
+
* A judge on AgentOS's `generateObject`, with this pack's schema: one call, sent once (`maxAttempts: 1`), with no
|
|
216
|
+
* second call for an answer that is not the schema (`maxRetries: 0`) and no other provider (`fallbackProviders: []`),
|
|
217
|
+
* ended at `deadlineMs` or by the caller's signal, whichever comes first, the body's read included.
|
|
218
|
+
*
|
|
219
|
+
* @param options - The model (`provider:model`, such as `openai:gpt-6-luna`), the key, the base address (a stand-in in
|
|
220
|
+
* a host's tests), each call's deadline, the most tokens it may answer, and the reasoning effort for a model that takes
|
|
221
|
+
* one.
|
|
222
|
+
* @returns The judge. It rejects with the caller's reason when the caller's signal aborts, with a timeout when the
|
|
223
|
+
* deadline passes, and with the provider's error otherwise.
|
|
224
|
+
*/
|
|
225
|
+
export declare function claimJudge(options: {
|
|
226
|
+
model: string;
|
|
227
|
+
apiKey: string;
|
|
228
|
+
baseUrl?: string;
|
|
229
|
+
deadlineMs: number;
|
|
230
|
+
maxOutputTokens: number;
|
|
231
|
+
effort?: string;
|
|
232
|
+
}): ClaimJudge;
|
|
233
|
+
/**
|
|
234
|
+
* Checks spoken claims against passages from any source, as `check` says, and decides later passages by code against a
|
|
235
|
+
* first decision. One checker serves many checks: it keeps nothing of a check but what `agree` and `revisit` read
|
|
236
|
+
* again, held weakly by the outcome `check` answered.
|
|
237
|
+
*
|
|
238
|
+
* @example
|
|
239
|
+
* ```ts
|
|
240
|
+
* import { ClaimChecker, claimJudge } from '@framers/agentos-ext-citation-check';
|
|
241
|
+
* import { ValueAgreementChecker } from '@framers/agentos-ext-grounding-guard';
|
|
242
|
+
*
|
|
243
|
+
* const checker = new ClaimChecker({
|
|
244
|
+
* embed: async (texts, signal) => myEmbeddings(texts, signal), // { vectors, usage: { in } }
|
|
245
|
+
* judge: claimJudge({ model: 'openai:gpt-6-luna', apiKey, deadlineMs: 8_000, maxOutputTokens: 400 }),
|
|
246
|
+
* checker: new ValueAgreementChecker({ names: true }),
|
|
247
|
+
* });
|
|
248
|
+
* const outcome = await checker.check(
|
|
249
|
+
* { heard, heardSentence, statement },
|
|
250
|
+
* passages, // ClaimPassage[]: { content, title, url?, group, kind }
|
|
251
|
+
* { signal },
|
|
252
|
+
* );
|
|
253
|
+
* ```
|
|
254
|
+
*/
|
|
255
|
+
export declare class ClaimChecker {
|
|
256
|
+
private readonly embed;
|
|
257
|
+
private readonly judge;
|
|
258
|
+
private readonly checker;
|
|
259
|
+
private readonly maxPassages;
|
|
260
|
+
private readonly publicVectors;
|
|
261
|
+
private readonly grader;
|
|
262
|
+
private readonly verdictSecret;
|
|
263
|
+
private readonly vectorReadMs;
|
|
264
|
+
/**
|
|
265
|
+
* What a decided check keeps for `agree` and `revisit`, by the outcome `check` answered: every passage it found, the
|
|
266
|
+
* held-back ones among them.
|
|
267
|
+
*/
|
|
268
|
+
private readonly decided;
|
|
269
|
+
/**
|
|
270
|
+
* The texts each decision this checker answered was decided on (`check`'s and `revisit`'s alike), which the host's
|
|
271
|
+
* copy in `decision.texts` may since have been changed from. Kept by the decision's `pairs`, the offsets that index
|
|
272
|
+
* them: every decision answered holds its own copy of that array, which a host's shallow copy of the decision keeps.
|
|
273
|
+
*/
|
|
274
|
+
private readonly decisionTexts;
|
|
275
|
+
/**
|
|
276
|
+
* @param options - The embeddings call, the judge, the grounding guard's checker, the most passages compared, and
|
|
277
|
+
* the store of public vectors, the grader and the verdict secret.
|
|
278
|
+
* @throws {RangeError} For a `maxPassages` that is not a whole number of at least 1, a `vectorReadMs` that is not a
|
|
279
|
+
* positive number, or a grader whose dimensions are not a whole number of at least 1.
|
|
280
|
+
* @throws {TypeError} For `publicVectors` or `verdictSecret` without `grader`.
|
|
281
|
+
*/
|
|
282
|
+
constructor(options: ClaimCheckerOptions);
|
|
283
|
+
/** A decision with the host's passages in place of the grounding guard's sentences, and their texts as read. */
|
|
284
|
+
private toHost;
|
|
285
|
+
/** Every passage found for a decided check; the decision's own passages when the outcome is not one `check` answered. */
|
|
286
|
+
private foundOf;
|
|
287
|
+
/**
|
|
288
|
+
* The verdict key: an HMAC-SHA-256, under a key derived by HKDF-SHA-256 from the secret with the context, over a
|
|
289
|
+
* canonical encoding of everything that decides the verdict. That is the grader (this pack's and the grounding guard's
|
|
290
|
+
* versions, the judge's model, the embedding model and dimensions, `maxPassages`), the claim, each compared sentence
|
|
291
|
+
* in order with its text, title, kind, group and `ref` and, for one ranked by a passage vector, the passage's id, its
|
|
292
|
+
* `inputHash` and a hash of the vector, and each sentence found with its text, group, `ref` and whether it is held
|
|
293
|
+
* back. `undefined` without a secret.
|
|
294
|
+
*/
|
|
295
|
+
private verdictKey;
|
|
296
|
+
/**
|
|
297
|
+
* The sentences whose vectors may be read from and written to the public store: those marked `publicKey` that the
|
|
298
|
+
* host's `clean` passes. A `clean` that throws refuses its sentence.
|
|
299
|
+
*/
|
|
300
|
+
private storable;
|
|
301
|
+
/** The stored vectors of `storable`, by sentence; a store that rejects or answers after `vectorReadMs` gives none. */
|
|
302
|
+
private readPublic;
|
|
303
|
+
/** Writes the new public vectors, waiting at most `vectorReadMs`; a failed or late write changes nothing. */
|
|
304
|
+
private writePublic;
|
|
305
|
+
/**
|
|
306
|
+
* Checks a claim against passages, in order:
|
|
307
|
+
*
|
|
308
|
+
* 1. the grounding guard's `passages` keeps the sentences of `given` about the claim; none ends `no_passage`, with
|
|
309
|
+
* nothing sent anywhere. The first `maxPassages` kept, one a text (a text two sources share is compared as the
|
|
310
|
+
* first of them, whose title and kind the judge then reads), are the ones compared;
|
|
311
|
+
* 2. with `verdictSecret`, the verdict key is made; a key in `options.known` answers `known`, with nothing sent;
|
|
312
|
+
* 3. `options.beforeSend` is awaited once, before anything is sent; unless it answers true the check ends `limited`;
|
|
313
|
+
* 4. one sentence compared is named with no embeddings call (AgentOS's `CitationVerifier` with `pickOnly`), its
|
|
314
|
+
* verdict's `confidence` 0 and `scored` false;
|
|
315
|
+
* 5. otherwise the claim is embedded in one call with every compared sentence that carries no passage vector and has
|
|
316
|
+
* no stored public vector. A sentence marked `publicKey` that the host's `clean` passes is looked up in
|
|
317
|
+
* `publicVectors` first; a store that rejects or answers after `vectorReadMs` counts as empty. Sentence vectors and
|
|
318
|
+
* passage vectors are not on one scale, so the verifier ranks each class by its own cosine, and of the two classes'
|
|
319
|
+
* best the one earlier among the compared sentences is named; a sentence whose cosine is 0 or less is never named,
|
|
320
|
+
* and none ends `no_best`. The public vectors made are then stored, a failed or late write changing nothing;
|
|
321
|
+
* 6. the judge is asked once about the named sentence, under {@link claimJudgeSystem} for its kind, with
|
|
322
|
+
* {@link claimJudgePrompt}; `none` ends `judge_none`;
|
|
323
|
+
* 7. the judge's quote must hold one to {@link CLAIM_QUOTE_MAX_WORDS} words and stand in the sentence word for word
|
|
324
|
+
* (`verifyQuotes` with `fold`); else it ends `quote`;
|
|
325
|
+
* 8. the grounding guard's `decide` takes the judge's relation as its proposal, the named sentence, the quote, and as
|
|
326
|
+
* `found` every sentence kept in step 1 and every one of `options.found`; `null` ends `undecided`.
|
|
327
|
+
*
|
|
328
|
+
* `options.found` is every passage about the claim the host holds, the ones it may not send included (a sentence that
|
|
329
|
+
* holds personal data, for example): they are never embedded, judged or sent, and take part in step 8 alone, where a
|
|
330
|
+
* passage that agrees with every heard value stops a "differs". Left out, it is `given`'s sentences about the claim.
|
|
331
|
+
*
|
|
332
|
+
* @param heard - The claim heard, read once when the call begins.
|
|
333
|
+
* @param given - The passages that may be sent to the embeddings and the judge.
|
|
334
|
+
* @param options - The check's signal; every passage found; the verdict keys the host holds; the host's say before
|
|
335
|
+
* anything is sent; and the context the verdict key is derived for, such as the account's and the run's ids.
|
|
336
|
+
* @throws The signal's reason when it aborts, at once, whatever is pending; the embeddings' or the judge's error as it
|
|
337
|
+
* came, which the verifier alone would have swallowed; `beforeSend`'s error; a `RangeError` for a passage vector of
|
|
338
|
+
* another size than the claim's.
|
|
339
|
+
*/
|
|
340
|
+
check<P extends ClaimPassage>(heard: SpokenClaim, given: readonly P[], options: {
|
|
341
|
+
signal: AbortSignal;
|
|
342
|
+
found?: readonly P[];
|
|
343
|
+
known?: Iterable<string>;
|
|
344
|
+
beforeSend?: () => boolean | Promise<boolean>;
|
|
345
|
+
verdictContext?: string;
|
|
346
|
+
}): Promise<ClaimCheckOutcome<P>>;
|
|
347
|
+
/**
|
|
348
|
+
* The passages that arrived after a first decision and agree with it, decided by code alone, with no model call. Each
|
|
349
|
+
* source of `more` (by `group`) other than the first decision's is read once, by its passage the grounding guard
|
|
350
|
+
* ranks first about the claim, which joins when:
|
|
351
|
+
*
|
|
352
|
+
* 1. it holds every content stem of the heard words;
|
|
353
|
+
* 2. the grounding guard decides the first decision's relation for it, on the values, over every passage found (the
|
|
354
|
+
* first check's and every one of `more`, whatever its length);
|
|
355
|
+
* 3. its first value pair pairs the same heard value as the first decision's;
|
|
356
|
+
* 4. for a "differs", its marked value is the first decision's as written; for a "matches", each of the first
|
|
357
|
+
* decision's pairs finds its partner written alike, the decision's sentence read as the call that answered it
|
|
358
|
+
* (`check`, or `revisit` for a decision the host put in the outcome's place) read it, whatever the host has
|
|
359
|
+
* written into `decision.texts` since.
|
|
360
|
+
*
|
|
361
|
+
* A "disagree", a decision on the judge's quote alone, and a decision with no value pair take no passage.
|
|
362
|
+
*
|
|
363
|
+
* @param first - The decided outcome `check` answered.
|
|
364
|
+
* @param claim - The claim it checked.
|
|
365
|
+
* @param more - The later passages.
|
|
366
|
+
* @returns The passages that join, in the grounding guard's order.
|
|
367
|
+
*/
|
|
368
|
+
agree<P extends ClaimPassage>(first: Extract<ClaimCheckOutcome<P>, {
|
|
369
|
+
kind: 'decided';
|
|
370
|
+
}>, claim: SpokenClaim, more: readonly P[]): P[];
|
|
371
|
+
/**
|
|
372
|
+
* The first decision read again over every passage found by then: the grounding guard's rule with the same proposal,
|
|
373
|
+
* named passage and quote, its `found` every passage the check found (held-back ones among them) and every passage of
|
|
374
|
+
* `found`, as `check` reads its own `found`, whatever its length. A "differs" a later passage stops (one that agrees with every heard value) answers `null`,
|
|
375
|
+
* so a host that holds a "differs" until its slower sources have answered shows nothing.
|
|
376
|
+
*
|
|
377
|
+
* @param first - The decided outcome `check` answered.
|
|
378
|
+
* @param claim - The claim it checked.
|
|
379
|
+
* @param found - Every passage found by then.
|
|
380
|
+
* @returns The decision, its relation the first's, or `null`.
|
|
381
|
+
*/
|
|
382
|
+
revisit<P extends ClaimPassage>(first: Extract<ClaimCheckOutcome<P>, {
|
|
383
|
+
kind: 'decided';
|
|
384
|
+
}>, claim: SpokenClaim, found: readonly P[]): ClaimDecision<P> | null;
|
|
385
|
+
}
|
|
386
|
+
//# sourceMappingURL=claims.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"claims.d.ts","sourceRoot":"","sources":["../src/claims.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;GAcG;AAMH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AACrD,OAAO,KAAK,EAAE,QAAQ,EAAkB,qBAAqB,EAAE,MAAM,sCAAsC,CAAC;AAI5G,+CAA+C;AAC/C,eAAO,MAAM,qBAAqB,KAAK,CAAC;AAExC,iIAAiI;AACjI,MAAM,WAAW,WAAW;IAC1B,6EAA6E;IAC7E,KAAK,EAAE,MAAM,CAAC;IACd,wEAAwE;IACxE,aAAa,EAAE,MAAM,CAAC;IACtB,yFAAyF;IACzF,SAAS,EAAE,MAAM,CAAC;CACnB;AAED;;;GAGG;AACH,MAAM,WAAW,YAAY;IAC3B,wCAAwC;IACxC,OAAO,EAAE,MAAM,CAAC;IAChB,sGAAsG;IACtG,KAAK,EAAE,MAAM,CAAC;IACd,6CAA6C;IAC7C,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,0CAA0C;IAC1C,KAAK,EAAE,MAAM,CAAC;IACd;;;OAGG;IACH,IAAI,EAAE,MAAM,CAAC;IACb;;;OAGG;IACH,MAAM,CAAC,EAAE,kBAAkB,CAAC;IAC5B;;;OAGG;IACH,SAAS,CAAC,EAAE,OAAO,CAAC;IACpB,iFAAiF;IACjF,GAAG,CAAC,EAAE,MAAM,CAAC;CACd;AAED,yGAAyG;AACzG,MAAM,WAAW,kBAAkB;IACjC,SAAS,EAAE,SAAS,MAAM,EAAE,CAAC;IAC7B,KAAK,EAAE,SAAS,CAAC;IACjB,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,EAAE,MAAM,CAAC;CACnB;AAED;;;GAGG;AACH,MAAM,WAAW,kBAAkB;IACjC,qFAAqF;IACrF,GAAG,CAAC,MAAM,EAAE,SAAS,MAAM,EAAE,GAAG,OAAO,CAAC,WAAW,CAAC,MAAM,EAAE,SAAS,MAAM,EAAE,CAAC,CAAC,CAAC;IAChF,yCAAyC;IACzC,GAAG,CAAC,IAAI,EAAE,aAAa,CAAC;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,SAAS,MAAM,EAAE,CAAA;KAAE,CAAC,GAAG,OAAO,CAAC,OAAO,CAAC,CAAC;IACxF,kHAAkH;IAClH,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC,CAAC;CACjD;AAED,yHAAyH;AACzH,MAAM,WAAW,WAAW;IAC1B,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAA;KAAE,CAAC;IACjD;;;OAGG;IACH,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,2HAA2H;AAC3H,MAAM,WAAW,UAAU;IACzB,OAAO,EAAE,MAAM,CAAC;IAChB,QAAQ,EAAE,MAAM,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;IAChB,UAAU,EAAE,MAAM,CAAC;CACpB;AAED,wGAAwG;AACxG,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,SAAS,GAAG,SAAS,GAAG,MAAM,CAAC;IACzC,+FAA+F;IAC/F,KAAK,EAAE,MAAM,CAAC;CACf;AAED,yHAAyH;AACzH,MAAM,MAAM,UAAU,GAAG,CAAC,OAAO,EAAE;IAAE,MAAM,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,WAAW,CAAA;CAAE,KAAK,OAAO,CAAC;IACrG,KAAK,EAAE,cAAc,CAAC;IACtB,KAAK,EAAE;QAAE,EAAE,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAA;KAAE,CAAC;CACpC,CAAC,CAAC;AAEH,gDAAgD;AAChD,MAAM,WAAW,mBAAmB;IAClC,iGAAiG;IACjG,KAAK,EAAE,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,MAAM,EAAE,WAAW,KAAK,OAAO,CAAC;QAAE,OAAO,EAAE,MAAM,EAAE,EAAE,CAAC;QAAC,KAAK,EAAE;YAAE,EAAE,EAAE,MAAM,CAAA;SAAE,CAAA;KAAE,CAAC,CAAC;IACzG,+CAA+C;IAC/C,KAAK,EAAE,UAAU,CAAC;IAClB;;;OAGG;IACH,OAAO,EAAE,IAAI,CAAC,qBAAqB,EAAE,UAAU,GAAG,QAAQ,GAAG,OAAO,CAAC,CAAC;IACtE,iEAAiE;IACjE,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,+FAA+F;IAC/F,aAAa,CAAC,EAAE,kBAAkB,CAAC;IACnC,sFAAsF;IACtF,MAAM,CAAC,EAAE,WAAW,CAAC;IACrB,8FAA8F;IAC9F,aAAa,CAAC,EAAE,MAAM,GAAG,UAAU,CAAC;IACpC,iHAAiH;IACjH,YAAY,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,6EAA6E;AAC7E,MAAM,MAAM,aAAa,CAAC,CAAC,SAAS,YAAY,GAAG,YAAY,IAAI,IAAI,CAAC,QAAQ,EAAE,WAAW,CAAC,GAAG;IAC/F,+FAA+F;IAC/F,SAAS,EAAE,CAAC,EAAE,CAAC;IACf;;;OAGG;IACH,KAAK,EAAE,MAAM,EAAE,CAAC;CACjB,CAAC;AAEF;;;;;;;;;;;;;;;GAeG;AACH,MAAM,MAAM,iBAAiB,CAAC,CAAC,SAAS,YAAY,GAAG,YAAY,IAC/D;IAAE,IAAI,EAAE,SAAS,CAAC;IAAC,QAAQ,EAAE,aAAa,CAAC,CAAC,CAAC,CAAC;IAAC,KAAK,EAAE,CAAC,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,YAAY,CAAC;IAAC,MAAM,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,UAAU,CAAC;IAAC,GAAG,CAAC,EAAE,MAAM,CAAA;CAAE,GAChJ;IAAE,IAAI,EAAE,OAAO,CAAC;IAAC,MAAM,EAAE,YAAY,GAAG,SAAS,GAAG,SAAS,GAAG,YAAY,GAAG,OAAO,GAAG,WAAW,CAAC;IAAC,MAAM,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,UAAU,CAAC;IAAC,GAAG,CAAC,EAAE,MAAM,CAAA;CAAE,GACvJ;IAAE,IAAI,EAAE,OAAO,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,CAAC,CAAC;IAAC,KAAK,EAAE,UAAU,CAAA;CAAE,CAAC;AA+GjE;;;;;;;;;GASG;AACH,wBAAgB,gBAAgB,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAWrD;AAED;;;;;;;GAOG;AACH,wBAAgB,gBAAgB,CAAC,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,GAAG,MAAM,CAE1G;AA+BD;;;;;;;;;;GAUG;AACH,wBAAgB,UAAU,CAAC,OAAO,EAAE;IAClC,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,eAAe,EAAE,MAAM,CAAC;IACxB,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB,GAAG,UAAU,CA6Bb;AAmED;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,qBAAa,YAAY;IACvB,OAAO,CAAC,QAAQ,CAAC,KAAK,CAA+B;IACrD,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAa;IACnC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAiC;IACzD,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAS;IACrC,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAiC;IAC/D,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA0B;IACjD,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAyB;IACvD,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAS;IACtC;;;OAGG;IACH,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAoD;IAC5E;;;;OAIG;IACH,OAAO,CAAC,QAAQ,CAAC,aAAa,CAA4C;IAE1E;;;;;;OAMG;gBACS,OAAO,EAAE,mBAAmB;IAiCxC,gHAAgH;IAChH,OAAO,CAAC,MAAM;IAad,yHAAyH;IACzH,OAAO,CAAC,OAAO;IAKf;;;;;;;OAOG;IACH,OAAO,CAAC,UAAU;IA6BlB;;;OAGG;YACW,QAAQ;IAmBtB,sHAAsH;YACxG,UAAU;IAqBxB,6GAA6G;YAC/F,WAAW;IASzB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;OAkCG;IACG,KAAK,CAAC,CAAC,SAAS,YAAY,EAChC,KAAK,EAAE,WAAW,EAClB,KAAK,EAAE,SAAS,CAAC,EAAE,EACnB,OAAO,EAAE;QACP,MAAM,EAAE,WAAW,CAAC;QACpB,KAAK,CAAC,EAAE,SAAS,CAAC,EAAE,CAAC;QACrB,KAAK,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC;QACzB,UAAU,CAAC,EAAE,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC,CAAC;QAC9C,cAAc,CAAC,EAAE,MAAM,CAAC;KACzB,GACA,OAAO,CAAC,iBAAiB,CAAC,CAAC,CAAC,CAAC;IAkOhC;;;;;;;;;;;;;;;;;;;;OAoBG;IACH,KAAK,CAAC,CAAC,SAAS,YAAY,EAAE,KAAK,EAAE,OAAO,CAAC,iBAAiB,CAAC,CAAC,CAAC,EAAE;QAAE,IAAI,EAAE,SAAS,CAAA;KAAE,CAAC,EAAE,KAAK,EAAE,WAAW,EAAE,IAAI,EAAE,SAAS,CAAC,EAAE,GAAG,CAAC,EAAE;IA0CrI;;;;;;;;;;OAUG;IACH,OAAO,CAAC,CAAC,SAAS,YAAY,EAAE,KAAK,EAAE,OAAO,CAAC,iBAAiB,CAAC,CAAC,CAAC,EAAE;QAAE,IAAI,EAAE,SAAS,CAAA;KAAE,CAAC,EAAE,KAAK,EAAE,WAAW,EAAE,KAAK,EAAE,SAAS,CAAC,EAAE,GAAG,aAAa,CAAC,CAAC,CAAC,GAAG,IAAI;CAW7J"}
|