@framers/agentos-ext-citation-check 0.1.4 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -39,7 +39,7 @@ and exits 1. The output is the markdown for a document's "Citation check log" se
39
39
  npm install @framers/agentos-ext-citation-check
40
40
  ```
41
41
 
42
- It needs Node.js 22 or later, as `@framers/agentos` does. It reads pages with [public-sources](https://www.npmjs.com/package/@framers/agentos-ext-public-sources) (`htmlToBlocks` in page mode and `citationSources`), the [web scraper](https://www.npmjs.com/package/@framers/agentos-ext-web-scraper) (a browser's request headers and the fallback tiers) and the `values` entry of [grounding-guard](https://www.npmjs.com/package/@framers/agentos-ext-grounding-guard) (the sentence rule and the word rules). `@framers/agentos` 0.13.120 or later, the first release with the checked egress the tool reads pages through, is a peer, loaded only to grade with embeddings or to read pages through checked egress. Grounding-guard lists `@huggingface/transformers` as an optional dependency for its NLI model; `npm install --omit=optional` leaves it out when you do not use NLI.
42
+ It needs Node.js 22 or later, as `@framers/agentos` does. It reads pages with [public-sources](https://www.npmjs.com/package/@framers/agentos-ext-public-sources) (`htmlToBlocks` in page mode, `citationSources` and the `SourceDocument` a page answers as, from 0.2.3), the [web scraper](https://www.npmjs.com/package/@framers/agentos-ext-web-scraper) (a browser's request headers and the fallback tiers) the `values` entry of [grounding-guard](https://www.npmjs.com/package/@framers/agentos-ext-grounding-guard) (the sentence rule and the word rules) and [entities](https://www.npmjs.com/package/entities) (the character references in a page's title, names and addresses). `@framers/agentos` 0.13.120 or later, the first release with the checked egress the tool reads pages through, is a peer, loaded only to grade with embeddings or to read pages through checked egress. Grounding-guard lists `@huggingface/transformers` as an optional dependency for its NLI model; `npm install --omit=optional` leaves it out when you do not use NLI.
43
43
 
44
44
  ## Input
45
45
 
@@ -53,7 +53,7 @@ A guide notes that "the lighthouse stands 40 metres tall" (<https://example.org/
53
53
  The page lists "Five berths", and "Two cranes lift the containers" (https://example.org/port).
54
54
  ```
55
55
 
56
- Several quotations waiting in a sentence share the address that follows them, as the last line's two do. An address ends at white space or a quotation mark, without the punctuation after it and without a closing bracket it did not open, so `(https://example.org/wiki/River_(Delta), read 1 Mar 2026)` gives `https://example.org/wiki/River_(Delta)`. A quotation of four words or more that no address follows before its sentence ends is reported as uncited; a shorter one reads as a term or a title and is passed over. Quotations in fenced code and code spans, and escaped quotation marks (`\"`), are not claims.
56
+ Several quotations waiting in a sentence share the address that follows them, as the last line's two do. An address ends at white space or a quotation mark, without the punctuation after it and without a closing bracket it did not open, so `(https://example.org/wiki/River_(Delta), read 1 Mar 2026)` gives `https://example.org/wiki/River_(Delta)`. A quotation of four words or more that no address follows before its sentence ends is reported as uncited; a shorter one reads as a term or a title and is passed over. Quotations in fenced code and code spans, escaped quotation marks (`\"`) and a markdown link's title (`[the log](https://example.org/log "Title")`) are not claims. A quotation's paired emphasis marks (`*only*`, `_only_`, `**only**`), across a line break too, and backticks are removed from its words; an unpaired mark (`2*3`, `char*`), an underscore inside a word (`snake_case`) and the marks inside a code span (`` `*cat*` `` reads `*cat*`) stay.
57
57
 
58
58
  A JSON document: an array of claims.
59
59
 
@@ -81,7 +81,7 @@ A JSON document: an array of claims.
81
81
  | `unreadable` | The page came but holds no text this check reads: a PDF, another binary type, a body past the size cap, or no text a reader sees |
82
82
  | `uncited` | The claim cites no address |
83
83
 
84
- Words are compared after both sides are read the same way: compatibility forms (NFKC), typographic quotation marks as plain ones, every dash a hyphen with no space around it, case, white space, and soft hyphens and zero-width characters. An ellipsis or words in square brackets (`[…]`, `[It]`) in a quotation are gaps: the pieces between them must stand on the page in order, all in one block of its text (a paragraph, a list item, a table cell, a caption or a quotation), so a sentence of one paragraph and a cell further down the page never pass as one quotation. A quotation without gaps may run on from one block into the next, as across a line break. A match never cuts a word or a number, so "rose to 5" does not stand in "rose to 50 percent". Each check carries the sentence its verdict rests on (for an exact quotation that runs past one sentence, its block), and a `not_found` claim the three sentences nearest it by the share of content words and numbers the two have in common.
84
+ Words are compared after both sides are read the same way: compatibility forms (NFKC), typographic quotation marks as plain ones, every dash a hyphen with no space around it (except one with white space after it before a digit, which keeps that space, so in "opened in 1887 — 4,200 ships" the number reads with no minus sign), case, white space, and soft hyphens and zero-width characters. An ellipsis or words in square brackets (`[…]`, `[It]`) in a quotation are gaps: the pieces between them must stand on the page in order, all in one block of its text (a paragraph, a list item, a table cell, a caption or a quotation), so a sentence of one paragraph and a cell further down the page never pass as one quotation. A quotation without gaps may run on from one block into the next, as across a line break. A match never cuts a word or a number, so "rose to 5" does not stand in "rose to 50 percent". Each check carries the sentence its verdict rests on (for an exact quotation that runs past one sentence, its block), and a `not_found` claim the three sentences nearest it by the share of content words and numbers the two have in common.
85
85
 
86
86
  `citationReportFails(report)` is true when a claim is `contradicted`, `not_found` or `unverifiable`, or when its page is `unreachable` or `unreadable`; `{ allowUnreachable: true }` leaves the last two out.
87
87
 
@@ -109,7 +109,7 @@ if (citationReportFails(report)) process.exitCode = 1;
109
109
 
110
110
  ### Reading pages
111
111
 
112
- Each distinct address is fetched once, its fragment removed, four at a time. A request sends a browser's headers from the web scraper pack, follows redirects, and stops at the deadline (`timeoutMs`, 15,000) or past the size cap (`maxBytes`, 5,000,000). HTML and XML are read with public-sources' `htmlToBlocks` in page mode, which keeps the tables, figures, quotations and code a page shows; other text by paragraphs; a PDF or another binary type not at all. The charset comes from a byte order mark, the content type or a `<meta>` tag. A page that times out, does not connect, answers an error status other than 404 and 410, or holds no text is retried once through the web scraper's `WebScraperService`, whose tiers run a headless browser when `playwright-core` is installed and a second plain request otherwise. A page that fails is a verdict, never an exception.
112
+ Each distinct address is fetched once, its fragment removed, four at a time. A request sends a browser's headers from the web scraper pack, follows up to five redirects (a sixth is the failure `redirects`), and stops at the deadline (`timeoutMs`, 15,000) or past the size cap (`maxBytes`, 5,000,000). HTML and XML are read with public-sources' `htmlToBlocks` in page mode, which keeps the tables, figures, quotations and code a page shows; other text by paragraphs; a PDF or another binary type not at all. The charset comes from a byte order mark, the content type or a `<meta>` tag; a body with no content type is read by its first bytes, a UTF-16 one after its byte order mark. A page that times out, does not connect, answers an error status other than 404 and 410, or holds no text is retried once through the web scraper's `WebScraperService`, whose tiers run a headless browser when `playwright-core` is installed and a second plain request otherwise. A page that fails is a verdict, never an exception.
113
113
 
114
114
  | Option | Default | What it does |
115
115
  | --- | --- | --- |
@@ -123,6 +123,94 @@ Each distinct address is fetched once, its fragment removed, four at a time. A r
123
123
 
124
124
  The global `fetch` reaches any address a document names. A host that checks documents from people it does not trust passes `egress: 'hosted'`, which reaches public https addresses on port 443 alone and checks every redirect, or a `fetch` of its own. The fallback's requests go through neither, so it is off once either is given unless `fallback: true` turns it on.
125
125
 
126
+ ### The page reader
127
+
128
+ `readPage(url, options)` is the reader `checkCitations` runs, exported for a host that reads pages of its own. It answers `{ status, blocks, passages, document, pageDate }`: the page's status as above, its text blocks, its sentences as passages, the page as a public-sources `SourceDocument` (null when it was not read) and the page's own date (null when it states none). It throws only the caller's own cancellation.
129
+
130
+ ```ts
131
+ import { egressTransport, readPage } from '@framers/agentos-ext-citation-check';
132
+
133
+ const transport = await egressTransport('hosted', { timeoutMs: 5_000, maxBytes: 2_000_000 }, 'ExampleReader/1.0 (https://example.org)');
134
+ const page = await readPage('https://example.org/harbour', {
135
+ transport,
136
+ scraper: null,
137
+ timeoutMs: 5_000,
138
+ maxBytes: 2_000_000,
139
+ textBudget: 300_000,
140
+ now: Date.now,
141
+ allowUrl: (url) => !new URL(url).hostname.endsWith('.internal.example'),
142
+ robots: { token: 'ExampleReader', timeoutMs: 2_000, maxBytes: 512_000, cacheMs: 600_000 },
143
+ });
144
+ if (page.document !== null) console.log(page.document.title, page.document.licence.name, page.document.publisher);
145
+ ```
146
+
147
+ `fetchTransport(fetch, userAgent?)` and `egressTransport(profile, bounds, userAgent?)` build the request a page goes through. A transport takes `(url, signal, rules?)` and answers `{ status, finalUrl, header, read, readPrefix, discard }`; a host can pass one of its own.
148
+
149
+ The document's `url` and `key` are the address after redirects. Its `title` is the page's `<title>`, its character references read, its white space collapsed and cut to 255 characters, or the host when the page has none. Its `licence` is the one the page states (below), or `{ name: 'not read', url: '' }` (`UNREAD_LICENCE`) when the reader finds none. `publisher` is the site's name from `og:site_name`, else from JSON-LD `publisher.name`, cut to 255 characters and left out when the page names none. `pageDate` is the page's own date from `article:modified_time`, else JSON-LD `dateModified`, else the `Last-Modified` header, left out when none parses. The markup is read as served, in one pass in document order: a value a script writes later is not seen, a tag or a JSON-LD block inside a comment is passed over, the content of a script, a style, a template, a `noscript` and a `textarea` is not read for tags, and the time grows with the page alone however its tags are left open. Of the links and anchors the pass keeps only those a licence is read from, at most 1,000 of each kind, and reads the text of a `rel="license"` anchor alone, so a page of a million other anchors holds none of them.
150
+
151
+ #### The host's rule and robots.txt, on every hop
152
+
153
+ `allowUrl(url)` is the host's rule for an address. `robots` reads robots.txt by [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) with the host's product token. Given either, the reader checks the first address before it is requested, and both transports check every redirect target before its request, following up to five redirects one at a time. A refused address ends the page as `unreachable` with `failure: 'refused'` and `detail: 'host'` or `detail: 'robots'`, and is never requested. A redirect to an address that is not http or https, such as a `data:` address, ends the page with `detail: 'scheme'`, as `fetch` itself ends one. The host's rule is asked once an address. The web scraper's retry is not used under either rule, since its requests go through neither. A transport called with the rules applies them the same way, so an image or an icon read through it obeys them on every hop and throws `PageRefusal` for one they refuse.
154
+
155
+ robots.txt is the file at the address's origin, read through the same transport with the host's rule on its own hops, under its own deadline (`timeoutMs`). The reader parses its first `maxBytes` bytes and ignores the rest along with the line the limit cuts; RFC 9309 sets that limit's floor at 500 KiB (`ROBOTS_PARSING_FLOOR`, 512,000 bytes). Groups are matched by product token without case, the `*` group when none names the token, and two groups for the token are read as one. The longest matching `Allow` or `Disallow` decides, an `Allow` over a `Disallow` of the same length; `*` matches any run of characters, a final `$` the end of the path and query, and paths are compared with their percent-encoding made one form. A 400-range answer allows every page. A 500-range answer, a failed request, a passed deadline or a hop the host's rule refuses disallows every page, as the RFC's sections 2.3.1.3 and 2.3.1.4 say. A refusal of the egress policy, for the file's address or a hop of it, is the policy's answer and not the site's: the page ends with the policy's reason, such as `detail: 'scheme'` for an http address under `hosted`, as the same read with no robots rule does, and nothing is kept for it. A file past five redirects is taken as unavailable and allows every page. Each origin's answer is kept for `cacheMs` per transport and parsing limit; an answer the host's rule decided, by refusing a hop of the file, is not kept, since the next read may be under another rule. Wildcards are matched piece by piece, so the time grows with the pattern and the path and not with the ways to split them. `readRobots(url, options)`, `robotsAllows(file, token, url)` and `parseRobots(text)` are exported.
156
+
157
+ #### The licence a page states
158
+
159
+ `readLicence(html, baseUrl)` reads, in this order: a `<link rel="license" href>`, an `<a rel="license" href>`, a JSON-LD `license` (an address, or an object with its `url` and `name`), then any anchor to a Creative Commons deed (`creativecommons.org/licenses/<code>/<version>`) or public domain tool (`creativecommons.org/publicdomain/zero/<version>`, `creativecommons.org/publicdomain/mark/<version>`) anywhere in the page, a footer's included; another page of Creative Commons' site is not a licence. A deed is named from its address (`CC BY-SA 4.0`, `CC BY 3.0 US`, `CC0 1.0`, `Public Domain Mark 1.0`); another address is named by the anchor's text, the link's `title` or the JSON-LD `name`, else by the address. The address is the one the page gives, resolved against the page's own. A licence written in prose with no address is not read, so "not read" means the reader found none, not that the page states none. The first 1,000 of each kind are read: links marked `rel="license"`, anchors marked so, and anchors to Creative Commons' site.
160
+
161
+ #### A page with no text
162
+
163
+ `fallback`, a `PageFallbackReader` with `read(url, { signal })` answering text blocks or null, reads a page that answered 200 with no text a reader sees, such as one a script draws. It is called once, for the page's address after its redirects, with the caller's signal, and keeps its own deadline; its blocks are held to `textBudget`, and the document keeps the page's own title, licence, publisher and date. With no fallback, or when it answers null or fails, the page is `unreadable` with `failure: 'no_text'`. A fallback that requests the page itself, such as a rendering service, does so outside the host's rule and robots.txt: the reader asks it only for an address its own request reached under them.
164
+
165
+ #### Firecrawl as the fallback
166
+
167
+ `FirecrawlPageReader` is a `PageFallbackReader` over [Firecrawl's scrape endpoint](https://docs.firecrawl.dev/api-reference/endpoint/scrape), with the host's key:
168
+
169
+ ```ts
170
+ import { FirecrawlPageReader, readPage } from '@framers/agentos-ext-citation-check';
171
+
172
+ const fallback = new FirecrawlPageReader({
173
+ apiKey: process.env.FIRECRAWL_API_KEY!, // sent in the Authorization header, never logged or shown
174
+ timeoutMs: 5_000,
175
+ maxBytes: 2_000_000,
176
+ textBudget: 300_000,
177
+ });
178
+ const page = await readPage('https://example.org/app', { transport, scraper: null, timeoutMs: 5_000, maxBytes: 2_000_000, textBudget: 300_000, now: Date.now, fallback });
179
+ ```
180
+
181
+ Each page it reads costs one request, `POST https://api.firecrawl.dev/v2/scrape` (`FIRECRAWL_SCRAPE_URL`, or `url` when given), whose body is the page's address with the format and the deadline asked for, `{ url, formats: ['html'], onlyMainContent: false, maxAge: 0, timeout }`, and nothing else of the page or of the search that found it: no title, phrase or sentence. Firecrawl counts it as one credit ([pricing](https://www.firecrawl.dev/pricing)), so the reader runs only when the reader's own request found no text. `maxAge: 0` asks for a fresh scrape: Firecrawl answers a cached copy up to two days old when it is left out, and the page's `readAt` is the reader's clock. The key goes in the `Authorization` header to that endpoint alone: `url` is an https address, or plain http to the loopback address for a stand-in in a test, a redirect of the endpoint is not followed, and the key stays out of the reader's JSON, its inspected form and every error it throws. The page's cleaned HTML is read the way the reader reads its own fetches, the whole page with its tables and notes, and its text is held to `textBudget`.
182
+
183
+ The request and its answer's body share one deadline, `timeoutMs`; Firecrawl is asked to stop its own work at the same bound, held to the 1,000 to 300,000 milliseconds it takes. An answer over `maxBytes`, by its declared length or as it streams, is refused. `read` answers null for an error status (402 for no credits, 429, 500 and the rest), an answer that is not a scrape's, a failed scrape, a page that answered an error status to Firecrawl, a page Firecrawl ended on at another origin than the address asked for (its `metadata.url`, where Firecrawl's own redirects ended), an answer with no text, and the deadline; the page is then `unreadable` with `failure: 'no_text'`. The caller's cancellation ends the request and is thrown. `fetchFn` sends the request through a `fetch` of the host's; the request and its body are raced against the deadline and the caller's signal, so a `fetch` that does not watch the signal, or a body that stalls after its headers, still ends with them, and a body is cancelled without waiting for the cancel to settle.
184
+
185
+ ### The open web as a public source
186
+
187
+ `WebPageSourceProvider` is a public-sources `PublicSourceProvider` over the open web. The host gives the search, so the pack calls no search service of its own:
188
+
189
+ ```ts
190
+ import { WebPageSourceProvider } from '@framers/agentos-ext-citation-check';
191
+
192
+ const web = new WebPageSourceProvider({
193
+ search: async (phrase, signal) => mySearch(phrase, signal), // page addresses, best first
194
+ userAgent: 'ExampleReader/1.0 (https://example.org)',
195
+ robots: { token: 'ExampleReader', timeoutMs: 2_000, maxBytes: 512_000, cacheMs: 600_000 },
196
+ allowUrl: (url, phrase) => myHostRule(url, phrase), // false refuses the address before it is requested
197
+ timeoutMs: 5_000,
198
+ maxBytes: 2_000_000,
199
+ });
200
+
201
+ const found = await web.search('harbour opening year', { limit: 3, onSend: () => quota.take() });
202
+ if (found.kind === 'hits') {
203
+ for (const hit of found.hits) {
204
+ const read = await web.read(hit, { signal: AbortSignal.timeout(10_000) });
205
+ if (read.kind === 'document') console.log(read.document.site, read.document.title, read.document.licence.name);
206
+ }
207
+ }
208
+ ```
209
+
210
+ `search(phrase, { limit, signal, onSend })` throws a cancellation that came first, then awaits `onSend` once and runs the host's search, and answers its addresses as hits: `key` the address without its fragment, `title` the address, each page once, at most `limit` (3 when left out), an address that is not http or https passed over. It answers `limited` when `onSend` throws a limiter's refusal, and `failed` when `onSend` throws anything else or the search throws.
211
+
212
+ `read(hit, { signal })` reads the page and answers `{ kind: 'document', document }`, with `site` the host without a leading "www.", `revision` null, `readAt` the clock's, and the title, licence, publisher and date as read; or `{ kind: 'skipped', status }` for a page that was not read, with its HTTP status when one came. `allowUrl(url, phrase)` is called on every address and redirect target with the phrase whose search answered the hit read, held by the hit object itself, so pass the hits as `search` answered them: a hit object this provider's search did not answer, a copy of one included, reads with the empty string. robots.txt is read with `robots`, or with `DEFAULT_ROBOTS` (the `*` group, 2,000 ms, 512,000 bytes, ten minutes kept) when left out, and not at all with `robots: null`. Pages go through AgentOS's checked egress (`egress`, `hosted` by default) unless `transport` is given, with no scraper fallback, since the scraper's requests bypass both; `fallback`, such as `FirecrawlPageReader`, reads a page with no text. The provider queues nothing: the host reads at most as many pages at once as it calls `read` for. The caller's own cancellation is thrown.
213
+
126
214
  ### Grading with embeddings
127
215
 
128
216
  ```ts
@@ -161,7 +249,12 @@ The tool reads pages through AgentOS's checked egress with the hosted policy, so
161
249
  | `CitationVerifyTool`, `createExtensionPack` | The `citation_verify` tool and the pack |
162
250
  | `runCitationCli(argv, io)`, `CITATION_CLI_USAGE` | The command, for a host that runs it in its own process |
163
251
  | `CITATION_VERDICTS`, `pageKey`, `isHttpUrl` | The verdicts in order, a page's address without its fragment, the address check |
164
- | `CitedClaim`, `CitationCheck`, `CitationVerdict`, `CitationPassage`, `PageStatus`, `PageFailure`, `CitationReport`, `CitationSummary`, `CitationCheckOptions`, `CitationLogger`, `EmbedFn`, `NliFn` and the rest | The types |
252
+ | `readPage(url, options)`, `fetchTransport(fetch, userAgent)`, `egressTransport(profile, bounds, userAgent)`, `PageRefusal`, `UNREAD_LICENCE` | The page reader, its two transports, a transport's refusal, the licence of a page that states none |
253
+ | `readRobots(url, options)`, `robotsAllows(file, token, url)`, `parseRobots(text)`, `ROBOTS_PARSING_FLOOR` | robots.txt read through a transport, its answer for an address, its parser, RFC 9309's parsing floor |
254
+ | `readLicence(html, baseUrl)` | The licence a page's markup states |
255
+ | `WebPageSourceProvider`, `DEFAULT_ROBOTS` | The open web as a public source over the host's search, and its robots.txt rule when none is given |
256
+ | `FirecrawlPageReader`, `FIRECRAWL_SCRAPE_URL` | The reader of a page with no text through Firecrawl's scrape endpoint, and that endpoint |
257
+ | `CitedClaim`, `CitationCheck`, `CitationVerdict`, `CitationPassage`, `PageStatus`, `PageFailure`, `CitationReport`, `CitationSummary`, `CitationCheckOptions`, `CitationLogger`, `EmbedFn`, `NliFn`, `PageReaderOptions`, `PageRead`, `PageFallbackReader`, `FirecrawlPageReaderOptions`, `Transport`, `TransportAnswer`, `TransportRules`, `RobotsRule`, `RobotsFile`, `WebPageSourceOptions` and the rest | The types |
165
258
 
166
259
  ## License
167
260
 
@@ -0,0 +1,77 @@
1
+ /**
2
+ * @file firecrawl.ts
3
+ * @description A reader for a page the page reader's own request found no text in, such as one a script draws, over
4
+ * Firecrawl's scrape endpoint with the host's key. It sends one request a page, carrying the page's address and
5
+ * nothing else of the page or of the search that found it, and answers the page's text as blocks the same way the
6
+ * reader makes its own, or null.
7
+ *
8
+ * Firecrawl's scrape endpoint, from its API reference (https://docs.firecrawl.dev/api-reference/endpoint/scrape, the
9
+ * OpenAPI document `v2-openapi.json`, read on 11 November 2026): `POST https://api.firecrawl.dev/v2/scrape` with
10
+ * `Authorization: Bearer <key>` and a JSON body whose one required field is `url`; `formats` is a list of format names
11
+ * (`html` asks for the page's cleaned HTML, its scripts and styles removed), `onlyMainContent` (true when left out) cuts
12
+ * the headers, navigation and footers, `timeout` is Firecrawl's own deadline in milliseconds, from 1,000 to 300,000
13
+ * (60,000 when left out), and `maxAge` lets it answer a cached copy younger than that many milliseconds (172,800,000, two
14
+ * days, when left out). A 200 answers `{ success, data: { html, metadata: { statusCode, url } } }`, where `statusCode`
15
+ * is the status the page itself answered and `url` the page's address after the redirects Firecrawl followed; an error
16
+ * answers 400, 402, 403, 404, 409, 429, 500 or 503. Firecrawl's pricing
17
+ * (https://www.firecrawl.dev/pricing, read the same day) counts a scrape as 1 credit a page, a page that answers an
18
+ * error status included, and a scrape that returns no result as none.
19
+ *
20
+ * @module agentos/extensions/research/citation-check/firecrawl
21
+ */
22
+ import { type PageFallbackReader } from './page.js';
23
+ /** Firecrawl's scrape endpoint, where the reader sends when the host names no other. */
24
+ export declare const FIRECRAWL_SCRAPE_URL = "https://api.firecrawl.dev/v2/scrape";
25
+ /** How the reader reaches Firecrawl and bounds what it keeps. */
26
+ export interface FirecrawlPageReaderOptions {
27
+ /** The host's Firecrawl key: sent in the `Authorization` header to the endpoint alone, never logged or shown. */
28
+ apiKey: string;
29
+ /**
30
+ * The scrape endpoint; {@link FIRECRAWL_SCRAPE_URL} when left out. An https address, or plain http to the loopback
31
+ * address, where a host's test runs a stand-in.
32
+ */
33
+ url?: string;
34
+ /**
35
+ * The deadline of the request, its answer's body included, in milliseconds. Firecrawl is asked to stop its own work
36
+ * at the same bound, held to the 1,000 to 300,000 it takes.
37
+ */
38
+ timeoutMs: number;
39
+ /** The most bytes of Firecrawl's answer read; a longer answer is refused. */
40
+ maxBytes: number;
41
+ /** The most characters of the page's text kept. */
42
+ textBudget: number;
43
+ /**
44
+ * The `fetch` the request goes through; the global one when left out. The reader ends with its deadline and the
45
+ * caller's signal whether or not this `fetch` watches the signal.
46
+ */
47
+ fetchFn?: typeof globalThis.fetch;
48
+ }
49
+ /**
50
+ * A {@link PageFallbackReader} over Firecrawl's scrape endpoint. `read(url, { signal })` sends one request for the
51
+ * address, the HTML of the whole page asked for, and answers its text as blocks through the reader's own HTML reading,
52
+ * held to `textBudget`; null for an error status, an answer over `maxBytes`, an answer that is not a scrape's, a page
53
+ * that answered an error status, no text, or the reader's own deadline. The caller's cancellation ends the request and
54
+ * is thrown.
55
+ */
56
+ export declare class FirecrawlPageReader implements PageFallbackReader {
57
+ #private;
58
+ /**
59
+ * @param options - The key, the endpoint and the bounds.
60
+ * @throws {TypeError} For an empty key, an endpoint that is neither an https address nor an http address on the
61
+ * loopback address, or a bound that is not a whole number of at least 1; the message never holds the key.
62
+ */
63
+ constructor(options: FirecrawlPageReaderOptions);
64
+ /**
65
+ * Reads a page's text through Firecrawl. The request and its body are raced against the deadline and the caller's
66
+ * signal, so a `fetch` of the host's that does not watch the signal still ends with them; a body is cancelled without
67
+ * waiting for the cancel to settle.
68
+ *
69
+ * @param url - The page's address, the one request's only page-borne field.
70
+ * @param options - The caller's signal, which ends the request.
71
+ * @throws The caller's cancellation reason, whenever it fired before the answer.
72
+ */
73
+ read(url: string, options: {
74
+ signal: AbortSignal;
75
+ }): Promise<string[] | null>;
76
+ }
77
+ //# sourceMappingURL=firecrawl.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"firecrawl.d.ts","sourceRoot":"","sources":["../src/firecrawl.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAEH,OAAO,EAAiD,KAAK,kBAAkB,EAAE,MAAM,WAAW,CAAC;AAEnG,wFAAwF;AACxF,eAAO,MAAM,oBAAoB,wCAAwC,CAAC;AAM1E,iEAAiE;AACjE,MAAM,WAAW,0BAA0B;IACzC,iHAAiH;IACjH,MAAM,EAAE,MAAM,CAAC;IACf;;;OAGG;IACH,GAAG,CAAC,EAAE,MAAM,CAAC;IACb;;;OAGG;IACH,SAAS,EAAE,MAAM,CAAC;IAClB,6EAA6E;IAC7E,QAAQ,EAAE,MAAM,CAAC;IACjB,mDAAmD;IACnD,UAAU,EAAE,MAAM,CAAC;IACnB;;;OAGG;IACH,OAAO,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;CACnC;AAuDD;;;;;;GAMG;AACH,qBAAa,mBAAoB,YAAW,kBAAkB;;IAS5D;;;;OAIG;gBACS,OAAO,EAAE,0BAA0B;IAU/C;;;;;;;;OAQG;IACG,IAAI,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,EAAE;QAAE,MAAM,EAAE,WAAW,CAAA;KAAE,GAAG,OAAO,CAAC,MAAM,EAAE,GAAG,IAAI,CAAC;CAuDpF"}
@@ -0,0 +1,184 @@
1
+ /**
2
+ * @file firecrawl.ts
3
+ * @description A reader for a page the page reader's own request found no text in, such as one a script draws, over
4
+ * Firecrawl's scrape endpoint with the host's key. It sends one request a page, carrying the page's address and
5
+ * nothing else of the page or of the search that found it, and answers the page's text as blocks the same way the
6
+ * reader makes its own, or null.
7
+ *
8
+ * Firecrawl's scrape endpoint, from its API reference (https://docs.firecrawl.dev/api-reference/endpoint/scrape, the
9
+ * OpenAPI document `v2-openapi.json`, read on 11 November 2026): `POST https://api.firecrawl.dev/v2/scrape` with
10
+ * `Authorization: Bearer <key>` and a JSON body whose one required field is `url`; `formats` is a list of format names
11
+ * (`html` asks for the page's cleaned HTML, its scripts and styles removed), `onlyMainContent` (true when left out) cuts
12
+ * the headers, navigation and footers, `timeout` is Firecrawl's own deadline in milliseconds, from 1,000 to 300,000
13
+ * (60,000 when left out), and `maxAge` lets it answer a cached copy younger than that many milliseconds (172,800,000, two
14
+ * days, when left out). A 200 answers `{ success, data: { html, metadata: { statusCode, url } } }`, where `statusCode`
15
+ * is the status the page itself answered and `url` the page's address after the redirects Firecrawl followed; an error
16
+ * answers 400, 402, 403, 404, 409, 429, 500 or 503. Firecrawl's pricing
17
+ * (https://www.firecrawl.dev/pricing, read the same day) counts a scrape as 1 credit a page, a page that answers an
18
+ * error status included, and a scrape that returns no result as none.
19
+ *
20
+ * @module agentos/extensions/research/citation-check/firecrawl
21
+ */
22
+ import { originOf, pageBlocks, readCapped, whenAborted } from './page.js';
23
+ /** Firecrawl's scrape endpoint, where the reader sends when the host names no other. */
24
+ export const FIRECRAWL_SCRAPE_URL = 'https://api.firecrawl.dev/v2/scrape';
25
+ /** The shortest and the longest deadline Firecrawl's scrape takes, in milliseconds. */
26
+ const FIRECRAWL_TIMEOUT_FLOOR = 1_000;
27
+ const FIRECRAWL_TIMEOUT_CEILING = 300_000;
28
+ /** A whole number of at least 1. */
29
+ function whole(name, value) {
30
+ if (typeof value !== 'number' || !Number.isInteger(value) || value < 1)
31
+ throw new TypeError(`${name} must be a whole number of at least 1`);
32
+ return value;
33
+ }
34
+ /** Whether a host name is the loopback address: `localhost`, 127.0.0.0/8 or `[::1]`. */
35
+ function isLoopback(hostname) {
36
+ return hostname === 'localhost' || hostname === '[::1]' || /^127(?:\.\d{1,3}){3}$/u.test(hostname);
37
+ }
38
+ /**
39
+ * The endpoint the key may go to: an https address, or plain http to the loopback address (a stand-in in a test), so
40
+ * the key never crosses a network in the clear. A refusal names the option alone.
41
+ */
42
+ function endpointOf(value) {
43
+ if (value === undefined)
44
+ return FIRECRAWL_SCRAPE_URL;
45
+ try {
46
+ const { protocol, hostname } = new URL(String(value));
47
+ if (protocol === 'https:' || (protocol === 'http:' && isLoopback(hostname)))
48
+ return String(value);
49
+ }
50
+ catch {
51
+ // Refused below.
52
+ }
53
+ throw new TypeError('the Firecrawl scrape endpoint must be an https address, or an http address on the loopback address');
54
+ }
55
+ /** Cancels a body without waiting: a stream whose cancel never settles must not hold the answer. */
56
+ function discard(response) {
57
+ if (response.body !== null && !response.body.locked)
58
+ void response.body.cancel().catch(() => undefined);
59
+ return null;
60
+ }
61
+ /**
62
+ * The page's HTML from a scrape's answer, or null for a failed scrape, an error page, a page Firecrawl ended on at
63
+ * another origin than the address asked for, or an answer with none.
64
+ */
65
+ function htmlOf(answer, asked) {
66
+ if (answer === null || typeof answer !== 'object')
67
+ return null;
68
+ const { success, data } = answer;
69
+ if (success !== true || data === null || typeof data !== 'object')
70
+ return null;
71
+ const { html, metadata } = data;
72
+ const meta = metadata !== null && typeof metadata === 'object' ? metadata : {};
73
+ // Firecrawl answers a page that answered an error status with that page's text, which is no text of the page cited.
74
+ if (typeof meta.statusCode === 'number' && (meta.statusCode < 200 || meta.statusCode > 299))
75
+ return null;
76
+ // Firecrawl follows redirects itself and names the page it ended on; one at another origin is not the page cited, and
77
+ // neither the host's rule nor robots.txt cleared it.
78
+ if (typeof meta.url === 'string') {
79
+ const ended = originOf(meta.url);
80
+ if (ended === 'unparsable' || ended !== originOf(asked))
81
+ return null;
82
+ }
83
+ return typeof html === 'string' && html !== '' ? html : null;
84
+ }
85
+ /**
86
+ * A {@link PageFallbackReader} over Firecrawl's scrape endpoint. `read(url, { signal })` sends one request for the
87
+ * address, the HTML of the whole page asked for, and answers its text as blocks through the reader's own HTML reading,
88
+ * held to `textBudget`; null for an error status, an answer over `maxBytes`, an answer that is not a scrape's, a page
89
+ * that answered an error status, no text, or the reader's own deadline. The caller's cancellation ends the request and
90
+ * is thrown.
91
+ */
92
+ export class FirecrawlPageReader {
93
+ // Private fields stay out of `JSON.stringify` and `util.inspect`, so the key is shown nowhere.
94
+ #apiKey;
95
+ #url;
96
+ #fetchFn;
97
+ #timeoutMs;
98
+ #maxBytes;
99
+ #textBudget;
100
+ /**
101
+ * @param options - The key, the endpoint and the bounds.
102
+ * @throws {TypeError} For an empty key, an endpoint that is neither an https address nor an http address on the
103
+ * loopback address, or a bound that is not a whole number of at least 1; the message never holds the key.
104
+ */
105
+ constructor(options) {
106
+ if (typeof options.apiKey !== 'string' || options.apiKey.trim() === '')
107
+ throw new TypeError('a Firecrawl reader needs an API key');
108
+ this.#apiKey = options.apiKey;
109
+ this.#url = endpointOf(options.url);
110
+ this.#fetchFn = options.fetchFn;
111
+ this.#timeoutMs = whole('timeoutMs', options.timeoutMs);
112
+ this.#maxBytes = whole('maxBytes', options.maxBytes);
113
+ this.#textBudget = whole('textBudget', options.textBudget);
114
+ }
115
+ /**
116
+ * Reads a page's text through Firecrawl. The request and its body are raced against the deadline and the caller's
117
+ * signal, so a `fetch` of the host's that does not watch the signal still ends with them; a body is cancelled without
118
+ * waiting for the cancel to settle.
119
+ *
120
+ * @param url - The page's address, the one request's only page-borne field.
121
+ * @param options - The caller's signal, which ends the request.
122
+ * @throws The caller's cancellation reason, whenever it fired before the answer.
123
+ */
124
+ async read(url, options) {
125
+ const caller = options.signal;
126
+ // A signal that has already fired sends nothing.
127
+ caller.throwIfAborted();
128
+ const signal = AbortSignal.any([caller, AbortSignal.timeout(this.#timeoutMs)]);
129
+ const ended = whenAborted(signal);
130
+ let blocks = null;
131
+ try {
132
+ const sent = this.#send(url, signal);
133
+ // A response that comes after the read ended is cancelled, so its connection is not left open.
134
+ void ended.promise.catch(() => sent.then(discard, () => undefined));
135
+ blocks = await this.#blocksOf(url, await Promise.race([sent, ended.promise]), signal);
136
+ }
137
+ catch {
138
+ // Every failure answers null; the caller's cancellation is thrown below.
139
+ }
140
+ finally {
141
+ ended.release();
142
+ }
143
+ caller.throwIfAborted();
144
+ return blocks;
145
+ }
146
+ /** The one request: the page's address, the format and the deadline, with the key in the `Authorization` header. */
147
+ #send(url, signal) {
148
+ const fetchFn = this.#fetchFn ?? globalThis.fetch;
149
+ return fetchFn(this.#url, {
150
+ method: 'POST',
151
+ headers: { authorization: `Bearer ${this.#apiKey}`, 'content-type': 'application/json', accept: 'application/json' },
152
+ // The whole page, as the reader reads its own fetches, rather than Firecrawl's main-content cut; and a fresh
153
+ // scrape, since Firecrawl answers a cached copy up to two days old unless `maxAge` says otherwise, and the
154
+ // document is dated by the reader's clock.
155
+ body: JSON.stringify({
156
+ url,
157
+ formats: ['html'],
158
+ onlyMainContent: false,
159
+ maxAge: 0,
160
+ timeout: Math.min(FIRECRAWL_TIMEOUT_CEILING, Math.max(FIRECRAWL_TIMEOUT_FLOOR, this.#timeoutMs)),
161
+ }),
162
+ // The key goes to the endpoint alone, never to an address a redirect names.
163
+ redirect: 'error',
164
+ signal,
165
+ });
166
+ }
167
+ /** The page's text from Firecrawl's answer for the address asked for, or null. */
168
+ async #blocksOf(url, response, signal) {
169
+ if (!response.ok)
170
+ return discard(response);
171
+ const declared = Number(response.headers.get('content-length') ?? '');
172
+ if (Number.isFinite(declared) && declared > this.#maxBytes)
173
+ return discard(response);
174
+ const bytes = await readCapped(response, this.#maxBytes, signal);
175
+ if (bytes === null)
176
+ return null;
177
+ const html = htmlOf(JSON.parse(new TextDecoder('utf-8').decode(bytes)), url);
178
+ if (html === null)
179
+ return null;
180
+ const blocks = pageBlocks('html', html, this.#textBudget);
181
+ return blocks.length === 0 ? null : blocks;
182
+ }
183
+ }
184
+ //# sourceMappingURL=firecrawl.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"firecrawl.js","sourceRoot":"","sources":["../src/firecrawl.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAEH,OAAO,EAAE,QAAQ,EAAE,UAAU,EAAE,UAAU,EAAE,WAAW,EAA2B,MAAM,WAAW,CAAC;AAEnG,wFAAwF;AACxF,MAAM,CAAC,MAAM,oBAAoB,GAAG,qCAAqC,CAAC;AAE1E,uFAAuF;AACvF,MAAM,uBAAuB,GAAG,KAAK,CAAC;AACtC,MAAM,yBAAyB,GAAG,OAAO,CAAC;AA2B1C,oCAAoC;AACpC,SAAS,KAAK,CAAC,IAAY,EAAE,KAAc;IACzC,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,GAAG,CAAC;QAAE,MAAM,IAAI,SAAS,CAAC,GAAG,IAAI,uCAAuC,CAAC,CAAC;IAC5I,OAAO,KAAK,CAAC;AACf,CAAC;AAED,wFAAwF;AACxF,SAAS,UAAU,CAAC,QAAgB;IAClC,OAAO,QAAQ,KAAK,WAAW,IAAI,QAAQ,KAAK,OAAO,IAAI,wBAAwB,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;AACrG,CAAC;AAED;;;GAGG;AACH,SAAS,UAAU,CAAC,KAAc;IAChC,IAAI,KAAK,KAAK,SAAS;QAAE,OAAO,oBAAoB,CAAC;IACrD,IAAI,CAAC;QACH,MAAM,EAAE,QAAQ,EAAE,QAAQ,EAAE,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;QACtD,IAAI,QAAQ,KAAK,QAAQ,IAAI,CAAC,QAAQ,KAAK,OAAO,IAAI,UAAU,CAAC,QAAQ,CAAC,CAAC;YAAE,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC;IACpG,CAAC;IAAC,MAAM,CAAC;QACP,iBAAiB;IACnB,CAAC;IACD,MAAM,IAAI,SAAS,CAAC,oGAAoG,CAAC,CAAC;AAC5H,CAAC;AAED,oGAAoG;AACpG,SAAS,OAAO,CAAC,QAAkB;IACjC,IAAI,QAAQ,CAAC,IAAI,KAAK,IAAI,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,MAAM;QAAE,KAAK,QAAQ,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;IACxG,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;GAGG;AACH,SAAS,MAAM,CAAC,MAAe,EAAE,KAAa;IAC5C,IAAI,MAAM,KAAK,IAAI,IAAI,OAAO,MAAM,KAAK,QAAQ;QAAE,OAAO,IAAI,CAAC;IAC/D,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,GAAG,MAA+C,CAAC;IAC1E,IAAI,OAAO,KAAK,IAAI,IAAI,IAAI,KAAK,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ;QAAE,OAAO,IAAI,CAAC;IAC/E,MAAM,EAAE,IAAI,EAAE,QAAQ,EAAE,GAAG,IAA8C,CAAC;IAC1E,MAAM,IAAI,GAAG,QAAQ,KAAK,IAAI,IAAI,OAAO,QAAQ,KAAK,QAAQ,CAAC,CAAC,CAAE,QAAoD,CAAC,CAAC,CAAC,EAAE,CAAC;IAC5H,oHAAoH;IACpH,IAAI,OAAO,IAAI,CAAC,UAAU,KAAK,QAAQ,IAAI,CAAC,IAAI,CAAC,UAAU,GAAG,GAAG,IAAI,IAAI,CAAC,UAAU,GAAG,GAAG,CAAC;QAAE,OAAO,IAAI,CAAC;IACzG,sHAAsH;IACtH,qDAAqD;IACrD,IAAI,OAAO,IAAI,CAAC,GAAG,KAAK,QAAQ,EAAE,CAAC;QACjC,MAAM,KAAK,GAAG,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACjC,IAAI,KAAK,KAAK,YAAY,IAAI,KAAK,KAAK,QAAQ,CAAC,KAAK,CAAC;YAAE,OAAO,IAAI,CAAC;IACvE,CAAC;IACD,OAAO,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC;AAC/D,CAAC;AAED;;;;;;GAMG;AACH,MAAM,OAAO,mBAAmB;IAC9B,+FAA+F;IACtF,OAAO,CAAS;IAChB,IAAI,CAAS;IACb,QAAQ,CAAsC;IAC9C,UAAU,CAAS;IACnB,SAAS,CAAS;IAClB,WAAW,CAAS;IAE7B;;;;OAIG;IACH,YAAY,OAAmC;QAC7C,IAAI,OAAO,OAAO,CAAC,MAAM,KAAK,QAAQ,IAAI,OAAO,CAAC,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE;YAAE,MAAM,IAAI,SAAS,CAAC,qCAAqC,CAAC,CAAC;QACnI,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC;QAC9B,IAAI,CAAC,IAAI,GAAG,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;QACpC,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC,OAAO,CAAC;QAChC,IAAI,CAAC,UAAU,GAAG,KAAK,CAAC,WAAW,EAAE,OAAO,CAAC,SAAS,CAAC,CAAC;QACxD,IAAI,CAAC,SAAS,GAAG,KAAK,CAAC,UAAU,EAAE,OAAO,CAAC,QAAQ,CAAC,CAAC;QACrD,IAAI,CAAC,WAAW,GAAG,KAAK,CAAC,YAAY,EAAE,OAAO,CAAC,UAAU,CAAC,CAAC;IAC7D,CAAC;IAED;;;;;;;;OAQG;IACH,KAAK,CAAC,IAAI,CAAC,GAAW,EAAE,OAAgC;QACtD,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;QAC9B,iDAAiD;QACjD,MAAM,CAAC,cAAc,EAAE,CAAC;QACxB,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;QAC/E,MAAM,KAAK,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC;QAClC,IAAI,MAAM,GAAoB,IAAI,CAAC;QACnC,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE,MAAM,CAAC,CAAC;YACrC,+FAA+F;YAC/F,KAAK,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC,CAAC;YACpE,MAAM,GAAG,MAAM,IAAI,CAAC,SAAS,CAAC,GAAG,EAAE,MAAM,OAAO,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,CAAC,OAAO,CAAC,CAAC,EAAE,MAAM,CAAC,CAAC;QACxF,CAAC;QAAC,MAAM,CAAC;YACP,yEAAyE;QAC3E,CAAC;gBAAS,CAAC;YACT,KAAK,CAAC,OAAO,EAAE,CAAC;QAClB,CAAC;QACD,MAAM,CAAC,cAAc,EAAE,CAAC;QACxB,OAAO,MAAM,CAAC;IAChB,CAAC;IAED,oHAAoH;IACpH,KAAK,CAAC,GAAW,EAAE,MAAmB;QACpC,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,IAAI,UAAU,CAAC,KAAK,CAAC;QAClD,OAAO,OAAO,CAAC,IAAI,CAAC,IAAI,EAAE;YACxB,MAAM,EAAE,MAAM;YACd,OAAO,EAAE,EAAE,aAAa,EAAE,UAAU,IAAI,CAAC,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,MAAM,EAAE,kBAAkB,EAAE;YACpH,6GAA6G;YAC7G,2GAA2G;YAC3G,2CAA2C;YAC3C,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;gBACnB,GAAG;gBACH,OAAO,EAAE,CAAC,MAAM,CAAC;gBACjB,eAAe,EAAE,KAAK;gBACtB,MAAM,EAAE,CAAC;gBACT,OAAO,EAAE,IAAI,CAAC,GAAG,CAAC,yBAAyB,EAAE,IAAI,CAAC,GAAG,CAAC,uBAAuB,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;aACjG,CAAC;YACF,4EAA4E;YAC5E,QAAQ,EAAE,OAAO;YACjB,MAAM;SACP,CAAC,CAAC;IACL,CAAC;IAED,kFAAkF;IAClF,KAAK,CAAC,SAAS,CAAC,GAAW,EAAE,QAAkB,EAAE,MAAmB;QAClE,IAAI,CAAC,QAAQ,CAAC,EAAE;YAAE,OAAO,OAAO,CAAC,QAAQ,CAAC,CAAC;QAC3C,MAAM,QAAQ,GAAG,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,gBAAgB,CAAC,IAAI,EAAE,CAAC,CAAC;QACtE,IAAI,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,IAAI,QAAQ,GAAG,IAAI,CAAC,SAAS;YAAE,OAAO,OAAO,CAAC,QAAQ,CAAC,CAAC;QACrF,MAAM,KAAK,GAAG,MAAM,UAAU,CAAC,QAAQ,EAAE,IAAI,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC;QACjE,IAAI,KAAK,KAAK,IAAI;YAAE,OAAO,IAAI,CAAC;QAChC,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,WAAW,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;QAC7E,IAAI,IAAI,KAAK,IAAI;YAAE,OAAO,IAAI,CAAC;QAC/B,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,CAAC,WAAW,CAAC,CAAC;QAC1D,OAAO,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC;IAC7C,CAAC;CACF"}
package/dist/index.d.ts CHANGED
@@ -4,7 +4,10 @@
4
4
  * no search service: `parseCitedClaims` finds the claims and their addresses, `checkCitations` reads each page once and
5
5
  * grades each claim (word for word first, then through AgentOS's `CitationVerifier` when an embedding function is
6
6
  * given), `renderCitationLog` writes the markdown for a "Citation check log" section, and `citationReportFails` decides
7
- * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check.
7
+ * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check. The page
8
+ * reader is exported (`readPage` with its transports, the host rule and robots.txt on every hop, the licence read),
9
+ * and `WebPageSourceProvider` makes the open web a public source over a search the host gives. `FirecrawlPageReader`
10
+ * reads a page with no text through Firecrawl's scrape endpoint with the host's key.
8
11
  *
9
12
  * @module @framers/agentos-ext-citation-check
10
13
  */
@@ -14,7 +17,11 @@ import type { EmbedFn, NliFn } from './types.js';
14
17
  export * from './types.js';
15
18
  export { parseCitedClaims, CitationInputError, isHttpUrl, type CitedClaimFormat } from './parse.js';
16
19
  export { normalizeForMatch, quoteFragments, locateQuote, lexicalNearest, withoutInvisible } from './match.js';
17
- export { pageKey, type EgressProfile, type PageScraper } from './page.js';
20
+ export { pageKey, readPage, fetchTransport, egressTransport, PageRefusal, UNREAD_LICENCE, type EgressProfile, type PageScraper, type PageReaderOptions, type PageRead, type PageFallbackReader, type Transport, type TransportAnswer, type TransportRules, } from './page.js';
21
+ export { readRobots, robotsAllows, parseRobots, ROBOTS_PARSING_FLOOR, type RobotsRule, type RobotsFile, type RobotsGroup, type RobotsLine, type ReadRobotsOptions } from './robots.js';
22
+ export { readLicence } from './licence.js';
23
+ export { WebPageSourceProvider, DEFAULT_ROBOTS, type WebPageSourceOptions } from './web-source.js';
24
+ export { FirecrawlPageReader, FIRECRAWL_SCRAPE_URL, type FirecrawlPageReaderOptions } from './firecrawl.js';
18
25
  export { checkCitations, checkCitationDocument, CitationSetupError, type CitationCheckOptions } from './runner.js';
19
26
  export { renderCitationLog, citationReportFails, FAILING_VERDICTS, type CitationReportOptions } from './report.js';
20
27
  export { openAIEmbeddings, managerEmbeddings, type OpenAIEmbeddingOptions, type EmbeddingManagerLike } from './embeddings.js';
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAE,oBAAoB,EAAE,MAAM,kBAAkB,CAAC;AAE5E,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,WAAW,CAAC;AAE/C,OAAO,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAEjD,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAE,KAAK,gBAAgB,EAAE,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EAAE,OAAO,EAAE,KAAK,aAAa,EAAE,KAAK,WAAW,EAAE,MAAM,WAAW,CAAC;AAC1E,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAAE,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,KAAK,qBAAqB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,KAAK,sBAAsB,EAAE,KAAK,oBAAoB,EAAE,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAE,KAAK,mBAAmB,EAAE,KAAK,oBAAoB,EAAE,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAE,KAAK,aAAa,EAAE,MAAM,UAAU,CAAC;AAElF,2EAA2E;AAC3E,MAAM,WAAW,wBAAwB;IACvC,sHAAsH;IACtH,UAAU,CAAC,EAAE,MAAM,GAAG,QAAQ,CAAC;IAC/B,mEAAmE;IACnE,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,0HAA0H;IAC1H,MAAM,CAAC,EAAE,aAAa,CAAC;IACvB,0FAA0F;IAC1F,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;IAChC,yEAAyE;IACzE,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,qHAAqH;IACrH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd,yEAAyE;IACzE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,iEAAiE;IACjE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED;;;;;;GAMG;AACH,wBAAgB,mBAAmB,CAAC,OAAO,GAAE,oBAAyB,GAAG,aAAa,CAgCrF;AAED,eAAe,mBAAmB,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;GAYG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAE,oBAAoB,EAAE,MAAM,kBAAkB,CAAC;AAE5E,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,WAAW,CAAC;AAE/C,OAAO,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAEjD,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAE,KAAK,gBAAgB,EAAE,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EACL,OAAO,EACP,QAAQ,EACR,cAAc,EACd,eAAe,EACf,WAAW,EACX,cAAc,EACd,KAAK,aAAa,EAClB,KAAK,WAAW,EAChB,KAAK,iBAAiB,EACtB,KAAK,QAAQ,EACb,KAAK,kBAAkB,EACvB,KAAK,SAAS,EACd,KAAK,eAAe,EACpB,KAAK,cAAc,GACpB,MAAM,WAAW,CAAC;AACnB,OAAO,EAAE,UAAU,EAAE,YAAY,EAAE,WAAW,EAAE,oBAAoB,EAAE,KAAK,UAAU,EAAE,KAAK,UAAU,EAAE,KAAK,WAAW,EAAE,KAAK,UAAU,EAAE,KAAK,iBAAiB,EAAE,MAAM,aAAa,CAAC;AACvL,OAAO,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAC3C,OAAO,EAAE,qBAAqB,EAAE,cAAc,EAAE,KAAK,oBAAoB,EAAE,MAAM,iBAAiB,CAAC;AACnG,OAAO,EAAE,mBAAmB,EAAE,oBAAoB,EAAE,KAAK,0BAA0B,EAAE,MAAM,gBAAgB,CAAC;AAC5G,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAAE,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,KAAK,qBAAqB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,KAAK,sBAAsB,EAAE,KAAK,oBAAoB,EAAE,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAE,KAAK,mBAAmB,EAAE,KAAK,oBAAoB,EAAE,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAE,KAAK,aAAa,EAAE,MAAM,UAAU,CAAC;AAElF,2EAA2E;AAC3E,MAAM,WAAW,wBAAwB;IACvC,sHAAsH;IACtH,UAAU,CAAC,EAAE,MAAM,GAAG,QAAQ,CAAC;IAC/B,mEAAmE;IACnE,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,0HAA0H;IAC1H,MAAM,CAAC,EAAE,aAAa,CAAC;IACvB,0FAA0F;IAC1F,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;IAChC,yEAAyE;IACzE,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,qHAAqH;IACrH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd,yEAAyE;IACzE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,iEAAiE;IACjE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED;;;;;;GAMG;AACH,wBAAgB,mBAAmB,CAAC,OAAO,GAAE,oBAAyB,GAAG,aAAa,CAgCrF;AAED,eAAe,mBAAmB,CAAC"}
package/dist/index.js CHANGED
@@ -4,7 +4,10 @@
4
4
  * no search service: `parseCitedClaims` finds the claims and their addresses, `checkCitations` reads each page once and
5
5
  * grades each claim (word for word first, then through AgentOS's `CitationVerifier` when an embedding function is
6
6
  * given), `renderCitationLog` writes the markdown for a "Citation check log" section, and `citationReportFails` decides
7
- * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check.
7
+ * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check. The page
8
+ * reader is exported (`readPage` with its transports, the host rule and robots.txt on every hop, the licence read),
9
+ * and `WebPageSourceProvider` makes the open web a public source over a search the host gives. `FirecrawlPageReader`
10
+ * reads a page with no text through Firecrawl's scrape endpoint with the host's key.
8
11
  *
9
12
  * @module @framers/agentos-ext-citation-check
10
13
  */
@@ -13,7 +16,11 @@ import { CitationVerifyTool } from './tools/citationVerify.js';
13
16
  export * from './types.js';
14
17
  export { parseCitedClaims, CitationInputError, isHttpUrl } from './parse.js';
15
18
  export { normalizeForMatch, quoteFragments, locateQuote, lexicalNearest, withoutInvisible } from './match.js';
16
- export { pageKey } from './page.js';
19
+ export { pageKey, readPage, fetchTransport, egressTransport, PageRefusal, UNREAD_LICENCE, } from './page.js';
20
+ export { readRobots, robotsAllows, parseRobots, ROBOTS_PARSING_FLOOR } from './robots.js';
21
+ export { readLicence } from './licence.js';
22
+ export { WebPageSourceProvider, DEFAULT_ROBOTS } from './web-source.js';
23
+ export { FirecrawlPageReader, FIRECRAWL_SCRAPE_URL } from './firecrawl.js';
17
24
  export { checkCitations, checkCitationDocument, CitationSetupError } from './runner.js';
18
25
  export { renderCitationLog, citationReportFails, FAILING_VERDICTS } from './report.js';
19
26
  export { openAIEmbeddings, managerEmbeddings } from './embeddings.js';
package/dist/index.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAGH,OAAO,EAAE,gBAAgB,EAAE,MAAM,iBAAiB,CAAC;AAEnD,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAG/D,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAyB,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EAAE,OAAO,EAAwC,MAAM,WAAW,CAAC;AAC1E,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAA6B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAA8B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAA0D,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAuD,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAsB,MAAM,UAAU,CAAC;AA0BlF;;;;;;GAMG;AACH,MAAM,UAAU,mBAAmB,CAAC,UAAgC,EAAE;IACpE,MAAM,OAAO,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,EAAE,CAA6B,CAAC;IACpE,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,MAAM,CAAC;IAChD,IAAI,UAAU,KAAK,MAAM,IAAI,UAAU,KAAK,QAAQ;QAAE,MAAM,IAAI,SAAS,CAAC,0CAA0C,IAAI,CAAC,SAAS,CAAC,UAAU,CAAC,EAAE,CAAC,CAAC;IAClJ,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,IAAI,OAAO,CAAC,MAAM,KAAK,QAAQ,IAAI,OAAO,CAAC,MAAM,KAAK,OAAO,EAAE,CAAC;QAC9F,MAAM,IAAI,SAAS,CAAC,uCAAuC,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;IAC/F,CAAC;IACD,MAAM,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC,eAAe,CAAC,CAAC;IACpD,MAAM,OAAO,GACX,OAAO,CAAC,OAAO;QACf,CAAC,UAAU,KAAK,QAAQ,CAAC,CAAC,CAAC,gBAAgB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,GAAG,CAAC,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;IACpK,MAAM,IAAI,GAAG,IAAI,kBAAkB,CAAC;QAClC,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;QACpG,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC7C,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAChE,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC5E,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACpE,CAAC,CAAC;IACH,OAAO;QACL,IAAI,EAAE,qCAAqC;QAC3C,WAAW,EAAE;YACX;gBACE,EAAE,EAAE,IAAI,CAAC,IAAI;gBACb,IAAI,EAAE,MAAM;gBACZ,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,EAAE;gBAChC,OAAO,EAAE,IAAI;gBACb,eAAe,EAAE,CAAC,EAAE,EAAE,EAAE,eAAe,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC;aAC3D;SACF;KACF,CAAC;AACJ,CAAC;AAED,eAAe,mBAAmB,CAAC"}
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;GAYG;AAGH,OAAO,EAAE,gBAAgB,EAAE,MAAM,iBAAiB,CAAC;AAEnD,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAG/D,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAyB,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EACL,OAAO,EACP,QAAQ,EACR,cAAc,EACd,eAAe,EACf,WAAW,EACX,cAAc,GASf,MAAM,WAAW,CAAC;AACnB,OAAO,EAAE,UAAU,EAAE,YAAY,EAAE,WAAW,EAAE,oBAAoB,EAA+F,MAAM,aAAa,CAAC;AACvL,OAAO,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAC3C,OAAO,EAAE,qBAAqB,EAAE,cAAc,EAA6B,MAAM,iBAAiB,CAAC;AACnG,OAAO,EAAE,mBAAmB,EAAE,oBAAoB,EAAmC,MAAM,gBAAgB,CAAC;AAC5G,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAA6B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAA8B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAA0D,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAuD,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAsB,MAAM,UAAU,CAAC;AA0BlF;;;;;;GAMG;AACH,MAAM,UAAU,mBAAmB,CAAC,UAAgC,EAAE;IACpE,MAAM,OAAO,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,EAAE,CAA6B,CAAC;IACpE,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,MAAM,CAAC;IAChD,IAAI,UAAU,KAAK,MAAM,IAAI,UAAU,KAAK,QAAQ;QAAE,MAAM,IAAI,SAAS,CAAC,0CAA0C,IAAI,CAAC,SAAS,CAAC,UAAU,CAAC,EAAE,CAAC,CAAC;IAClJ,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,IAAI,OAAO,CAAC,MAAM,KAAK,QAAQ,IAAI,OAAO,CAAC,MAAM,KAAK,OAAO,EAAE,CAAC;QAC9F,MAAM,IAAI,SAAS,CAAC,uCAAuC,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;IAC/F,CAAC;IACD,MAAM,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC,eAAe,CAAC,CAAC;IACpD,MAAM,OAAO,GACX,OAAO,CAAC,OAAO;QACf,CAAC,UAAU,KAAK,QAAQ,CAAC,CAAC,CAAC,gBAAgB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,GAAG,CAAC,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;IACpK,MAAM,IAAI,GAAG,IAAI,kBAAkB,CAAC;QAClC,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;QACpG,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC7C,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAChE,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC5E,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACpE,CAAC,CAAC;IACH,OAAO;QACL,IAAI,EAAE,qCAAqC;QAC3C,WAAW,EAAE;YACX;gBACE,EAAE,EAAE,IAAI,CAAC,IAAI;gBACb,IAAI,EAAE,MAAM;gBACZ,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,EAAE;gBAChC,OAAO,EAAE,IAAI;gBACb,eAAe,EAAE,CAAC,EAAE,EAAE,EAAE,eAAe,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC;aAC3D;SACF;KACF,CAAC;AACJ,CAAC;AAED,eAAe,mBAAmB,CAAC"}
@@ -0,0 +1,31 @@
1
+ /**
2
+ * @file licence.ts
3
+ * @description The licence a page states, read from its markup in this order: a `<link rel="license" href>`, an
4
+ * `<a rel="license" href>`, a JSON-LD `license`, then any anchor to a Creative Commons deed
5
+ * (`creativecommons.org/licenses/<code>/<version>`) or public domain tool (`creativecommons.org/publicdomain/zero/<version>`,
6
+ * `creativecommons.org/publicdomain/mark/<version>`), wherever it stands, a footer's included. A deed is named from its address (`CC BY-SA 4.0`, `CC BY 3.0 US`,
7
+ * `CC0 1.0`); another address is named by the anchor's text, the link's `title` or the JSON-LD `name`, else by the
8
+ * address itself. The address is the one the page gives, resolved against the page's own. A licence written in prose
9
+ * with no address ("Text licensed under CC BY 4.0") is not read. The first 1,000 of each kind are read: links marked
10
+ * `rel="license"`, anchors marked so, and anchors to Creative Commons' site.
11
+ *
12
+ * @module agentos/extensions/research/citation-check/licence
13
+ */
14
+ import type { SourceLicence } from '@framers/agentos-ext-public-sources';
15
+ import { type MarkupScan } from './meta.js';
16
+ /**
17
+ * The licence a page states, or null when it states none this reader finds.
18
+ *
19
+ * @param html - The page's markup as served.
20
+ * @param baseUrl - The page's own address, after redirects, which relative addresses resolve against.
21
+ */
22
+ export declare function readLicence(html: string, baseUrl: string): SourceLicence | null;
23
+ /**
24
+ * {@link readLicence} over markup already read: one pass's links and anchors, and the page's JSON-LD nodes.
25
+ *
26
+ * @param scan - The pass over the markup, from `scanMarkup` with the same address.
27
+ * @param nodes - The page's JSON-LD nodes, from `jsonLdNodes`.
28
+ * @param baseUrl - The page's own address.
29
+ */
30
+ export declare function licenceOf(scan: MarkupScan, nodes: ReadonlyArray<Record<string, unknown>>, baseUrl: string): SourceLicence | null;
31
+ //# sourceMappingURL=licence.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"licence.d.ts","sourceRoot":"","sources":["../src/licence.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;GAYG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,qCAAqC,CAAC;AACzE,OAAO,EAAsF,KAAK,UAAU,EAAE,MAAM,WAAW,CAAC;AA0DhI;;;;;GAKG;AACH,wBAAgB,WAAW,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,aAAa,GAAG,IAAI,CAG/E;AAED;;;;;;GAMG;AACH,wBAAgB,SAAS,CAAC,IAAI,EAAE,UAAU,EAAE,KAAK,EAAE,aAAa,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,GAAG,aAAa,GAAG,IAAI,CAsBhI"}