@framers/agentos-ext-citation-check 0.1.4 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -39,7 +39,7 @@ and exits 1. The output is the markdown for a document's "Citation check log" se
39
39
  npm install @framers/agentos-ext-citation-check
40
40
  ```
41
41
 
42
- It needs Node.js 22 or later, as `@framers/agentos` does. It reads pages with [public-sources](https://www.npmjs.com/package/@framers/agentos-ext-public-sources) (`htmlToBlocks` in page mode and `citationSources`), the [web scraper](https://www.npmjs.com/package/@framers/agentos-ext-web-scraper) (a browser's request headers and the fallback tiers) and the `values` entry of [grounding-guard](https://www.npmjs.com/package/@framers/agentos-ext-grounding-guard) (the sentence rule and the word rules). `@framers/agentos` 0.13.120 or later, the first release with the checked egress the tool reads pages through, is a peer, loaded only to grade with embeddings or to read pages through checked egress. Grounding-guard lists `@huggingface/transformers` as an optional dependency for its NLI model; `npm install --omit=optional` leaves it out when you do not use NLI.
42
+ It needs Node.js 22 or later, as `@framers/agentos` does. It reads pages with [public-sources](https://www.npmjs.com/package/@framers/agentos-ext-public-sources) (`htmlToBlocks` in page mode, `citationSources` and the `SourceDocument` a page answers as, from 0.2.3), the [web scraper](https://www.npmjs.com/package/@framers/agentos-ext-web-scraper) (a browser's request headers and the fallback tiers) the `values` entry of [grounding-guard](https://www.npmjs.com/package/@framers/agentos-ext-grounding-guard) (the sentence rule and the word rules) and [entities](https://www.npmjs.com/package/entities) (the character references in a page's title, names and addresses). `@framers/agentos` 0.13.120 or later, the first release with the checked egress the tool reads pages through, is a peer, loaded only to grade with embeddings or to read pages through checked egress. Grounding-guard lists `@huggingface/transformers` as an optional dependency for its NLI model; `npm install --omit=optional` leaves it out when you do not use NLI.
43
43
 
44
44
  ## Input
45
45
 
@@ -123,6 +123,74 @@ Each distinct address is fetched once, its fragment removed, four at a time. A r
123
123
 
124
124
  The global `fetch` reaches any address a document names. A host that checks documents from people it does not trust passes `egress: 'hosted'`, which reaches public https addresses on port 443 alone and checks every redirect, or a `fetch` of its own. The fallback's requests go through neither, so it is off once either is given unless `fallback: true` turns it on.
125
125
 
126
+ ### The page reader
127
+
128
+ `readPage(url, options)` is the reader `checkCitations` runs, exported for a host that reads pages of its own. It answers `{ status, blocks, passages, document, pageDate }`: the page's status as above, its text blocks, its sentences as passages, the page as a public-sources `SourceDocument` (null when it was not read) and the page's own date (null when it states none). It throws only the caller's own cancellation.
129
+
130
+ ```ts
131
+ import { egressTransport, readPage } from '@framers/agentos-ext-citation-check';
132
+
133
+ const transport = await egressTransport('hosted', { timeoutMs: 5_000, maxBytes: 2_000_000 }, 'ExampleReader/1.0 (https://example.org)');
134
+ const page = await readPage('https://example.org/harbour', {
135
+ transport,
136
+ scraper: null,
137
+ timeoutMs: 5_000,
138
+ maxBytes: 2_000_000,
139
+ textBudget: 300_000,
140
+ now: Date.now,
141
+ allowUrl: (url) => !new URL(url).hostname.endsWith('.internal.example'),
142
+ robots: { token: 'ExampleReader', timeoutMs: 2_000, maxBytes: 512_000, cacheMs: 600_000 },
143
+ });
144
+ if (page.document !== null) console.log(page.document.title, page.document.licence.name, page.document.publisher);
145
+ ```
146
+
147
+ `fetchTransport(fetch, userAgent?)` and `egressTransport(profile, bounds, userAgent?)` build the request a page goes through. A transport takes `(url, signal, rules?)` and answers `{ status, finalUrl, header, read, readPrefix, discard }`; a host can pass one of its own.
148
+
149
+ The document's `url` and `key` are the address after redirects. Its `title` is the page's `<title>`, its character references read, its white space collapsed and cut to 255 characters, or the host when the page has none. Its `licence` is the one the page states (below), or `{ name: 'not read', url: '' }` (`UNREAD_LICENCE`) when the reader finds none. `publisher` is the site's name from `og:site_name`, else from JSON-LD `publisher.name`, cut to 255 characters and left out when the page names none. `pageDate` is the page's own date from `article:modified_time`, else JSON-LD `dateModified`, else the `Last-Modified` header, left out when none parses. The markup is read as served, in one pass in document order: a value a script writes later is not seen, a tag or a JSON-LD block inside a comment is passed over, the content of a script, a style, a template, a `noscript` and a `textarea` is not read for tags, and the time grows with the page alone however its tags are left open. Of the links and anchors the pass keeps only those a licence is read from, at most 1,000 of each kind, and reads the text of a `rel="license"` anchor alone, so a page of a million other anchors holds none of them.
150
+
151
+ #### The host's rule and robots.txt, on every hop
152
+
153
+ `allowUrl(url)` is the host's rule for an address. `robots` reads robots.txt by [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) with the host's product token. Given either, the reader checks the first address before it is requested, and both transports check every redirect target before its request, following up to five redirects one at a time. A refused address ends the page as `unreachable` with `failure: 'refused'` and `detail: 'host'` or `detail: 'robots'`, and is never requested. A redirect to an address that is not http or https, such as a `data:` address, ends the page with `detail: 'scheme'`, as `fetch` itself ends one. The host's rule is asked once an address. The web scraper's retry is not used under either rule, since its requests go through neither. A transport called with the rules applies them the same way, so an image or an icon read through it obeys them on every hop and throws `PageRefusal` for one they refuse.
154
+
155
+ robots.txt is the file at the address's origin, read through the same transport with the host's rule on its own hops, under its own deadline (`timeoutMs`). The reader parses its first `maxBytes` bytes and ignores the rest along with the line the limit cuts; RFC 9309 sets that limit's floor at 500 KiB (`ROBOTS_PARSING_FLOOR`, 512,000 bytes). Groups are matched by product token without case, the `*` group when none names the token, and two groups for the token are read as one. The longest matching `Allow` or `Disallow` decides, an `Allow` over a `Disallow` of the same length; `*` matches any run of characters, a final `$` the end of the path and query, and paths are compared with their percent-encoding made one form. A 400-range answer allows every page. A 500-range answer, a failed request, a passed deadline or a hop the host's rule refuses disallows every page, as the RFC's sections 2.3.1.3 and 2.3.1.4 say. A refusal of the egress policy, for the file's address or a hop of it, is the policy's answer and not the site's: the page ends with the policy's reason, such as `detail: 'scheme'` for an http address under `hosted`, as the same read with no robots rule does, and nothing is kept for it. A file past five redirects is taken as unavailable and allows every page. Each origin's answer is kept for `cacheMs` per transport and parsing limit; an answer the host's rule decided, by refusing a hop of the file, is not kept, since the next read may be under another rule. Wildcards are matched piece by piece, so the time grows with the pattern and the path and not with the ways to split them. `readRobots(url, options)`, `robotsAllows(file, token, url)` and `parseRobots(text)` are exported.
156
+
157
+ #### The licence a page states
158
+
159
+ `readLicence(html, baseUrl)` reads, in this order: a `<link rel="license" href>`, an `<a rel="license" href>`, a JSON-LD `license` (an address, or an object with its `url` and `name`), then any anchor to a Creative Commons deed (`creativecommons.org/licenses/<code>/<version>`) or public domain tool (`creativecommons.org/publicdomain/zero/<version>`, `creativecommons.org/publicdomain/mark/<version>`) anywhere in the page, a footer's included; another page of Creative Commons' site is not a licence. A deed is named from its address (`CC BY-SA 4.0`, `CC BY 3.0 US`, `CC0 1.0`, `Public Domain Mark 1.0`); another address is named by the anchor's text, the link's `title` or the JSON-LD `name`, else by the address. The address is the one the page gives, resolved against the page's own. A licence written in prose with no address is not read, so "not read" means the reader found none, not that the page states none. The first 1,000 of each kind are read: links marked `rel="license"`, anchors marked so, and anchors to Creative Commons' site.
160
+
161
+ #### A page with no text
162
+
163
+ `fallback`, a `PageFallbackReader` with `read(url, { signal })` answering text blocks or null, reads a page that answered 200 with no text a reader sees, such as one a script draws. It is called once, for the page's address after its redirects, with the caller's signal, and keeps its own deadline; its blocks are held to `textBudget`, and the document keeps the page's own title, licence, publisher and date. With no fallback, or when it answers null or fails, the page is `unreadable` with `failure: 'no_text'`. A fallback that requests the page itself, such as a rendering service, does so outside the host's rule and robots.txt: the reader asks it only for an address its own request reached under them.
164
+
165
+ ### The open web as a public source
166
+
167
+ `WebPageSourceProvider` is a public-sources `PublicSourceProvider` over the open web. The host gives the search, so the pack calls no search service of its own:
168
+
169
+ ```ts
170
+ import { WebPageSourceProvider } from '@framers/agentos-ext-citation-check';
171
+
172
+ const web = new WebPageSourceProvider({
173
+ search: async (phrase, signal) => mySearch(phrase, signal), // page addresses, best first
174
+ userAgent: 'ExampleReader/1.0 (https://example.org)',
175
+ robots: { token: 'ExampleReader', timeoutMs: 2_000, maxBytes: 512_000, cacheMs: 600_000 },
176
+ allowUrl: (url, phrase) => myHostRule(url, phrase), // false refuses the address before it is requested
177
+ timeoutMs: 5_000,
178
+ maxBytes: 2_000_000,
179
+ });
180
+
181
+ const found = await web.search('harbour opening year', { limit: 3, onSend: () => quota.take() });
182
+ if (found.kind === 'hits') {
183
+ for (const hit of found.hits) {
184
+ const read = await web.read(hit, { signal: AbortSignal.timeout(10_000) });
185
+ if (read.kind === 'document') console.log(read.document.site, read.document.title, read.document.licence.name);
186
+ }
187
+ }
188
+ ```
189
+
190
+ `search(phrase, { limit, signal, onSend })` throws a cancellation that came first, then awaits `onSend` once and runs the host's search, and answers its addresses as hits: `key` the address without its fragment, `title` the address, each page once, at most `limit` (3 when left out), an address that is not http or https passed over. It answers `limited` when `onSend` throws a limiter's refusal, and `failed` when `onSend` throws anything else or the search throws.
191
+
192
+ `read(hit, { signal })` reads the page and answers `{ kind: 'document', document }`, with `site` the host without a leading "www.", `revision` null, `readAt` the clock's, and the title, licence, publisher and date as read; or `{ kind: 'skipped', status }` for a page that was not read, with its HTTP status when one came. `allowUrl(url, phrase)` is called on every address and redirect target with the phrase whose search answered the hit read, held by the hit object itself, so pass the hits as `search` answered them: a hit object this provider's search did not answer, a copy of one included, reads with the empty string. robots.txt is read with `robots`, or with `DEFAULT_ROBOTS` (the `*` group, 2,000 ms, 512,000 bytes, ten minutes kept) when left out, and not at all with `robots: null`. Pages go through AgentOS's checked egress (`egress`, `hosted` by default) unless `transport` is given, with no scraper fallback, since the scraper's requests bypass both; `fallback` reads a page with no text. The provider queues nothing: the host reads at most as many pages at once as it calls `read` for. The caller's own cancellation is thrown.
193
+
126
194
  ### Grading with embeddings
127
195
 
128
196
  ```ts
@@ -161,7 +229,11 @@ The tool reads pages through AgentOS's checked egress with the hosted policy, so
161
229
  | `CitationVerifyTool`, `createExtensionPack` | The `citation_verify` tool and the pack |
162
230
  | `runCitationCli(argv, io)`, `CITATION_CLI_USAGE` | The command, for a host that runs it in its own process |
163
231
  | `CITATION_VERDICTS`, `pageKey`, `isHttpUrl` | The verdicts in order, a page's address without its fragment, the address check |
164
- | `CitedClaim`, `CitationCheck`, `CitationVerdict`, `CitationPassage`, `PageStatus`, `PageFailure`, `CitationReport`, `CitationSummary`, `CitationCheckOptions`, `CitationLogger`, `EmbedFn`, `NliFn` and the rest | The types |
232
+ | `readPage(url, options)`, `fetchTransport(fetch, userAgent)`, `egressTransport(profile, bounds, userAgent)`, `PageRefusal`, `UNREAD_LICENCE` | The page reader, its two transports, a transport's refusal, the licence of a page that states none |
233
+ | `readRobots(url, options)`, `robotsAllows(file, token, url)`, `parseRobots(text)`, `ROBOTS_PARSING_FLOOR` | robots.txt read through a transport, its answer for an address, its parser, RFC 9309's parsing floor |
234
+ | `readLicence(html, baseUrl)` | The licence a page's markup states |
235
+ | `WebPageSourceProvider`, `DEFAULT_ROBOTS` | The open web as a public source over the host's search, and its robots.txt rule when none is given |
236
+ | `CitedClaim`, `CitationCheck`, `CitationVerdict`, `CitationPassage`, `PageStatus`, `PageFailure`, `CitationReport`, `CitationSummary`, `CitationCheckOptions`, `CitationLogger`, `EmbedFn`, `NliFn`, `PageReaderOptions`, `PageRead`, `PageFallbackReader`, `Transport`, `TransportAnswer`, `TransportRules`, `RobotsRule`, `RobotsFile`, `WebPageSourceOptions` and the rest | The types |
165
237
 
166
238
  ## License
167
239
 
package/dist/index.d.ts CHANGED
@@ -4,7 +4,9 @@
4
4
  * no search service: `parseCitedClaims` finds the claims and their addresses, `checkCitations` reads each page once and
5
5
  * grades each claim (word for word first, then through AgentOS's `CitationVerifier` when an embedding function is
6
6
  * given), `renderCitationLog` writes the markdown for a "Citation check log" section, and `citationReportFails` decides
7
- * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check.
7
+ * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check. The page
8
+ * reader is exported (`readPage` with its transports, the host rule and robots.txt on every hop, the licence read),
9
+ * and `WebPageSourceProvider` makes the open web a public source over a search the host gives.
8
10
  *
9
11
  * @module @framers/agentos-ext-citation-check
10
12
  */
@@ -14,7 +16,10 @@ import type { EmbedFn, NliFn } from './types.js';
14
16
  export * from './types.js';
15
17
  export { parseCitedClaims, CitationInputError, isHttpUrl, type CitedClaimFormat } from './parse.js';
16
18
  export { normalizeForMatch, quoteFragments, locateQuote, lexicalNearest, withoutInvisible } from './match.js';
17
- export { pageKey, type EgressProfile, type PageScraper } from './page.js';
19
+ export { pageKey, readPage, fetchTransport, egressTransport, PageRefusal, UNREAD_LICENCE, type EgressProfile, type PageScraper, type PageReaderOptions, type PageRead, type PageFallbackReader, type Transport, type TransportAnswer, type TransportRules, } from './page.js';
20
+ export { readRobots, robotsAllows, parseRobots, ROBOTS_PARSING_FLOOR, type RobotsRule, type RobotsFile, type RobotsGroup, type RobotsLine, type ReadRobotsOptions } from './robots.js';
21
+ export { readLicence } from './licence.js';
22
+ export { WebPageSourceProvider, DEFAULT_ROBOTS, type WebPageSourceOptions } from './web-source.js';
18
23
  export { checkCitations, checkCitationDocument, CitationSetupError, type CitationCheckOptions } from './runner.js';
19
24
  export { renderCitationLog, citationReportFails, FAILING_VERDICTS, type CitationReportOptions } from './report.js';
20
25
  export { openAIEmbeddings, managerEmbeddings, type OpenAIEmbeddingOptions, type EmbeddingManagerLike } from './embeddings.js';
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAE,oBAAoB,EAAE,MAAM,kBAAkB,CAAC;AAE5E,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,WAAW,CAAC;AAE/C,OAAO,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAEjD,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAE,KAAK,gBAAgB,EAAE,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EAAE,OAAO,EAAE,KAAK,aAAa,EAAE,KAAK,WAAW,EAAE,MAAM,WAAW,CAAC;AAC1E,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAAE,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,KAAK,qBAAqB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,KAAK,sBAAsB,EAAE,KAAK,oBAAoB,EAAE,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAE,KAAK,mBAAmB,EAAE,KAAK,oBAAoB,EAAE,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAE,KAAK,aAAa,EAAE,MAAM,UAAU,CAAC;AAElF,2EAA2E;AAC3E,MAAM,WAAW,wBAAwB;IACvC,sHAAsH;IACtH,UAAU,CAAC,EAAE,MAAM,GAAG,QAAQ,CAAC;IAC/B,mEAAmE;IACnE,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,0HAA0H;IAC1H,MAAM,CAAC,EAAE,aAAa,CAAC;IACvB,0FAA0F;IAC1F,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;IAChC,yEAAyE;IACzE,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,qHAAqH;IACrH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd,yEAAyE;IACzE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,iEAAiE;IACjE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED;;;;;;GAMG;AACH,wBAAgB,mBAAmB,CAAC,OAAO,GAAE,oBAAyB,GAAG,aAAa,CAgCrF;AAED,eAAe,mBAAmB,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAE,oBAAoB,EAAE,MAAM,kBAAkB,CAAC;AAE5E,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,WAAW,CAAC;AAE/C,OAAO,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAEjD,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAE,KAAK,gBAAgB,EAAE,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EACL,OAAO,EACP,QAAQ,EACR,cAAc,EACd,eAAe,EACf,WAAW,EACX,cAAc,EACd,KAAK,aAAa,EAClB,KAAK,WAAW,EAChB,KAAK,iBAAiB,EACtB,KAAK,QAAQ,EACb,KAAK,kBAAkB,EACvB,KAAK,SAAS,EACd,KAAK,eAAe,EACpB,KAAK,cAAc,GACpB,MAAM,WAAW,CAAC;AACnB,OAAO,EAAE,UAAU,EAAE,YAAY,EAAE,WAAW,EAAE,oBAAoB,EAAE,KAAK,UAAU,EAAE,KAAK,UAAU,EAAE,KAAK,WAAW,EAAE,KAAK,UAAU,EAAE,KAAK,iBAAiB,EAAE,MAAM,aAAa,CAAC;AACvL,OAAO,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAC3C,OAAO,EAAE,qBAAqB,EAAE,cAAc,EAAE,KAAK,oBAAoB,EAAE,MAAM,iBAAiB,CAAC;AACnG,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAAE,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,KAAK,qBAAqB,EAAE,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,KAAK,sBAAsB,EAAE,KAAK,oBAAoB,EAAE,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAE,KAAK,mBAAmB,EAAE,KAAK,oBAAoB,EAAE,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAE,KAAK,aAAa,EAAE,MAAM,UAAU,CAAC;AAElF,2EAA2E;AAC3E,MAAM,WAAW,wBAAwB;IACvC,sHAAsH;IACtH,UAAU,CAAC,EAAE,MAAM,GAAG,QAAQ,CAAC;IAC/B,mEAAmE;IACnE,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,0HAA0H;IAC1H,MAAM,CAAC,EAAE,aAAa,CAAC;IACvB,0FAA0F;IAC1F,KAAK,CAAC,EAAE,OAAO,UAAU,CAAC,KAAK,CAAC;IAChC,yEAAyE;IACzE,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,qHAAqH;IACrH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd,yEAAyE;IACzE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,iEAAiE;IACjE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED;;;;;;GAMG;AACH,wBAAgB,mBAAmB,CAAC,OAAO,GAAE,oBAAyB,GAAG,aAAa,CAgCrF;AAED,eAAe,mBAAmB,CAAC"}
package/dist/index.js CHANGED
@@ -4,7 +4,9 @@
4
4
  * no search service: `parseCitedClaims` finds the claims and their addresses, `checkCitations` reads each page once and
5
5
  * grades each claim (word for word first, then through AgentOS's `CitationVerifier` when an embedding function is
6
6
  * given), `renderCitationLog` writes the markdown for a "Citation check log" section, and `citationReportFails` decides
7
- * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check.
7
+ * the result. The pack's one tool is `citation_verify`; the `agentos-citations` command runs the same check. The page
8
+ * reader is exported (`readPage` with its transports, the host rule and robots.txt on every hop, the licence read),
9
+ * and `WebPageSourceProvider` makes the open web a public source over a search the host gives.
8
10
  *
9
11
  * @module @framers/agentos-ext-citation-check
10
12
  */
@@ -13,7 +15,10 @@ import { CitationVerifyTool } from './tools/citationVerify.js';
13
15
  export * from './types.js';
14
16
  export { parseCitedClaims, CitationInputError, isHttpUrl } from './parse.js';
15
17
  export { normalizeForMatch, quoteFragments, locateQuote, lexicalNearest, withoutInvisible } from './match.js';
16
- export { pageKey } from './page.js';
18
+ export { pageKey, readPage, fetchTransport, egressTransport, PageRefusal, UNREAD_LICENCE, } from './page.js';
19
+ export { readRobots, robotsAllows, parseRobots, ROBOTS_PARSING_FLOOR } from './robots.js';
20
+ export { readLicence } from './licence.js';
21
+ export { WebPageSourceProvider, DEFAULT_ROBOTS } from './web-source.js';
17
22
  export { checkCitations, checkCitationDocument, CitationSetupError } from './runner.js';
18
23
  export { renderCitationLog, citationReportFails, FAILING_VERDICTS } from './report.js';
19
24
  export { openAIEmbeddings, managerEmbeddings } from './embeddings.js';
package/dist/index.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAGH,OAAO,EAAE,gBAAgB,EAAE,MAAM,iBAAiB,CAAC;AAEnD,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAG/D,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAyB,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EAAE,OAAO,EAAwC,MAAM,WAAW,CAAC;AAC1E,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAA6B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAA8B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAA0D,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAuD,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAsB,MAAM,UAAU,CAAC;AA0BlF;;;;;;GAMG;AACH,MAAM,UAAU,mBAAmB,CAAC,UAAgC,EAAE;IACpE,MAAM,OAAO,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,EAAE,CAA6B,CAAC;IACpE,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,MAAM,CAAC;IAChD,IAAI,UAAU,KAAK,MAAM,IAAI,UAAU,KAAK,QAAQ;QAAE,MAAM,IAAI,SAAS,CAAC,0CAA0C,IAAI,CAAC,SAAS,CAAC,UAAU,CAAC,EAAE,CAAC,CAAC;IAClJ,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,IAAI,OAAO,CAAC,MAAM,KAAK,QAAQ,IAAI,OAAO,CAAC,MAAM,KAAK,OAAO,EAAE,CAAC;QAC9F,MAAM,IAAI,SAAS,CAAC,uCAAuC,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;IAC/F,CAAC;IACD,MAAM,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC,eAAe,CAAC,CAAC;IACpD,MAAM,OAAO,GACX,OAAO,CAAC,OAAO;QACf,CAAC,UAAU,KAAK,QAAQ,CAAC,CAAC,CAAC,gBAAgB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,GAAG,CAAC,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;IACpK,MAAM,IAAI,GAAG,IAAI,kBAAkB,CAAC;QAClC,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;QACpG,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC7C,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAChE,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC5E,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACpE,CAAC,CAAC;IACH,OAAO;QACL,IAAI,EAAE,qCAAqC;QAC3C,WAAW,EAAE;YACX;gBACE,EAAE,EAAE,IAAI,CAAC,IAAI;gBACb,IAAI,EAAE,MAAM;gBACZ,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,EAAE;gBAChC,OAAO,EAAE,IAAI;gBACb,eAAe,EAAE,CAAC,EAAE,EAAE,EAAE,eAAe,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC;aAC3D;SACF;KACF,CAAC;AACJ,CAAC;AAED,eAAe,mBAAmB,CAAC"}
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAGH,OAAO,EAAE,gBAAgB,EAAE,MAAM,iBAAiB,CAAC;AAEnD,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAG/D,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,SAAS,EAAyB,MAAM,YAAY,CAAC;AACpG,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,WAAW,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9G,OAAO,EACL,OAAO,EACP,QAAQ,EACR,cAAc,EACd,eAAe,EACf,WAAW,EACX,cAAc,GASf,MAAM,WAAW,CAAC;AACnB,OAAO,EAAE,UAAU,EAAE,YAAY,EAAE,WAAW,EAAE,oBAAoB,EAA+F,MAAM,aAAa,CAAC;AACvL,OAAO,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAC3C,OAAO,EAAE,qBAAqB,EAAE,cAAc,EAA6B,MAAM,iBAAiB,CAAC;AACnG,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,kBAAkB,EAA6B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAA8B,MAAM,aAAa,CAAC;AACnH,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAA0D,MAAM,iBAAiB,CAAC;AAC9H,OAAO,EAAE,kBAAkB,EAAuD,MAAM,2BAA2B,CAAC;AACpH,OAAO,EAAE,cAAc,EAAE,kBAAkB,EAAsB,MAAM,UAAU,CAAC;AA0BlF;;;;;;GAMG;AACH,MAAM,UAAU,mBAAmB,CAAC,UAAgC,EAAE;IACpE,MAAM,OAAO,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,EAAE,CAA6B,CAAC;IACpE,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,MAAM,CAAC;IAChD,IAAI,UAAU,KAAK,MAAM,IAAI,UAAU,KAAK,QAAQ;QAAE,MAAM,IAAI,SAAS,CAAC,0CAA0C,IAAI,CAAC,SAAS,CAAC,UAAU,CAAC,EAAE,CAAC,CAAC;IAClJ,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,IAAI,OAAO,CAAC,MAAM,KAAK,QAAQ,IAAI,OAAO,CAAC,MAAM,KAAK,OAAO,EAAE,CAAC;QAC9F,MAAM,IAAI,SAAS,CAAC,uCAAuC,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;IAC/F,CAAC;IACD,MAAM,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC,eAAe,CAAC,CAAC;IACpD,MAAM,OAAO,GACX,OAAO,CAAC,OAAO;QACf,CAAC,UAAU,KAAK,QAAQ,CAAC,CAAC,CAAC,gBAAgB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,GAAG,CAAC,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;IACpK,MAAM,IAAI,GAAG,IAAI,kBAAkB,CAAC;QAClC,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;QACpG,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC7C,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAChE,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC5E,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,GAAG,CAAC,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACpE,CAAC,CAAC;IACH,OAAO;QACL,IAAI,EAAE,qCAAqC;QAC3C,WAAW,EAAE;YACX;gBACE,EAAE,EAAE,IAAI,CAAC,IAAI;gBACb,IAAI,EAAE,MAAM;gBACZ,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,EAAE;gBAChC,OAAO,EAAE,IAAI;gBACb,eAAe,EAAE,CAAC,EAAE,EAAE,EAAE,eAAe,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC;aAC3D;SACF;KACF,CAAC;AACJ,CAAC;AAED,eAAe,mBAAmB,CAAC"}
@@ -0,0 +1,31 @@
1
+ /**
2
+ * @file licence.ts
3
+ * @description The licence a page states, read from its markup in this order: a `<link rel="license" href>`, an
4
+ * `<a rel="license" href>`, a JSON-LD `license`, then any anchor to a Creative Commons deed
5
+ * (`creativecommons.org/licenses/<code>/<version>`) or public domain tool (`creativecommons.org/publicdomain/zero/<version>`,
6
+ * `creativecommons.org/publicdomain/mark/<version>`), wherever it stands, a footer's included. A deed is named from its address (`CC BY-SA 4.0`, `CC BY 3.0 US`,
7
+ * `CC0 1.0`); another address is named by the anchor's text, the link's `title` or the JSON-LD `name`, else by the
8
+ * address itself. The address is the one the page gives, resolved against the page's own. A licence written in prose
9
+ * with no address ("Text licensed under CC BY 4.0") is not read. The first 1,000 of each kind are read: links marked
10
+ * `rel="license"`, anchors marked so, and anchors to Creative Commons' site.
11
+ *
12
+ * @module agentos/extensions/research/citation-check/licence
13
+ */
14
+ import type { SourceLicence } from '@framers/agentos-ext-public-sources';
15
+ import { type MarkupScan } from './meta.js';
16
+ /**
17
+ * The licence a page states, or null when it states none this reader finds.
18
+ *
19
+ * @param html - The page's markup as served.
20
+ * @param baseUrl - The page's own address, after redirects, which relative addresses resolve against.
21
+ */
22
+ export declare function readLicence(html: string, baseUrl: string): SourceLicence | null;
23
+ /**
24
+ * {@link readLicence} over markup already read: one pass's links and anchors, and the page's JSON-LD nodes.
25
+ *
26
+ * @param scan - The pass over the markup, from `scanMarkup` with the same address.
27
+ * @param nodes - The page's JSON-LD nodes, from `jsonLdNodes`.
28
+ * @param baseUrl - The page's own address.
29
+ */
30
+ export declare function licenceOf(scan: MarkupScan, nodes: ReadonlyArray<Record<string, unknown>>, baseUrl: string): SourceLicence | null;
31
+ //# sourceMappingURL=licence.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"licence.d.ts","sourceRoot":"","sources":["../src/licence.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;GAYG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,qCAAqC,CAAC;AACzE,OAAO,EAAsF,KAAK,UAAU,EAAE,MAAM,WAAW,CAAC;AA0DhI;;;;;GAKG;AACH,wBAAgB,WAAW,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,aAAa,GAAG,IAAI,CAG/E;AAED;;;;;;GAMG;AACH,wBAAgB,SAAS,CAAC,IAAI,EAAE,UAAU,EAAE,KAAK,EAAE,aAAa,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,GAAG,aAAa,GAAG,IAAI,CAsBhI"}
@@ -0,0 +1,122 @@
1
+ /**
2
+ * @file licence.ts
3
+ * @description The licence a page states, read from its markup in this order: a `<link rel="license" href>`, an
4
+ * `<a rel="license" href>`, a JSON-LD `license`, then any anchor to a Creative Commons deed
5
+ * (`creativecommons.org/licenses/<code>/<version>`) or public domain tool (`creativecommons.org/publicdomain/zero/<version>`,
6
+ * `creativecommons.org/publicdomain/mark/<version>`), wherever it stands, a footer's included. A deed is named from its address (`CC BY-SA 4.0`, `CC BY 3.0 US`,
7
+ * `CC0 1.0`); another address is named by the anchor's text, the link's `title` or the JSON-LD `name`, else by the
8
+ * address itself. The address is the one the page gives, resolved against the page's own. A licence written in prose
9
+ * with no address ("Text licensed under CC BY 4.0") is not read. The first 1,000 of each kind are read: links marked
10
+ * `rel="license"`, anchors marked so, and anchors to Creative Commons' site.
11
+ *
12
+ * @module agentos/extensions/research/citation-check/licence
13
+ */
14
+ import { cleanAttribute, cleanText, cutText, jsonLdNodes, relNames, resolveHttp, scanMarkup } from './meta.js';
15
+ /** The Creative Commons licences by their codes in a deed's address. */
16
+ const CODES = new Set(['by', 'by-sa', 'by-nd', 'by-nc', 'by-nc-sa', 'by-nc-nd']);
17
+ /** A deed's version, such as `4.0`. */
18
+ const VERSION = /^\d\.\d$/u;
19
+ /** A ported deed's place, such as `us` or `igo`. */
20
+ const PLACE = /^[a-z]{2,3}$/u;
21
+ /** The most characters of an anchor's text a licence's name keeps. */
22
+ const ANCHOR_NAME_MAX = 100;
23
+ /** The path segments of a Creative Commons address, in lower case, or null for another host. */
24
+ function creativeCommonsPath(address) {
25
+ let url;
26
+ try {
27
+ url = new URL(address);
28
+ }
29
+ catch {
30
+ return null;
31
+ }
32
+ if (url.hostname.toLowerCase().replace(/^www\./u, '') !== 'creativecommons.org')
33
+ return null;
34
+ return url.pathname
35
+ .split('/')
36
+ .filter((part) => part !== '')
37
+ .map((part) => part.toLowerCase());
38
+ }
39
+ /** A Creative Commons deed's name from its address, or null for an address that is not a deed. */
40
+ function deedName(address) {
41
+ const parts = creativeCommonsPath(address);
42
+ if (parts === null)
43
+ return null;
44
+ const [kind, code = '', version = '', place = ''] = parts;
45
+ if (kind === 'licenses' && CODES.has(code) && VERSION.test(version))
46
+ return `CC ${code.toUpperCase()} ${version}${PLACE.test(place) ? ` ${place.toUpperCase()}` : ''}`;
47
+ if (kind === 'publicdomain' && code === 'zero' && VERSION.test(version))
48
+ return `CC0 ${version}`;
49
+ if (kind === 'publicdomain' && code === 'mark' && VERSION.test(version))
50
+ return `Public Domain Mark ${version}`;
51
+ return null;
52
+ }
53
+ /** A licence at an address, named by its deed or by the name the page gives it, else by the address. */
54
+ function licenceAt(url, given) {
55
+ return { name: deedName(url) ?? (given !== null && given !== '' ? given : url), url };
56
+ }
57
+ /** The licence a JSON-LD `license` value names: an address, an object with its `url` or `@id`, or a list of these. */
58
+ function jsonLdLicence(value, base) {
59
+ for (const item of Array.isArray(value) ? value : [value]) {
60
+ if (typeof item === 'string') {
61
+ const url = resolveHttp(item, base);
62
+ if (url !== null)
63
+ return licenceAt(url, null);
64
+ }
65
+ else if (item !== null && typeof item === 'object') {
66
+ const node = item;
67
+ const href = typeof node.url === 'string' ? node.url : typeof node['@id'] === 'string' ? node['@id'] : undefined;
68
+ const url = resolveHttp(href, base);
69
+ if (url !== null)
70
+ return licenceAt(url, typeof node.name === 'string' ? cutText(cleanText(node.name), ANCHOR_NAME_MAX) : null);
71
+ }
72
+ }
73
+ return null;
74
+ }
75
+ /**
76
+ * The licence a page states, or null when it states none this reader finds.
77
+ *
78
+ * @param html - The page's markup as served.
79
+ * @param baseUrl - The page's own address, after redirects, which relative addresses resolve against.
80
+ */
81
+ export function readLicence(html, baseUrl) {
82
+ const scan = scanMarkup(html, baseUrl);
83
+ return licenceOf(scan, jsonLdNodes(scan.jsonLd), baseUrl);
84
+ }
85
+ /**
86
+ * {@link readLicence} over markup already read: one pass's links and anchors, and the page's JSON-LD nodes.
87
+ *
88
+ * @param scan - The pass over the markup, from `scanMarkup` with the same address.
89
+ * @param nodes - The page's JSON-LD nodes, from `jsonLdNodes`.
90
+ * @param baseUrl - The page's own address.
91
+ */
92
+ export function licenceOf(scan, nodes, baseUrl) {
93
+ for (const attributes of scan.links) {
94
+ if (!relNames(attributes, 'license'))
95
+ continue;
96
+ const url = resolveHttp(attributes.get('href'), baseUrl);
97
+ if (url !== null)
98
+ return licenceAt(url, attributes.has('title') ? cutText(cleanAttribute(attributes.get('title')), ANCHOR_NAME_MAX) : null);
99
+ }
100
+ for (const anchor of scan.anchors) {
101
+ if (!relNames(anchor.attributes, 'license'))
102
+ continue;
103
+ const url = resolveHttp(anchor.attributes.get('href'), baseUrl);
104
+ if (url !== null)
105
+ return licenceAt(url, cutText(cleanText(anchor.text), ANCHOR_NAME_MAX));
106
+ }
107
+ for (const node of nodes) {
108
+ if (!('license' in node))
109
+ continue;
110
+ const licence = jsonLdLicence(node.license, baseUrl);
111
+ if (licence !== null)
112
+ return licence;
113
+ }
114
+ for (const anchor of scan.anchors) {
115
+ const url = resolveHttp(anchor.attributes.get('href'), baseUrl);
116
+ const name = url === null ? null : deedName(url);
117
+ if (url !== null && name !== null)
118
+ return { name, url };
119
+ }
120
+ return null;
121
+ }
122
+ //# sourceMappingURL=licence.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"licence.js","sourceRoot":"","sources":["../src/licence.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;GAYG;AAGH,OAAO,EAAE,cAAc,EAAE,SAAS,EAAE,OAAO,EAAE,WAAW,EAAE,QAAQ,EAAE,WAAW,EAAE,UAAU,EAAmB,MAAM,WAAW,CAAC;AAEhI,wEAAwE;AACxE,MAAM,KAAK,GAAwB,IAAI,GAAG,CAAC,CAAC,IAAI,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,UAAU,EAAE,UAAU,CAAC,CAAC,CAAC;AACtG,uCAAuC;AACvC,MAAM,OAAO,GAAG,WAAW,CAAC;AAC5B,oDAAoD;AACpD,MAAM,KAAK,GAAG,eAAe,CAAC;AAC9B,sEAAsE;AACtE,MAAM,eAAe,GAAG,GAAG,CAAC;AAE5B,gGAAgG;AAChG,SAAS,mBAAmB,CAAC,OAAe;IAC1C,IAAI,GAAQ,CAAC;IACb,IAAI,CAAC;QACH,GAAG,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC;IACzB,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;IACd,CAAC;IACD,IAAI,GAAG,CAAC,QAAQ,CAAC,WAAW,EAAE,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,KAAK,qBAAqB;QAAE,OAAO,IAAI,CAAC;IAC7F,OAAO,GAAG,CAAC,QAAQ;SAChB,KAAK,CAAC,GAAG,CAAC;SACV,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;SAC7B,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,WAAW,EAAE,CAAC,CAAC;AACvC,CAAC;AAED,kGAAkG;AAClG,SAAS,QAAQ,CAAC,OAAe;IAC/B,MAAM,KAAK,GAAG,mBAAmB,CAAC,OAAO,CAAC,CAAC;IAC3C,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,IAAI,CAAC;IAChC,MAAM,CAAC,IAAI,EAAE,IAAI,GAAG,EAAE,EAAE,OAAO,GAAG,EAAE,EAAE,KAAK,GAAG,EAAE,CAAC,GAAG,KAAK,CAAC;IAC1D,IAAI,IAAI,KAAK,UAAU,IAAI,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC;QAAE,OAAO,MAAM,IAAI,CAAC,WAAW,EAAE,IAAI,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,WAAW,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC;IACvK,IAAI,IAAI,KAAK,cAAc,IAAI,IAAI,KAAK,MAAM,IAAI,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC;QAAE,OAAO,OAAO,OAAO,EAAE,CAAC;IACjG,IAAI,IAAI,KAAK,cAAc,IAAI,IAAI,KAAK,MAAM,IAAI,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC;QAAE,OAAO,sBAAsB,OAAO,EAAE,CAAC;IAChH,OAAO,IAAI,CAAC;AACd,CAAC;AAED,wGAAwG;AACxG,SAAS,SAAS,CAAC,GAAW,EAAE,KAAoB;IAClD,OAAO,EAAE,IAAI,EAAE,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,KAAK,KAAK,IAAI,IAAI,KAAK,KAAK,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,GAAG,EAAE,CAAC;AACxF,CAAC;AAED,sHAAsH;AACtH,SAAS,aAAa,CAAC,KAAc,EAAE,IAAY;IACjD,KAAK,MAAM,IAAI,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,EAAE,CAAC;QAC1D,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;YAC7B,MAAM,GAAG,GAAG,WAAW,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;YACpC,IAAI,GAAG,KAAK,IAAI;gBAAE,OAAO,SAAS,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;QAChD,CAAC;aAAM,IAAI,IAAI,KAAK,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;YACrD,MAAM,IAAI,GAAG,IAA+B,CAAC;YAC7C,MAAM,IAAI,GAAG,OAAO,IAAI,CAAC,GAAG,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAE,IAAI,CAAC,KAAK,CAAY,CAAC,CAAC,CAAC,SAAS,CAAC;YAC7H,MAAM,GAAG,GAAG,WAAW,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;YACpC,IAAI,GAAG,KAAK,IAAI;gBAAE,OAAO,SAAS,CAAC,GAAG,EAAE,OAAO,IAAI,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,eAAe,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QACjI,CAAC;IACH,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,WAAW,CAAC,IAAY,EAAE,OAAe;IACvD,MAAM,IAAI,GAAG,UAAU,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;IACvC,OAAO,SAAS,CAAC,IAAI,EAAE,WAAW,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,OAAO,CAAC,CAAC;AAC5D,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,SAAS,CAAC,IAAgB,EAAE,KAA6C,EAAE,OAAe;IACxG,KAAK,MAAM,UAAU,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;QACpC,IAAI,CAAC,QAAQ,CAAC,UAAU,EAAE,SAAS,CAAC;YAAE,SAAS;QAC/C,MAAM,GAAG,GAAG,WAAW,CAAC,UAAU,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,OAAO,CAAC,CAAC;QACzD,IAAI,GAAG,KAAK,IAAI;YAAE,OAAO,SAAS,CAAC,GAAG,EAAE,UAAU,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,cAAc,CAAC,UAAU,CAAC,GAAG,CAAC,OAAO,CAAW,CAAC,EAAE,eAAe,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;IACxJ,CAAC;IACD,KAAK,MAAM,MAAM,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;QAClC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,UAAU,EAAE,SAAS,CAAC;YAAE,SAAS;QACtD,MAAM,GAAG,GAAG,WAAW,CAAC,MAAM,CAAC,UAAU,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,OAAO,CAAC,CAAC;QAChE,IAAI,GAAG,KAAK,IAAI;YAAE,OAAO,SAAS,CAAC,GAAG,EAAE,OAAO,CAAC,SAAS,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,eAAe,CAAC,CAAC,CAAC;IAC5F,CAAC;IACD,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,IAAI,CAAC,CAAC,SAAS,IAAI,IAAI,CAAC;YAAE,SAAS;QACnC,MAAM,OAAO,GAAG,aAAa,CAAC,IAAI,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QACrD,IAAI,OAAO,KAAK,IAAI;YAAE,OAAO,OAAO,CAAC;IACvC,CAAC;IACD,KAAK,MAAM,MAAM,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;QAClC,MAAM,GAAG,GAAG,WAAW,CAAC,MAAM,CAAC,UAAU,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,OAAO,CAAC,CAAC;QAChE,MAAM,IAAI,GAAG,GAAG,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;QACjD,IAAI,GAAG,KAAK,IAAI,IAAI,IAAI,KAAK,IAAI;YAAE,OAAO,EAAE,IAAI,EAAE,GAAG,EAAE,CAAC;IAC1D,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC"}
package/dist/meta.d.ts ADDED
@@ -0,0 +1,66 @@
1
+ /**
2
+ * @file meta.ts
3
+ * @description What a page says about itself in its markup as served: its `<title>`, its `<meta>` tags, the links and
4
+ * anchors a licence is read from, and its JSON-LD. One pass over the markup, in document order, finds them: a comment
5
+ * is passed over whole (so a tag or a JSON-LD block inside one is not the page's), the content of a script, a style, a
6
+ * template, a `noscript`, a `textarea` and the title is not read for tags (so a comment opener or a tag written inside
7
+ * one is theirs), and a tag runs to the first `>` outside a quoted attribute value. Each step of the pass starts where
8
+ * the one before it ended, so the time grows with the page alone, however its tags are left open. Of the links and
9
+ * anchors it keeps those a licence is read from, at most 1,000 of each kind, and reads the text of a `rel="license"`
10
+ * anchor alone, so what it holds does not grow with the page. A value a script writes later is not seen. Character
11
+ * references are read with the `entities` package's HTML decoder.
12
+ *
13
+ * @module agentos/extensions/research/citation-check/meta
14
+ */
15
+ /** The most characters a page's title or its publisher's name keeps. */
16
+ export declare const NAME_MAX = 255;
17
+ /** One element's attributes by name in lower case, the first of a name kept, the values as written. */
18
+ export type Attributes = Map<string, string>;
19
+ /** What one pass over a page's markup found. */
20
+ export interface MarkupScan {
21
+ /** The first `<title>`'s content as written; null when the page has none. */
22
+ title: string | null;
23
+ /** The first `content` of each `<meta>` by its `property`, else its `name`, else its `itemprop`, in lower case. */
24
+ metas: Map<string, string>;
25
+ /** Each `<link>` whose `rel` names `license`, its attributes, in document order; at most 1,000. */
26
+ links: Attributes[];
27
+ /**
28
+ * Each `<a>` a licence is read from, in document order: one whose `rel` names `license`, with the start of its
29
+ * content as written, and one whose address, resolved against the page's own, is on creativecommons.org, with no
30
+ * content; at most 1,000 of each kind.
31
+ */
32
+ anchors: Array<{
33
+ attributes: Attributes;
34
+ text: string;
35
+ }>;
36
+ /** The content of each JSON-LD script, in document order. */
37
+ jsonLd: string[];
38
+ }
39
+ /** At most `max` characters of a text, never half of a surrogate pair. */
40
+ export declare function cutText(text: string, max: number): string;
41
+ /** Markup as a reader of it sees it: tags removed, references read, NFC, invisible characters gone, white space collapsed. */
42
+ export declare function cleanText(markup: string): string;
43
+ /** An attribute's value as a reader sees it: references read, NFC, invisible characters gone, white space collapsed. */
44
+ export declare function cleanAttribute(value: string): string;
45
+ /**
46
+ * One pass over a page's markup; see the module's description.
47
+ *
48
+ * @param html - The page's markup as served.
49
+ * @param baseUrl - The page's own address, which an anchor's address is resolved against.
50
+ */
51
+ export declare function scanMarkup(html: string, baseUrl: string): MarkupScan;
52
+ /** Whether an element's `rel` names a link type, its tokens read without case. */
53
+ export declare function relNames(attributes: Attributes, type: string): boolean;
54
+ /** An address as an attribute writes it, its references read and resolved against the page's own; null when it does not parse or is not http or https. */
55
+ export declare function resolveHttp(href: string | undefined, base: string): string | null;
56
+ /** The page's `<title>`, its text read and cut to {@link NAME_MAX} characters; null when it has none. */
57
+ export declare function pageTitle(scan: MarkupScan): string | null;
58
+ /** The JSON-LD objects of the page's blocks, in document order: each block's top object or array, and the nodes of each `@graph`. */
59
+ export declare function jsonLdNodes(blocks: readonly string[]): Array<Record<string, unknown>>;
60
+ /** A date as milliseconds since 1970, or null for one that does not parse or names no four-digit year. */
61
+ export declare function dateOf(text: string | null | undefined): number | null;
62
+ /** The site's name: `og:site_name`, else the first JSON-LD `publisher.name`; null when the page names none. */
63
+ export declare function publisherOf(metas: Map<string, string>, nodes: ReadonlyArray<Record<string, unknown>>): string | null;
64
+ /** The page's own date: `article:modified_time`, else the first JSON-LD `dateModified`, else `Last-Modified`. */
65
+ export declare function pageDateOf(metas: Map<string, string>, nodes: ReadonlyArray<Record<string, unknown>>, lastModified: string | null): number | null;
66
+ //# sourceMappingURL=meta.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"meta.d.ts","sourceRoot":"","sources":["../src/meta.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;AAMH,wEAAwE;AACxE,eAAO,MAAM,QAAQ,MAAM,CAAC;AAgB5B,uGAAuG;AACvG,MAAM,MAAM,UAAU,GAAG,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;AAE7C,gDAAgD;AAChD,MAAM,WAAW,UAAU;IACzB,6EAA6E;IAC7E,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;IACrB,mHAAmH;IACnH,KAAK,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAC3B,mGAAmG;IACnG,KAAK,EAAE,UAAU,EAAE,CAAC;IACpB;;;;OAIG;IACH,OAAO,EAAE,KAAK,CAAC;QAAE,UAAU,EAAE,UAAU,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACzD,6DAA6D;IAC7D,MAAM,EAAE,MAAM,EAAE,CAAC;CAClB;AAED,0EAA0E;AAC1E,wBAAgB,OAAO,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,GAAG,MAAM,CAMzD;AAED,8HAA8H;AAC9H,wBAAgB,SAAS,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,CAEhD;AAED,wHAAwH;AACxH,wBAAgB,cAAc,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAEpD;AAmDD;;;;;GAKG;AACH,wBAAgB,UAAU,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,UAAU,CA8DpE;AAED,kFAAkF;AAClF,wBAAgB,QAAQ,CAAC,UAAU,EAAE,UAAU,EAAE,IAAI,EAAE,MAAM,GAAG,OAAO,CAEtE;AAED,0JAA0J;AAC1J,wBAAgB,WAAW,CAAC,IAAI,EAAE,MAAM,GAAG,SAAS,EAAE,IAAI,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI,CASjF;AAED,yGAAyG;AACzG,wBAAgB,SAAS,CAAC,IAAI,EAAE,UAAU,GAAG,MAAM,GAAG,IAAI,CAIzD;AAED,qIAAqI;AACrI,wBAAgB,WAAW,CAAC,MAAM,EAAE,SAAS,MAAM,EAAE,GAAG,KAAK,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,CA0BrF;AAED,0GAA0G;AAC1G,wBAAgB,MAAM,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,GAAG,MAAM,GAAG,IAAI,CAIrE;AAQD,+GAA+G;AAC/G,wBAAgB,WAAW,CAAC,KAAK,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,KAAK,EAAE,aAAa,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,GAAG,MAAM,GAAG,IAAI,CAepH;AAED,iHAAiH;AACjH,wBAAgB,UAAU,CAAC,KAAK,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,KAAK,EAAE,aAAa,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,EAAE,YAAY,EAAE,MAAM,GAAG,IAAI,GAAG,MAAM,GAAG,IAAI,CAQhJ"}