openbriefing 0.1.1 → 0.3.0

package/README.md

@@ -2,17 +2,24 @@
 
 This package is the reference parser and checker for the [Open Briefing Standard (OBS)](https://openbriefingstandard.org). OBS is how a publisher says how their content should be summarized and presented when software consumes it.
 
-**Version 0.1.** It contains the parser and `openbriefing check`. Later versions add `generate`, `preview`, `backfill`, `review` and a Ghost integration.
+**Version 0.3.** It contains the parser and `openbriefing check`, and reads spec version 1 (release candidate). One tag, `obs:summary`, is enough: `obs:modified` is optional, and the parser reports where a modified time came from. `check` reads a page, a feed or a whole sitemap from the web.
 
-## Check a page or a feed
+## Check a page, a feed or a site
 
 ```bash
+npx openbriefing check https://example.com/a-post/
 npx openbriefing check ./post.html
-npx openbriefing check ./feed.xml
-curl -s https://example.com/post/ | npx openbriefing check - --url https://example.com/post/
+npx openbriefing check https://example.com/rss/ --feed --pages
+npx openbriefing check https://example.com/sitemap.xml --sitemap
 ```
 
-`check` exits 1 when the tags are unusable, so a site's CI can gate on it. Add `--json` to get the parsed result as JSON.
+- **A page:** prints each form with its word count, the keep lines, the highlights (and whether each is in the page's text), the modified time and its source, and a fingerprint.
+- **`--feed`:** checks every item. With `--pages` it also fetches each item's page and says which of the two stands.
+- **`--sitemap`:** one line per page (`ok`, `ok, 2 warnings`, `no tags`, `not usable`) and a total. `--limit 20` checks the first 20.
+
+`check` exits 1 when tags are present but unusable, so a site's CI can gate on it. A page with no tags is not an error in a sitemap. Add `--json` for the parsed result. For a file with no canonical link, `--url` gives the page's address; `-` reads standard input.
+
+Fetching is bounded: `http` and `https` only, 10 seconds, 5 MB, 5 redirects, and it names itself `openbriefing-check/<version>`.
 
 ## Use the parser
 
@@ -22,6 +29,8 @@ import { parseObs } from "openbriefing";
 
 const dom = new JSDOM(html, { url });
 const { brief, errors, warnings } = parseObs(dom.window.document, url);
+// brief.summaries, brief.keep, brief.highlights (each { text, words, label?, inContent? }),
+// brief.modified (optional) and brief.modifiedFrom ("obs" | "article" | "feed" | "none"), …
 ```
 
 | Export | What it does |
@@ -29,10 +38,11 @@ const { brief, errors, warnings } = parseObs(dom.window.document, url);
 | `parseObs(document, pageUrl)` | Reads a page's `obs:` meta tags |
 | `parseFeedObs(itemElement, itemUrl?)` | Reads one RSS `<item>` or Atom `<entry>` |
 | `validateObs(object)` | Checks an object already in the `ObsBrief` shape |
-| `mergeObs(page, feed)` | Applies the feed-against-page rule: the later `obs:modified` stands, and the page wins a tie |
-| `tierFor(words)`, `SIZES`, `normalizeUrl` | Helpers that use the spec's numbers |
+| `mergeObs(page, feed)` | Applies the feed-against-page rule: the later effective modified time stands; the page wins a tie, and when either has no time |
+| `fingerprint(brief)` | A stable 16-character hash of every text value and setting. Compare two to tell whether a brief changed, with or without a modified time |
+| `tierFor(words)`, `SIZES`, `normalizeUrl`, `looseText` | Helpers that use the spec's numbers and its loose text comparison |
 
-The parser has no runtime dependencies. It takes any DOM `Document` (jsdom, linkedom, or a browser's `DOMParser`), makes no network requests, and never throws.
+The parser has no runtime dependencies. It takes any DOM `Document` (jsdom, linkedom, or a browser's `DOMParser`), makes no network requests, and never throws. The bounded fetch that `check` uses is exported separately as `openbriefing/fetch`.
 
 The test fixtures ship with the package in `openbriefing/fixtures/`, with `expected.json`.
 

package/dist/cli.js

@@ -1,52 +1,83 @@
 #!/usr/bin/env node
 /**
- * openbriefing check <file | -> [--url <page url>] [--json]
+ * openbriefing check <url | file | -> [--url <page url>] [--json]
+ * openbriefing check <feed url | file> --feed [--pages]
+ * openbriefing check <sitemap url | file> --sitemap [--limit <n>]
  *
- * Checks a local HTML page or an RSS/Atom feed for Open Briefing Standard
- * tags and prints what a reader sees. Exits 1 when the tags are unusable.
- * Version 0.1 reads local files and standard input only; it makes no
- * network requests.
+ * Checks a page, a feed or a whole sitemap for Open Briefing Standard tags
+ * and prints what a reader sees. Exits 1 when tags are present but unusable.
  */
 import { readFileSync } from "node:fs";
 import { basename, resolve } from "node:path";
 import { pathToFileURL } from "node:url";
 import { JSDOM } from "jsdom";
-import { countWords, parseFeedObs, parseObs, SIZES } from "./obs.js";
-const USAGE = `Usage: openbriefing check <file | -> [--url <page url>] [--json]
+import { DEFAULT_LIMITS, fetchText, isHttpUrl, mapLimit, userAgent } from "./fetch.js";
+import { countWords, fingerprint, mergeObs, parseFeedObs, parseObs, SIZES } from "./obs.js";
+const VERSION = (() => {
+    try {
+        return JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8")).version;
+    }
+    catch {
+        return "0";
+    }
+})();
+const LIMITS = { ...DEFAULT_LIMITS, userAgent: userAgent(VERSION) };
+const USAGE = `Usage:
+  openbriefing check <url | file | -> [--url <page url>] [--json]
+  openbriefing check <feed url | file> --feed [--pages]
+  openbriefing check <sitemap url | file> --sitemap [--limit <n>]
 
-Checks a page or a feed for Open Briefing Standard tags.
-  --url   the page's address, used when the page has no canonical link
-  --json  print the parsed result as JSON
+Checks a page, a feed or a sitemap for Open Briefing Standard tags.
+  --url      the page's address, when checking a file with no canonical link
+  --feed     treat the input as an RSS or Atom feed (detected when omitted)
+  --pages    with a feed: also fetch each item's page and say which one stands
+  --sitemap  check every page a sitemap lists, one line each
+  --limit    with a sitemap: check only the first <n> pages
+  --json     print the parsed result as JSON
 Spec: https://openbriefingstandard.org`;
 function parseArgs(argv) {
-    const args = { json: false };
+    const args = { json: false, feed: false, pages: false, sitemap: false };
     for (let i = 0; i < argv.length; i++) {
         const a = argv[i];
         if (a === "--json")
             args.json = true;
-        else if (a === "--url")
-            args.url = argv[++i];
+        else if (a === "--feed")
+            args.feed = true;
+        else if (a === "--pages")
+            args.pages = true;
+        else if (a === "--sitemap")
+            args.sitemap = true;
+        else if (a === "--url") {
+            const v = argv[++i];
+            if (v === undefined)
+                args.bad = "--url needs an address";
+            else
+                args.url = v;
+        }
+        else if (a === "--limit") {
+            const n = Number.parseInt(argv[++i] ?? "", 10);
+            if (!Number.isInteger(n) || n < 1)
+                args.bad = "--limit needs a whole number above 0";
+            else
+                args.limit = n;
+        }
         else if (a === "-h" || a === "--help")
             args.command = "help";
+        else if (a === "-v" || a === "--version")
+            args.command = "version";
+        else if (a.startsWith("--"))
+            args.bad = `unknown option ${a}`;
         else if (args.command === undefined)
             args.command = a;
-        else if (args.file === undefined)
-            args.file = a;
+        else if (args.target === undefined)
+            args.target = a;
     }
     return args;
 }
-function readInput(file) {
-    return file === "-" ? readFileSync(0, "utf8") : readFileSync(file, "utf8");
-}
-function isFeed(text, file) {
-    if (/\.(xml|rss|atom)$/iu.test(file))
-        return true;
-    const head = text.slice(0, 500).trimStart();
-    return head.startsWith("<?xml") ? /<(rss|feed)\b/iu.test(text.slice(0, 2000)) : /^<(rss|feed)\b/iu.test(head);
-}
 /** Page text is untrusted: show control characters (ANSI escapes and the like) instead of sending them to the terminal. */
 const safe = (s) => s.replace(/[\u0000-\u0008\u000b-\u001f\u007f-\u009f]/gu, (c) => `\\x${c.charCodeAt(0).toString(16).padStart(2, "0")}`);
 const cut = (s, n = 60) => safe((s.length > n ? `${s.slice(0, n - 1)}…` : s).replace(/\n/gu, " "));
+const MODIFIED_FROM = { obs: "obs:modified", article: "article:modified_time", feed: "the feed item's date", none: "" };
 function describe(label, r) {
     const lines = [safe(label)];
     const b = r.brief;
@@ -61,13 +92,22 @@ function describe(label, r) {
             row(`summary`, `${s.tier.padEnd(6)} ${String(s.words).padStart(5)} words  (target ${lo}–${hi})  ${s.from}`);
         }
         b.keep.forEach((k, i) => row(i === 0 ? "keep (core)" : `keep ${i + 1}`, cut(k)));
+        b.highlights.forEach((h, i) => {
+            const notes = [`${h.words}/${SIZES.highlight} words`];
+            if (h.label !== undefined)
+                notes.push(cut(h.label, 30));
+            if (h.inContent === false)
+                notes.push("not in the text");
+            row(`highlight ${i + 1}`, `${cut(h.text)}  (${notes.join(", ")})`);
+        });
         row("derive", `${b.derive.mode}; translate ${b.derive.translate ? "yes" : "no"}; combine ${b.derive.combine ? "yes" : "no"}`);
         if (b.attribution !== undefined)
             row("attribution", cut(b.attribution));
         row("preference", b.preference);
         if (b.lang !== undefined)
             row("lang", safe(b.lang));
-        row("modified", safe(b.modified));
+        row("modified", b.modified === undefined ? "none; readers compare the text of the tags" : `${safe(b.modified)}  (from ${MODIFIED_FROM[b.modifiedFrom]})`);
+        row("fingerprint", fingerprint(b));
         if (b.url !== undefined)
             row("url", `${safe(b.url)}${b.describesThisPage ? "" : "  (describes another page)"}`);
     }
@@ -78,46 +118,205 @@ function describe(label, r) {
     lines.push(b ? `  ok: usable, ${r.warnings.length} warning${r.warnings.length === 1 ? "" : "s"}` : "  not usable");
     return lines.join("\n");
 }
-function check(args) {
-    if (!args.file) {
-        console.error(USAGE);
-        return 2;
+async function load(target, urlOverride) {
+    if (isHttpUrl(target)) {
+        const res = await fetchText(target, LIMITS);
+        if (!res.ok)
+            return { error: `Cannot fetch ${safe(target)}: ${safe(res.error)}` };
+        return { text: res.text, url: urlOverride ?? res.url, label: res.url, contentType: res.contentType };
     }
-    let text;
     try {
-        text = readInput(args.file);
+        const text = target === "-" ? readFileSync(0, "utf8") : readFileSync(target, "utf8");
+        const url = urlOverride ?? (target === "-" ? "https://localhost/" : pathToFileURL(resolve(target)).href);
+        return { text, url, label: target === "-" ? "(standard input)" : basename(target), contentType: "" };
     }
     catch (err) {
-        console.error(`Cannot read ${args.file}: ${err instanceof Error ? err.message : String(err)}`);
-        return 2;
+        return { error: `Cannot read ${safe(target)}: ${err instanceof Error ? err.message : String(err)}` };
+    }
+}
+function looksLikeFeed(text, name, contentType) {
+    if (/\b(rss|atom)\+xml\b/iu.test(contentType))
+        return true;
+    if (/\.(rss|atom)$/iu.test(name))
+        return true;
+    const head = text.slice(0, 2000).trimStart();
+    return /^(?:<\?xml[^>]*\?>\s*)?(?:<!--[\s\S]*?-->\s*)*(?:<\?[^>]*\?>\s*)*<(rss|feed)\b/iu.test(head);
+}
+const parsePage = (text, url) => parseObs(new JSDOM(text, { url }).window.document, url);
+function feedItems(text) {
+    const doc = new JSDOM(text, { contentType: "text/xml" }).window.document;
+    return [...Array.from(doc.getElementsByTagName("item")), ...Array.from(doc.getElementsByTagName("entry"))];
+}
+function itemLink(item) {
+    for (const el of Array.from(item.children)) {
+        if (el.localName !== "link" || el.namespaceURI === "https://openbriefingstandard.org/ns#")
+            continue;
+        const href = el.getAttribute("href");
+        if (href !== null) {
+            const rel = el.getAttribute("rel");
+            if (rel === null || rel === "alternate")
+                return href.trim();
+        }
+        else if (el.textContent?.trim())
+            return el.textContent.trim();
+    }
+    return undefined;
+}
+const NO_PAGE_TAGS = "No obs: tags on the page";
+const NO_ITEM_TAGS = "No OBS elements on the item";
+const hasNoTags = (r) => r.brief === undefined && (r.errors[0] === NO_PAGE_TAGS || r.errors[0] === NO_ITEM_TAGS);
+async function checkFeed(args, loaded) {
+    const items = feedItems(loaded.text);
+    if (items.length === 0) {
+        if (args.json)
+            console.log("[]");
+        else
+            console.log(`${safe(loaded.label)}\n  not usable: no <item> or <entry> elements`);
+        return 1;
     }
-    const pageUrl = args.url ?? (args.file === "-" ? "https://localhost/" : pathToFileURL(resolve(args.file)).href);
-    const label = args.file === "-" ? "(standard input)" : basename(args.file);
-    if (isFeed(text, args.file)) {
-        const doc = new JSDOM(text, { contentType: "text/xml" }).window.document;
-        const items = [...Array.from(doc.getElementsByTagName("item")), ...Array.from(doc.getElementsByTagName("entry"))];
-        const results = items.map((item) => parseFeedObs(item));
+    const results = items.map((item) => parseFeedObs(item));
+    if (!args.pages) {
         if (args.json)
             console.log(JSON.stringify(results, null, 2));
         else
-            results.forEach((r, i) => console.log(describe(`${label}, item ${i + 1}`, r)));
-        if (items.length === 0) {
-            if (!args.json)
-                console.log(`${label}\n  not usable: no <item> or <entry> elements`);
-            return 1;
+            results.forEach((r, i) => console.log(describe(`${loaded.label}, item ${i + 1}`, r)));
+        return results.some((r) => r.brief === undefined && !hasNoTags(r)) || results.every((r) => r.brief === undefined) ? 1 : 0;
+    }
+    // --pages: fetch each item's page and apply the feed-against-page rule (§14).
+    const rows = await mapLimit(items, 4, async (item, i) => {
+        const feed = results[i];
+        const link = itemLink(item);
+        if (link === undefined || !isHttpUrl(link))
+            return { link, feed, note: "the item has no link to fetch" };
+        const res = await fetchText(link, LIMITS);
+        if (!res.ok)
+            return { link, feed, note: `page not fetched: ${res.error}` };
+        const page = parsePage(res.text, res.url);
+        const stands = mergeObs(page.brief, feed.brief);
+        const which = stands === undefined ? "neither" : stands === page.brief ? "page" : "feed";
+        const same = page.brief && feed.brief ? fingerprint(page.brief) === fingerprint(feed.brief) : undefined;
+        return { link, feed, page, stands: which, same };
+    });
+    if (args.json)
+        console.log(JSON.stringify(rows, null, 2));
+    else {
+        rows.forEach((row, i) => {
+            console.log(describe(`${loaded.label}, item ${i + 1}${row.link ? `  ${row.link}` : ""}`, row.feed));
+            if ("note" in row)
+                console.log(`  page     ${safe(row.note ?? "")}`);
+            else {
+                const p = row.page;
+                const state = p.brief ? `usable, ${p.warnings.length} warning${p.warnings.length === 1 ? "" : "s"}` : hasNoTags(p) ? "no tags" : "not usable";
+                const agree = row.same === undefined ? "" : row.same ? "; page and feed say the same" : "; page and feed differ";
+                console.log(`  page     ${state}${agree}; the ${row.stands} stands`);
+                if (!p.brief && !hasNoTags(p))
+                    for (const e of p.errors)
+                        console.log(`  page error  ${safe(e)}`);
+            }
+        });
+    }
+    const broken = rows.some((row) => (row.feed.brief === undefined && !hasNoTags(row.feed)) || ("page" in row && row.page !== undefined && row.page.brief === undefined && !hasNoTags(row.page)));
+    return broken ? 1 : 0;
+}
+const unescapeXml = (s) => s.replace(/&lt;/gu, "<").replace(/&gt;/gu, ">").replace(/&quot;/gu, '"').replace(/&apos;/gu, "'").replace(/&amp;/gu, "&");
+function locs(xml) {
+    return [...xml.matchAll(/<loc>\s*(?:<!\[CDATA\[)?\s*([^<\]]+?)\s*(?:\]\]>)?\s*<\/loc>/giu)].map((m) => unescapeXml(m[1]));
+}
+/** Image and video sitemaps nest their own <loc> elements; only a <url>'s direct one is a page. */
+function pageLocs(xml) {
+    const out = [];
+    for (const m of xml.matchAll(/<url\b[^>]*>([\s\S]*?)<\/url>/giu)) {
+        const inner = m[1].replace(/<(image|video|news):[a-z]+\b[\s\S]*?<\/\1:[a-z]+>/giu, "");
+        const first = locs(inner)[0];
+        if (first !== undefined)
+            out.push(first);
+    }
+    return out;
+}
+async function checkSitemap(args, loaded) {
+    let pages;
+    if (/<sitemapindex\b/iu.test(loaded.text)) {
+        const children = locs(loaded.text).filter(isHttpUrl).slice(0, 50);
+        const sets = await mapLimit(children, 4, async (child) => {
+            const res = await fetchText(child, LIMITS);
+            return res.ok ? pageLocs(res.text) : [];
+        });
+        pages = sets.flat();
+    }
+    else {
+        pages = pageLocs(loaded.text);
+    }
+    pages = [...new Set(pages.filter(isHttpUrl))];
+    const total = pages.length;
+    if (args.limit !== undefined)
+        pages = pages.slice(0, args.limit);
+    if (pages.length === 0) {
+        if (args.json)
+            console.log("[]");
+        else
+            console.log(`${safe(loaded.label)}\n  no page addresses found`);
+        return 1;
+    }
+    const rows = await mapLimit(pages, 4, async (url) => {
+        const res = await fetchText(url, LIMITS);
+        if (!res.ok)
+            return { url, status: "not fetched", warnings: 0, detail: res.error };
+        const result = parsePage(res.text, res.url);
+        if (result.brief)
+            return { url, status: "ok", warnings: result.warnings.length, result };
+        if (hasNoTags(result))
+            return { url, status: "no tags", warnings: 0, result };
+        return { url, status: "not usable", warnings: result.warnings.length, detail: result.errors.join("; "), result };
+    });
+    const count = (s) => rows.filter((r) => r.status === s).length;
+    if (args.json)
+        console.log(JSON.stringify(rows, null, 2));
+    else {
+        for (const r of rows) {
+            const state = r.status === "ok" && r.warnings > 0 ? `ok, ${r.warnings} warning${r.warnings === 1 ? "" : "s"}` : r.status;
+            console.log(`${state.padEnd(16)} ${safe(r.url)}${r.detail && r.status !== "ok" ? `  (${safe(r.detail)})` : ""}`);
         }
-        return results.every((r) => r.brief) ? 0 : 1;
+        const parts = [`${rows.length} page${rows.length === 1 ? "" : "s"}`, `${count("ok")} ok`, `${count("no tags")} with no tags`, `${count("not usable")} not usable`];
+        if (count("not fetched") > 0)
+            parts.push(`${count("not fetched")} not fetched`);
+        console.log(`\n${parts.join(", ")}${rows.length < total ? ` (first ${rows.length} of ${total})` : ""}`);
+    }
+    return count("not usable") > 0 ? 1 : 0;
+}
+async function check(args) {
+    if (!args.target || args.bad) {
+        if (args.bad)
+            console.error(`${args.bad}\n`);
+        console.error(USAGE);
+        return 2;
     }
-    const doc = new JSDOM(text, { url: pageUrl }).window.document;
-    const r = parseObs(doc, pageUrl);
-    console.log(args.json ? JSON.stringify(r, null, 2) : describe(label, r));
+    const loaded = await load(args.target, args.url);
+    if ("error" in loaded) {
+        console.error(loaded.error);
+        return 2;
+    }
+    if (args.sitemap)
+        return checkSitemap(args, loaded);
+    if (args.feed || looksLikeFeed(loaded.text, args.target, loaded.contentType))
+        return checkFeed(args, loaded);
+    const r = parsePage(loaded.text, loaded.url);
+    console.log(args.json ? JSON.stringify(r, null, 2) : describe(loaded.label, r));
     return r.brief ? 0 : 1;
 }
-function main(argv) {
+async function main(argv) {
     const args = parseArgs(argv);
     if (args.command === "check")
         return check(args);
+    if (args.command === "version") {
+        console.log(VERSION);
+        return 0;
+    }
     console.error(USAGE);
     return args.command === "help" ? 0 : 2;
 }
-process.exitCode = main(process.argv.slice(2));
+main(process.argv.slice(2)).then((code) => {
+    process.exitCode = code;
+}, (err) => {
+    console.error(err instanceof Error ? err.message : String(err));
+    process.exitCode = 2;
+});

package/dist/fetch.d.ts

@@ -0,0 +1,33 @@
+/**
+ * The network code for `openbriefing check`. Kept apart from obs.ts so the
+ * parser stays usable in a browser, with no platform API.
+ *
+ * Everything fetched is untrusted. The limits here bound time, size and
+ * redirects; callers escape control characters before printing.
+ */
+export interface FetchLimits {
+    /** Whole-request budget in milliseconds, redirects included. */
+    timeoutMs: number;
+    /** The body is cut off, and the fetch fails, past this many bytes. */
+    maxBytes: number;
+    maxRedirects: number;
+    userAgent: string;
+}
+export declare const DEFAULT_LIMITS: Omit<FetchLimits, "userAgent">;
+export type FetchResult = {
+    ok: true;
+    url: string;
+    contentType: string;
+    text: string;
+} | {
+    ok: false;
+    url: string;
+    error: string;
+};
+export declare function userAgent(version: string): string;
+/** True for an absolute http or https URL. */
+export declare function isHttpUrl(input: string): boolean;
+/** Fetches a page's text within the limits. Never throws. */
+export declare function fetchText(input: string, limits: FetchLimits): Promise<FetchResult>;
+/** Runs `fn` over `items`, at most `width` at a time, keeping the order of results. */
+export declare function mapLimit<T, R>(items: readonly T[], width: number, fn: (item: T, index: number) => Promise<R>): Promise<R[]>;

package/dist/fetch.js

@@ -0,0 +1,122 @@
+/**
+ * The network code for `openbriefing check`. Kept apart from obs.ts so the
+ * parser stays usable in a browser, with no platform API.
+ *
+ * Everything fetched is untrusted. The limits here bound time, size and
+ * redirects; callers escape control characters before printing.
+ */
+export const DEFAULT_LIMITS = {
+    timeoutMs: 10_000,
+    maxBytes: 5 * 1024 * 1024,
+    maxRedirects: 5,
+};
+export function userAgent(version) {
+    return `openbriefing-check/${version} (+https://openbriefingstandard.org)`;
+}
+/** True for an absolute http or https URL. */
+export function isHttpUrl(input) {
+    try {
+        const u = new URL(input);
+        return u.protocol === "http:" || u.protocol === "https:";
+    }
+    catch {
+        return false;
+    }
+}
+/** Fetches a page's text within the limits. Never throws. */
+export async function fetchText(input, limits) {
+    if (!isHttpUrl(input))
+        return { ok: false, url: input, error: "only http and https addresses are fetched" };
+    const controller = new AbortController();
+    const timer = setTimeout(() => controller.abort(), limits.timeoutMs);
+    let url = input;
+    try {
+        for (let hop = 0;; hop++) {
+            const res = await fetch(url, {
+                redirect: "manual",
+                signal: controller.signal,
+                headers: { "user-agent": limits.userAgent, accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.5" },
+            });
+            if (res.status >= 300 && res.status < 400) {
+                const location = res.headers.get("location");
+                await res.body?.cancel();
+                if (!location)
+                    return { ok: false, url, error: `HTTP ${res.status} with no Location` };
+                if (hop >= limits.maxRedirects)
+                    return { ok: false, url, error: `more than ${limits.maxRedirects} redirects` };
+                const next = new URL(location, url).href;
+                if (!isHttpUrl(next))
+                    return { ok: false, url, error: "redirected to an address that is not http or https" };
+                url = next;
+                continue;
+            }
+            if (!res.ok) {
+                await res.body?.cancel();
+                return { ok: false, url, error: `HTTP ${res.status}` };
+            }
+            const declared = Number(res.headers.get("content-length") ?? "");
+            if (Number.isFinite(declared) && declared > limits.maxBytes) {
+                await res.body?.cancel();
+                return { ok: false, url, error: `larger than ${limits.maxBytes} bytes` };
+            }
+            const contentType = res.headers.get("content-type") ?? "";
+            const text = await readCapped(res, limits.maxBytes);
+            if (text === undefined)
+                return { ok: false, url, error: `larger than ${limits.maxBytes} bytes` };
+            return { ok: true, url, contentType, text };
+        }
+    }
+    catch (err) {
+        const aborted = controller.signal.aborted;
+        return { ok: false, url, error: aborted ? `no answer within ${limits.timeoutMs / 1000} s` : err instanceof Error ? cause(err) : String(err) };
+    }
+    finally {
+        clearTimeout(timer);
+    }
+}
+function cause(err) {
+    const c = err.cause;
+    if (c instanceof Error && c.message)
+        return c.message;
+    return err.message;
+}
+async function readCapped(res, maxBytes) {
+    if (!res.body)
+        return "";
+    const reader = res.body.getReader();
+    const chunks = [];
+    let total = 0;
+    for (;;) {
+        const { done, value } = await reader.read();
+        if (done)
+            break;
+        total += value.byteLength;
+        if (total > maxBytes) {
+            await reader.cancel();
+            return undefined;
+        }
+        chunks.push(value);
+    }
+    const all = new Uint8Array(total);
+    let at = 0;
+    for (const c of chunks) {
+        all.set(c, at);
+        at += c.byteLength;
+    }
+    return new TextDecoder("utf-8").decode(all);
+}
+/** Runs `fn` over `items`, at most `width` at a time, keeping the order of results. */
+export async function mapLimit(items, width, fn) {
+    const out = new Array(items.length);
+    let next = 0;
+    const worker = async () => {
+        for (;;) {
+            const i = next++;
+            if (i >= items.length)
+                return;
+            out[i] = await fn(items[i], i);
+        }
+    };
+    await Promise.all(Array.from({ length: Math.min(width, items.length) }, worker));
+    return out;
+}

package/dist/obs.d.ts

@@ -11,6 +11,8 @@ export declare const OBS_NS = "https://openbriefingstandard.org/ns#";
 export type ObsTier = "short" | "medium" | "long";
 export type DeriveMode = "verbatim" | "anchor" | "open";
 export type Preference = "summary" | "full";
+/** Where a brief's modified time came from (§13). */
+export type ModifiedFrom = "obs" | "article" | "feed" | "none";
 export interface ObsSummary {
     tier: ObsTier;
     text: string;
@@ -18,10 +20,21 @@ export interface ObsSummary {
     /** "tiered" for obs:summary:{tier}; "inferred" for a bare obs:summary (§4). */
     from: "tiered" | "inferred";
 }
+export interface ObsHighlight {
+    /** The passage, as written. */
+    text: string;
+    words: number;
+    /** obs:highlight:label, 6 words or fewer, when given. */
+    label?: string;
+    /** False when the page's text does not contain the highlight (§7). Absent for feed items and empty pages. */
+    inContent?: boolean;
+}
 export interface ObsBrief {
     version: number;
-    /** obs:modified, as given (ISO 8601). */
-    modified: string;
+    /** The effective modified time, ISO 8601 (§13). Absent when no source gave one. */
+    modified?: string;
+    /** Where `modified` came from: obs:modified, article:modified_time, the feed item's date, or nowhere. */
+    modifiedFrom: ModifiedFrom;
     /** obs:url, normalized, when given. */
     url?: string;
     /** False when obs:url names content other than this page (§11). */
@@ -32,6 +45,8 @@ export interface ObsBrief {
     summaries: ObsSummary[];
     /** Priority order, at most 3. keep[0] is the core message (§6). */
     keep: string[];
+    /** Priority order, at most 5 (§7). Empty when none. */
+    highlights: ObsHighlight[];
     derive: {
         mode: DeriveMode;
         translate: boolean;
@@ -56,6 +71,9 @@ export declare const SIZES: {
     readonly keep: 40;
     readonly attribution: 12;
     readonly keepMax: 3;
+    readonly highlight: 60;
+    readonly highlightLabel: 6;
+    readonly highlightMax: 5;
     readonly tiers: Record<ObsTier, {
         target: [number, number];
     }>;
@@ -68,8 +86,13 @@ export declare const SIZES: {
 export declare function tierFor(words: number): ObsTier;
 /** Words are runs of characters between white space (§4). */
 export declare function countWords(text: string): number;
-/** URL comparison of §11: scheme and host case, trailing slash, tracking parameters. */
+/** URL comparison of §12: scheme and host case, trailing slash, tracking parameters. */
 export declare function normalizeUrl(input: string, base?: string): string | undefined;
+/**
+ * The loose comparison of §7: compatibility forms, straight quotes for curly,
+ * one kind of dash, one kind of space, one case. Words themselves must match.
+ */
+export declare function looseText(s: string): string;
 /**
  * Reads a page's obs: meta tags. `pageUrl` is where the page was fetched
  * from; the page's <link rel="canonical"> takes precedence for §11.
@@ -82,5 +105,19 @@ export declare function parseObs(doc: Document, pageUrl: string): ParseResult;
 export declare function parseFeedObs(item: Element, itemUrl?: string): ParseResult;
 /** Checks an object in the ObsBrief shape and rebuilds it under the same rules. */
 export declare function validateObs(input: unknown): ParseResult;
-/** The later obs:modified stands; on a tie, the page stands. */
+/**
+ * The later effective modified time stands. When the two are equal, or either
+ * side has none, the page stands.
+ */
 export declare function mergeObs(page?: ObsBrief, feed?: ObsBrief): ObsBrief | undefined;
+/**
+ * A stable hash of what a brief says: every text value and every setting, in a
+ * fixed order. Two briefs with the same fingerprint present the same. It leaves
+ * out the modified time, and everything that depends on where the tags were
+ * read rather than on what they say (the language fallback, describesThisPage,
+ * a highlight's inContent), so a page and its feed item agree. Readers compare
+ * fingerprints to tell a change, above all when there is no modified time.
+ *
+ * SHA-256 of the canonical JSON below, in hex, first 16 characters.
+ */
+export declare function fingerprint(brief: ObsBrief): string;