@citation-media/html-parser 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Citation Media GmbH
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,103 @@
+# @citation-media/html-parser
+
+Extracts links, images, resources, metadata, and Markdown from HTML in Cloudflare Workers, in one streaming pass over the page. It builds on the runtime's [HTMLRewriter](https://developers.cloudflare.com/workers/runtime-apis/html-rewriter/), so it never holds a DOM: a page of several megabytes costs about as much memory as a small one.
+
+## Install
+
+```bash
+npm install @citation-media/html-parser
+```
+
+The package runs wherever `HTMLRewriter` is a global: Cloudflare Workers, `wrangler dev`, and `@cloudflare/vitest-pool-workers`. It has no dependencies.
+
+## Parse A Page
+
+`parseHtml` returns only what you ask for. The input is the page as text or a `Response`, such as the result of `fetch`.
+
+```ts
+import { parseHtml } from "@citation-media/html-parser";
+
+const response = await fetch("https://example.com/");
+const { links, images, meta, markdown } = await parseHtml(response, {
+  url: response.url,
+  links: { kinds: ["internal"], resources: false },
+  images: true,
+  meta: true,
+  markdown: true,
+});
+```
+
+| Option | What it collects |
+| --- | --- |
+| `url` | Required. Resolves relative URLs and tells internal links from external ones. |
+| `links` | `true` or a filter: `kinds` (`internal`, `external`, `special`, `anchor`) and `resources: false` to drop downloads such as PDFs. |
+| `images` | Images with `alt` (`null` when missing, `""` when decorative), `width`, `height`, `loading`, and whether they have a `srcset`. |
+| `resources` | Scripts with their `src`, `type`, and attribute names, frame URLs, and `<link>` elements with their `rel`. |
+| `meta` | Title, description, language, canonical URL, robots, viewport, generator, and Open Graph image. |
+| `headings` | The first 200 headings with their level and text, and `headingCounts` per level without a limit. |
+| `classPrefixes` | Counts of elements whose class starts with a prefix, such as `["elementor-", "brxe-", "wp-block-"]`. |
+| `markdown` | `true` or options: `links` (default true), `images` (default false), `maxLength` (default 100 000). |
+| `context` | A CSS selector such as `main` or `article`: only content inside matching elements counts. When nothing matches, the whole page counts, if the input is text. |
+| `skipAriaHidden` | Leaves `aria-hidden` content out of the Markdown. Default true; set it to false for DOMs taken from a browser while a consent dialog is open, which hides the page that way. |
+
+Links are deduplicated by URL, counted, and sorted by count. Each carries its `text`, `kind`, `resource`, `count`, and, where present, `rel`, `target`, and `aria-label`. URLs are absolute; paths keep their case.
+
+The Markdown leaves out navigation, footer, forms, scripts, consent dialogs, visually hidden helpers, and binary or EXIF text, so it holds what a reader reads.
+
+## Clean A Page
+
+`cleanHtml` returns HTML without what you remove. EXIF and XMP blocks and binary-looking text are always removed.
+
+```ts
+import { cleanHtml } from "@citation-media/html-parser";
+
+const html = await cleanHtml(page, {
+  url: "https://example.com/",
+  remove: {
+    scripts: true,
+    styles: true,
+    comments: true,
+    cookieConsent: true,
+    classes: true,
+  },
+  links: { kinds: ["internal"] },
+  context: "main",
+});
+```
+
+`remove` accepts `scripts`, `styles` (with style attributes and inline SVG), `comments`, `cookieConsent`, `ariaHidden`, `images`, `classes`, `ids`, `meta`, `linkTags`, and `hyphenation`. Links that `links` does not keep become their text.
+
+## Markdown By Workers AI
+
+`markdownWithAi` cleans the page and converts it with Workers AI's document conversion. It handles tables and nested formatting better than the streaming Markdown, at the cost of a service call.
+
+```ts
+import { markdownWithAi } from "@citation-media/html-parser";
+
+const markdown = await markdownWithAi(env.AI, page, { url, images: true });
+```
+
+## Limits
+
+The streaming Markdown follows HTMLRewriter's model: tables become rows without alignment, and bold or italic formatting is dropped. Use `markdownWithAi` where that matters. Elements that are opened but never closed, such as `<span/>`, keep their state to the end of the page.
+
+## Develop
+
+Development needs Node.js 22.18 or later, which loads the TypeScript configuration of Oxlint and Oxfmt; `.nvmrc` pins Node.js 24.
+
+```bash
+npm ci
+npm test
+npm run check
+npm run build
+```
+
+Tests run in workerd through `@cloudflare/vitest-pool-workers`, the runtime the package targets.
+
+## Release
+
+Bump `version` in `package.json` and add a section to `CHANGELOG.md` on `main`, then push a matching tag such as `v0.2.0`. The release workflow publishes to npm with provenance and creates the GitHub release.
+
+## License
+
+MIT

package/dist/ai-markdown.d.ts

@@ -0,0 +1,21 @@
+import type { CleanOptions } from "./clean.ts";
+/**
+ * Markdown by Workers AI's document conversion, the second way besides the streaming Markdown of
+ * `parseHtml`. It handles tables and nested formatting better, at the cost of a service call. The
+ * page is cleaned first: scripts, styles, consent banners, and hidden helpers only cost tokens.
+ */
+/** The part of the Workers AI binding this needs, so callers can pass `env.AI` as it is. */
+export interface MarkdownConverter {
+    toMarkdown: (files: {
+        name: string;
+        blob: Blob;
+    }[]) => Promise<{
+        format: string;
+        data?: string;
+    }[]>;
+}
+export interface AiMarkdownOptions extends Pick<CleanOptions, "url" | "links" | "context"> {
+    /** Keeps images in the Markdown. Default false. */
+    images?: boolean;
+}
+export declare const markdownWithAi: (ai: MarkdownConverter, input: string | Response, { images, ...options }: AiMarkdownOptions) => Promise<string>;

package/dist/ai-markdown.js

@@ -0,0 +1,23 @@
+import { cleanHtml } from "./clean.js";
+export const markdownWithAi = async (ai, input, { images = false, ...options }) => {
+    const html = await cleanHtml(input, {
+        ...options,
+        remove: {
+            ariaHidden: true,
+            classes: true,
+            comments: true,
+            cookieConsent: true,
+            hyphenation: true,
+            ids: true,
+            images: !images,
+            linkTags: true,
+            meta: true,
+            scripts: true,
+            styles: true,
+        },
+    });
+    const [converted] = await ai.toMarkdown([
+        { blob: new Blob([html], { type: "text/html" }), name: "index.html" },
+    ]);
+    return converted?.format === "markdown" ? (converted.data ?? "") : "";
+};

package/dist/clean.d.ts

@@ -0,0 +1,32 @@
+import type { LinkFilter } from "./parse.ts";
+/**
+ * Cleans a page in one streaming pass, for callers that need HTML rather than extracted facts,
+ * such as a Markdown conversion by Workers AI. EXIF and XMP blobs and binary-looking text are
+ * always removed; everything else is removed only when asked.
+ */
+export interface CleanOptions {
+    /** The page's address, needed to filter links by kind. */
+    url: string | URL;
+    remove?: {
+        scripts?: boolean;
+        /** Style elements, style attributes, and inline SVG. */
+        styles?: boolean;
+        comments?: boolean;
+        cookieConsent?: boolean;
+        ariaHidden?: boolean;
+        /** img, picture, and figure elements. */
+        images?: boolean;
+        classes?: boolean;
+        ids?: boolean;
+        meta?: boolean;
+        linkTags?: boolean;
+        /** Soft hyphens and `<wbr>`. */
+        hyphenation?: boolean;
+    };
+    /** Keeps only links that match; others become their text. */
+    links?: LinkFilter;
+    /** Keeps only the content of elements matching this selector. */
+    context?: string;
+}
+/** The cleaned page as HTML. */
+export declare const cleanHtml: (input: string | Response, options: CleanOptions) => Promise<string>;

package/dist/clean.js

@@ -0,0 +1,143 @@
+import { looksBinary } from "./text.js";
+import { absoluteUrl, isResource, linkKind } from "./urls.js";
+const cookieConsent = [
+    "[id^='brlbs-cmpnt']",
+    "[class^='brlbs-cmpnt']",
+    "[consent-skip-blocker]",
+    "#usercentrics-root",
+    "div[id*='consent-modal']",
+    "div[class*='consent-modal']",
+    "div[id*='consent-wrapper']",
+    "div[class*='consent-wrapper']",
+    "div[id*='consent-banner']",
+    "div[class*='consent-banner']",
+];
+const remove = {
+    element(element) {
+        element.remove();
+    },
+};
+/** The cleaned page as HTML. */
+export const cleanHtml = (input, options) => {
+    const base = new URL(options.url);
+    const removed = options.remove ?? {};
+    let rewriter = new HTMLRewriter();
+    if (options.context) {
+        // Everything outside the context is unwrapped or dropped; the context's subtrees stay whole.
+        let inContext = 0;
+        rewriter = rewriter
+            .on(options.context, {
+            element(element) {
+                try {
+                    element.onEndTag(() => {
+                        inContext -= 1;
+                    });
+                    inContext += 1;
+                }
+                catch {
+                    // A void element such as <img> has no content to keep.
+                }
+            },
+        })
+            .on("*", {
+            element(element) {
+                // The context handler above runs first, so a matching element already counts as inside.
+                if (inContext === 0) {
+                    element.removeAndKeepContent();
+                }
+            },
+        })
+            .onDocument({
+            text(chunk) {
+                if (inContext === 0) {
+                    chunk.remove();
+                }
+            },
+        });
+    }
+    rewriter = rewriter
+        .on("*", {
+        element(element) {
+            if (element.tagName.includes(":")) {
+                element.remove();
+                return;
+            }
+            for (const [name, value] of element.attributes) {
+                if (name && value && looksBinary(value)) {
+                    element.removeAttribute(name);
+                }
+            }
+            if (removed.classes) {
+                element.removeAttribute("class");
+            }
+            if (removed.ids) {
+                element.removeAttribute("id");
+            }
+            if (removed.styles) {
+                element.removeAttribute("style");
+            }
+        },
+    })
+        .onDocument({
+        comments(comment) {
+            if (removed.comments ||
+                /xpacket|xmpmeta|rdf:RDF|Adobe XMP Core|Exif|8BIM/iu.test(comment.text)) {
+                comment.remove();
+            }
+        },
+        text(chunk) {
+            if (looksBinary(chunk.text)) {
+                chunk.remove();
+            }
+            else if (removed.hyphenation && /&shy;|­/u.test(chunk.text)) {
+                chunk.replace(chunk.text.replaceAll(/&shy;|­/gu, ""), { html: true });
+            }
+        },
+    });
+    if (removed.scripts) {
+        rewriter = rewriter.on("script", remove);
+    }
+    if (removed.styles) {
+        rewriter = rewriter.on("style, svg", remove);
+    }
+    if (removed.cookieConsent) {
+        rewriter = rewriter.on(cookieConsent.join(", "), remove);
+    }
+    if (removed.ariaHidden) {
+        rewriter = rewriter.on("[aria-hidden='true'], .screen-reader-text, .sr-only, .visually-hidden, .skip-link", remove);
+    }
+    if (removed.images) {
+        rewriter = rewriter.on("img, picture, figure", remove);
+    }
+    if (removed.meta) {
+        rewriter = rewriter.on("meta", remove);
+    }
+    if (removed.linkTags) {
+        rewriter = rewriter.on("link", remove);
+    }
+    if (removed.hyphenation) {
+        rewriter = rewriter.on("wbr", remove);
+    }
+    const { links } = options;
+    if (links) {
+        rewriter = rewriter.on("a", {
+            element(element) {
+                const href = element.getAttribute("href");
+                if (!href) {
+                    return;
+                }
+                const url = absoluteUrl(href, base);
+                const kind = linkKind(href, url, base);
+                const kept = (!links.kinds || links.kinds.includes(kind)) &&
+                    (links.resources !== false || !isResource(url));
+                if (!kept) {
+                    element.removeAndKeepContent();
+                }
+            },
+        });
+    }
+    const response = input instanceof Response
+        ? input
+        : new Response(input, { headers: { "content-type": "text/html" } });
+    return rewriter.transform(response).text();
+};

package/dist/index.d.ts

@@ -0,0 +1,8 @@
+export { markdownWithAi } from "./ai-markdown.ts";
+export type { AiMarkdownOptions, MarkdownConverter } from "./ai-markdown.ts";
+export { cleanHtml } from "./clean.ts";
+export type { CleanOptions } from "./clean.ts";
+export { parseHtml } from "./parse.ts";
+export type { Heading, Image, Link, LinkFilter, MarkdownOptions, Meta, ParseOptions, ParseResult, Resources, Script, } from "./parse.ts";
+export { absoluteUrl, isResource, linkKind, siteOf, specialProtocol, } from "./urls.ts";
+export type { LinkKind } from "./urls.ts";

package/dist/index.js

@@ -0,0 +1,4 @@
+export { markdownWithAi } from "./ai-markdown.js";
+export { cleanHtml } from "./clean.js";
+export { parseHtml } from "./parse.js";
+export { absoluteUrl, isResource, linkKind, siteOf, specialProtocol, } from "./urls.js";

package/dist/markdown.d.ts

@@ -0,0 +1,20 @@
+/**
+ * Builds Markdown from parser events while the page streams through: blocks for headings,
+ * paragraphs, list items, quotes, and table rows, inline links and images, and preformatted text
+ * that keeps its whitespace. It follows HTMLRewriter's limits: no aligned tables, and formatting such
+ * as bold or italic is dropped, because the content, not its styling, is what the output is for.
+ */
+export declare class MarkdownWriter {
+    #private;
+    constructor(maxLength: number);
+    get full(): boolean;
+    text(chunk: string): void;
+    /** Starts a block such as `h2`, `p`, `li`, `blockquote`, `tr`, or `pre`. */
+    open(tag: string): void;
+    /** Ends a block; blocks without an end tag end when the next one opens. */
+    close(tag: string): void;
+    openLink(href: string): void;
+    closeLink(): void;
+    image(alt: string, src: string): void;
+    toString(): string;
+}

package/dist/markdown.js

@@ -0,0 +1,147 @@
+import { collapseWhitespace, decodeEntities, looksBinary, removeHyphenation, } from "./text.js";
+/**
+ * Builds Markdown from parser events while the page streams through: blocks for headings,
+ * paragraphs, list items, quotes, and table rows, inline links and images, and preformatted text
+ * that keeps its whitespace. It follows HTMLRewriter's limits: no aligned tables, and formatting such
+ * as bold or italic is dropped, because the content, not its styling, is what the output is for.
+ */
+export class MarkdownWriter {
+    #parts = [];
+    #text = [];
+    #length = 0;
+    #maxLength;
+    #links = [];
+    #lists = [];
+    #preformatted = 0;
+    constructor(maxLength) {
+        this.#maxLength = maxLength;
+    }
+    get full() {
+        return this.#length >= this.#maxLength;
+    }
+    text(chunk) {
+        if (!this.full) {
+            this.#text.push(chunk);
+        }
+    }
+    /** Starts a block such as `h2`, `p`, `li`, `blockquote`, `tr`, or `pre`. */
+    open(tag) {
+        this.#flush();
+        const level = /^h(?<level>[1-6])$/u.exec(tag)?.groups?.level;
+        if (level) {
+            this.#push(`\n\n${"#".repeat(Number(level))} `);
+        }
+        else if (tag === "li") {
+            const list = this.#lists.at(-1);
+            if (list?.ordered) {
+                list.count += 1;
+            }
+            const indent = "  ".repeat(Math.max(0, this.#lists.length - 1));
+            this.#push(`\n${indent}${list?.ordered ? `${list.count}.` : "-"} `);
+        }
+        else if (tag === "ul" || tag === "ol") {
+            this.#lists.push({ count: 0, ordered: tag === "ol" });
+            this.#push("\n");
+        }
+        else if (tag === "blockquote") {
+            this.#push("\n\n> ");
+        }
+        else if (tag === "tr") {
+            this.#push("\n|");
+        }
+        else if (tag === "td" || tag === "th") {
+            this.#push(" ");
+        }
+        else if (tag === "pre") {
+            this.#preformatted += 1;
+            this.#push("\n\n```\n");
+        }
+        else if (tag === "br") {
+            this.#push("\n");
+        }
+        else if (tag === "hr") {
+            this.#push("\n\n---\n\n");
+        }
+        else {
+            this.#push("\n\n");
+        }
+    }
+    /** Ends a block; blocks without an end tag end when the next one opens. */
+    close(tag) {
+        this.#flush();
+        if (tag === "ul" || tag === "ol") {
+            this.#lists.pop();
+            this.#push("\n");
+        }
+        else if (tag === "td" || tag === "th") {
+            this.#push(" |");
+        }
+        else if (tag === "pre") {
+            this.#preformatted = Math.max(0, this.#preformatted - 1);
+            this.#push("\n```\n\n");
+        }
+        else if (tag !== "li" && tag !== "tr") {
+            // List items and table rows end where the next one starts, so lists stay tight.
+            this.#push("\n\n");
+        }
+    }
+    openLink(href) {
+        this.#flush();
+        this.#links.push({ href, index: this.#parts.length });
+        this.#push("[");
+    }
+    closeLink() {
+        this.#flush();
+        const link = this.#links.pop();
+        if (!link) {
+            return;
+        }
+        // A link without text, such as an icon, keeps nothing: an empty [](url) helps no reader.
+        const label = this.#parts
+            .slice(link.index + 1)
+            .join("")
+            .trim();
+        if (!label) {
+            this.#parts.splice(link.index);
+            return;
+        }
+        this.#push(`](${link.href})`);
+    }
+    image(alt, src) {
+        this.#flush();
+        this.#push(`![${collapseWhitespace(alt).trim()}](${src})`);
+    }
+    toString() {
+        this.#flush();
+        // Code blocks keep their whitespace; everything between them is tidied.
+        return this.#parts
+            .join("")
+            .split(/(?<code>```[\s\S]*?```)/u)
+            .map((segment) => segment.startsWith("```")
+            ? segment
+            : segment
+                .replaceAll(/[ \t]+\n/gu, "\n")
+                .replaceAll(/\n[ \t]+(?=[^-\d> ])/gu, "\n"))
+            .join("")
+            .replaceAll(/\n{3,}/gu, "\n\n")
+            .trim()
+            .slice(0, this.#maxLength);
+    }
+    #flush() {
+        if (this.#text.length === 0) {
+            return;
+        }
+        const raw = removeHyphenation(decodeEntities(this.#text.join("")));
+        this.#text = [];
+        if (looksBinary(raw)) {
+            return;
+        }
+        this.#push(this.#preformatted > 0 ? raw : collapseWhitespace(raw));
+    }
+    #push(part) {
+        if (!this.full) {
+            this.#parts.push(part);
+            this.#length += part.length;
+        }
+    }
+}

package/dist/parse.d.ts

@@ -0,0 +1,108 @@
+import type { LinkKind } from "./urls.ts";
+/**
+ * One streaming pass over a page with Cloudflare's HTMLRewriter, which never builds a DOM: it
+ * collects links, images, resources, metadata, headings, and class counts, and writes Markdown at
+ * the same time. Memory stays flat even for pages of several megabytes.
+ */
+export interface Link {
+    url: string;
+    /** Visible text of the first link to this URL. */
+    text: string;
+    kind: LinkKind;
+    /** Whether the URL points to a file download such as a PDF. */
+    resource: boolean;
+    /** How often the page links to this URL. */
+    count: number;
+    rel?: string;
+    target?: string;
+    /** `aria-label` of the first link, which names links whose text does not. */
+    label?: string;
+}
+export interface Image {
+    url: string;
+    /** `null` when the attribute is missing, `""` when the image is marked as decorative. */
+    alt: string | null;
+    width?: string;
+    height?: string;
+    loading?: string;
+    srcset: boolean;
+}
+export interface Script {
+    src?: string;
+    type?: string;
+    /** Names of the element's attributes, for consent markers such as `data-cookieconsent`. */
+    attributes: string[];
+}
+export interface Resources {
+    scripts: Script[];
+    frames: string[];
+    /** `<link>` elements with their `rel`, such as stylesheets and preconnects. */
+    links: {
+        href: string;
+        rel: string;
+    }[];
+}
+export interface Meta {
+    title?: string;
+    description?: string;
+    language?: string;
+    canonical?: string;
+    robots?: string;
+    viewport?: string;
+    generator?: string;
+    image?: string;
+}
+export interface Heading {
+    level: number;
+    text: string;
+}
+export interface LinkFilter {
+    /** Kinds to keep; all when not set. */
+    kinds?: LinkKind[];
+    /** Whether to keep links to downloads such as PDFs. Default true. */
+    resources?: boolean;
+}
+export interface MarkdownOptions {
+    /** Writes `[text](url)` for links. Default true. */
+    links?: boolean;
+    /** Writes `![alt](url)` for images. Default false. */
+    images?: boolean;
+    /** Longest output in characters. Default 100 000. */
+    maxLength?: number;
+}
+export interface ParseOptions {
+    /** The page's address, to resolve relative URLs and tell internal links from external ones. */
+    url: string | URL;
+    links?: boolean | LinkFilter;
+    images?: boolean;
+    resources?: boolean;
+    meta?: boolean;
+    headings?: boolean;
+    /** Counts elements whose class starts with a prefix, such as page builders' `elementor-`. */
+    classPrefixes?: string[];
+    markdown?: boolean | MarkdownOptions;
+    /**
+     * Collects only inside elements matching this selector, such as `main` or `article`. When
+     * nothing matches, the whole page counts; for a `Response` input, nothing is collected then.
+     */
+    context?: string;
+    /**
+     * Leaves `aria-hidden` content out of the Markdown. Default true. Set it to false for DOMs taken
+     * from a browser while a dialog is open: consent tools hide the whole page behind it that way.
+     */
+    skipAriaHidden?: boolean;
+}
+export interface ParseResult {
+    links?: Link[];
+    images?: Image[];
+    resources?: Resources;
+    meta?: Meta;
+    /** The first headings with their text, see `maxHeadings`. */
+    headings?: Heading[];
+    /** How many headings of each level the page has, uncapped: `headingCounts[0]` counts `h1`. */
+    headingCounts?: number[];
+    classCounts?: Record<string, number>;
+    markdown?: string;
+}
+/** Parses a page, as text or as a `Response` such as a `fetch` result, for what `options` ask. */
+export declare const parseHtml: (input: string | Response, options: ParseOptions) => Promise<ParseResult>;

package/dist/parse.js

@@ -0,0 +1,380 @@
+import { MarkdownWriter } from "./markdown.js";
+import { collapseWhitespace, decodeEntities, removeHyphenation, } from "./text.js";
+import { absoluteUrl, isResource, linkKind } from "./urls.js";
+const blockTags = new Set([
+    "br",
+    "hr",
+    "h1",
+    "h2",
+    "h3",
+    "h4",
+    "h5",
+    "h6",
+    "p",
+    "li",
+    "ul",
+    "ol",
+    "blockquote",
+    "pre",
+    "tr",
+    "td",
+    "th",
+    "dt",
+    "dd",
+    "figcaption",
+    "section",
+    "article",
+    "div",
+]);
+// Content that is not what a reader reads: page chrome, code, and consent dialogs.
+const hiddenFromMarkdown = [
+    "script",
+    "style",
+    "noscript",
+    "template",
+    "svg",
+    "iframe",
+    "nav",
+    "footer",
+    "form",
+    "select",
+    "[hidden]",
+    "[role='dialog']",
+    "[aria-modal='true']",
+    "[id*='cookie']",
+    "[class*='cookie']",
+    "[id*='consent']",
+    "[class*='consent']",
+    "[id^='brlbs-cmpnt']",
+    "[class^='brlbs-cmpnt']",
+    "#BorlabsCookieBox",
+    "#usercentrics-root",
+    "#CybotCookiebotDialog",
+    "#cc-main",
+    ".screen-reader-text",
+    ".sr-only",
+    ".visually-hidden",
+];
+const maxHeadings = 200;
+const maxImages = 500;
+const maxResources = 500;
+// Follows an element to its end tag. Void elements such as <img> have none, and HTMLRewriter throws
+// for them; elements closed implicitly, such as <li> without </li>, end with their parent.
+const enter = (element, onEnd) => {
+    try {
+        element.onEndTag(onEnd);
+        return true;
+    }
+    catch {
+        return false;
+    }
+};
+const clean = (text) => collapseWhitespace(removeHyphenation(decodeEntities(text))).trim();
+const addLink = (links, found, base) => {
+    // Cloudflare's e-mail obfuscation rewrites addresses to a script-decoded link.
+    if (found.url.includes("/cdn-cgi/l/email-protection")) {
+        return;
+    }
+    const existing = links.get(found.url);
+    if (existing) {
+        existing.count += 1;
+        return;
+    }
+    const { href, ...link } = found;
+    links.set(found.url, {
+        ...link,
+        count: 1,
+        kind: linkKind(href, found.url, base),
+        resource: isResource(found.url),
+    });
+};
+const filterLinks = (links, filter) => links
+    .filter((link) => (!filter.kinds || filter.kinds.includes(link.kind)) &&
+    (filter.resources !== false || !link.resource))
+    .toSorted((first, second) => second.count - first.count);
+const collecting = (pass) => pass.inContext > 0;
+const writing = (pass) => collecting(pass) && pass.hidden === 0 && pass.markdown !== undefined;
+const hide = (pass) => ({
+    element(element) {
+        if (enter(element, () => (pass.hidden -= 1))) {
+            pass.hidden += 1;
+        }
+    },
+});
+const onStructure = (rewriter, pass) => {
+    const { context, classPrefixes = [] } = pass.options;
+    let next = rewriter;
+    if (context) {
+        next = next.on(context, {
+            element(element) {
+                pass.contextFound = true;
+                if (enter(element, () => (pass.inContext -= 1))) {
+                    pass.inContext += 1;
+                }
+            },
+        });
+    }
+    next = next.on(hiddenFromMarkdown.join(", "), hide(pass));
+    if (pass.options.skipAriaHidden ?? true) {
+        next = next.on("[aria-hidden='true']", hide(pass));
+    }
+    return next.on("*", {
+        element(element) {
+            // Namespaced tags such as <x:xmpmeta> carry image metadata, not content.
+            if (element.tagName.includes(":")) {
+                hide(pass).element(element);
+            }
+            if (!collecting(pass)) {
+                return;
+            }
+            const classes = element.getAttribute("class")?.split(/\s+/u) ?? [];
+            for (const prefix of classPrefixes) {
+                if (classes.some((name) => name.startsWith(prefix))) {
+                    pass.classCounts[prefix] = (pass.classCounts[prefix] ?? 0) + 1;
+                }
+            }
+            const tag = element.tagName.toLowerCase();
+            if (!writing(pass) || !blockTags.has(tag)) {
+                return;
+            }
+            pass.markdown?.open(tag);
+            enter(element, () => {
+                if (writing(pass)) {
+                    pass.markdown?.close(tag);
+                }
+            });
+        },
+    });
+};
+const onLinks = (rewriter, pass) => rewriter.on("a[href]", {
+    element(element) {
+        const href = element.getAttribute("href") ?? "";
+        if (!collecting(pass) || !href) {
+            return;
+        }
+        const url = absoluteUrl(href, pass.base);
+        const text = [];
+        pass.link = text;
+        // Attributes are readable only while this handler runs, not in the end-tag handler.
+        const label = element.getAttribute("aria-label") ?? undefined;
+        const rel = element.getAttribute("rel") ?? undefined;
+        const target = element.getAttribute("target") ?? undefined;
+        const written = writing(pass) &&
+            pass.markdownOptions?.links !== false &&
+            !href.startsWith("#");
+        if (written) {
+            pass.markdown?.openLink(url);
+        }
+        enter(element, () => {
+            if (written) {
+                pass.markdown?.closeLink();
+            }
+            if (pass.linkFilter) {
+                addLink(pass.links, { href, label, rel, target, text: clean(text.join("")), url }, pass.base);
+            }
+            pass.link = undefined;
+        });
+    },
+});
+const onImages = (rewriter, pass) => rewriter.on("img", {
+    element(element) {
+        // Lazy-loading scripts often keep the real address in data-src until the image scrolls in.
+        const src = element.getAttribute("src") ?? element.getAttribute("data-src");
+        if (!collecting(pass) || !src || src.startsWith("data:")) {
+            return;
+        }
+        const url = absoluteUrl(src, pass.base);
+        const alt = element.getAttribute("alt");
+        if (pass.options.images &&
+            pass.images.size < maxImages &&
+            !pass.images.has(url)) {
+            pass.images.set(url, {
+                alt: alt === null ? null : clean(alt),
+                height: element.getAttribute("height") ?? undefined,
+                loading: element.getAttribute("loading") ?? undefined,
+                srcset: element.hasAttribute("srcset"),
+                url,
+                width: element.getAttribute("width") ?? undefined,
+            });
+        }
+        if (writing(pass) && pass.markdownOptions?.images) {
+            pass.markdown?.image(alt ?? "", url);
+        }
+    },
+});
+const onResources = (rewriter, pass) => {
+    const { resources, base } = pass;
+    return rewriter
+        .on("script", {
+        element(element) {
+            if (resources.scripts.length < maxResources) {
+                const src = element.getAttribute("src");
+                resources.scripts.push({
+                    attributes: [...element.attributes].flatMap(([name]) => name ? [name] : []),
+                    src: src ? absoluteUrl(src, base) : undefined,
+                    type: element.getAttribute("type") ?? undefined,
+                });
+            }
+        },
+    })
+        .on("iframe[src]", {
+        element(element) {
+            if (resources.frames.length < maxResources) {
+                resources.frames.push(absoluteUrl(element.getAttribute("src") ?? "", base));
+            }
+        },
+    })
+        .on("link[href]", {
+        element(element) {
+            if (resources.links.length < maxResources) {
+                resources.links.push({
+                    href: absoluteUrl(element.getAttribute("href") ?? "", base),
+                    rel: element.getAttribute("rel") ?? "",
+                });
+            }
+        },
+    });
+};
+const metaFields = new Map([
+    ["description", "description"],
+    ["generator", "generator"],
+    ["og:image", "image"],
+    ["robots", "robots"],
+    ["viewport", "viewport"],
+]);
+const onMeta = (rewriter, pass) => rewriter
+    .on("html", {
+    element(element) {
+        pass.meta.language = element.getAttribute("lang") ?? undefined;
+    },
+})
+    .on("title", {
+    element(element) {
+        pass.title = [];
+        enter(element, () => {
+            pass.meta.title = clean((pass.title ?? []).join(""));
+            pass.title = undefined;
+        });
+    },
+})
+    .on("meta", {
+    element(element) {
+        const name = element.getAttribute("name") ??
+            element.getAttribute("property") ??
+            "";
+        const field = metaFields.get(name.toLowerCase());
+        if (field) {
+            pass.meta[field] = element.getAttribute("content") ?? undefined;
+        }
+    },
+})
+    .on("link[rel='canonical']", {
+    element(element) {
+        const href = element.getAttribute("href");
+        pass.meta.canonical = href ? absoluteUrl(href, pass.base) : undefined;
+    },
+});
+const onHeadings = (rewriter, pass) => rewriter.on("h1, h2, h3, h4, h5, h6", {
+    element(element) {
+        if (!collecting(pass)) {
+            return;
+        }
+        const level = Number(element.tagName.slice(1));
+        pass.headingCounts[level - 1] = (pass.headingCounts[level - 1] ?? 0) + 1;
+        pass.heading = { level, text: [] };
+        enter(element, () => {
+            const text = clean(pass.heading?.text.join("") ?? "");
+            if (pass.heading && text && pass.headings.length < maxHeadings) {
+                pass.headings.push({ level: pass.heading.level, text });
+            }
+            pass.heading = undefined;
+        });
+    },
+});
+const onText = (rewriter, pass) => rewriter.onDocument({
+    text(chunk) {
+        pass.title?.push(chunk.text);
+        if (!collecting(pass)) {
+            return;
+        }
+        pass.link?.push(chunk.text);
+        pass.heading?.text.push(chunk.text);
+        if (writing(pass)) {
+            pass.markdown?.text(chunk.text);
+        }
+    },
+});
+const resultOf = (pass) => {
+    const { options } = pass;
+    const result = {};
+    if (pass.linkFilter) {
+        result.links = filterLinks([...pass.links.values()], pass.linkFilter);
+    }
+    if (options.images) {
+        result.images = [...pass.images.values()];
+    }
+    if (options.resources) {
+        result.resources = pass.resources;
+    }
+    if (options.meta) {
+        result.meta = pass.meta;
+    }
+    if (options.headings) {
+        result.headings = pass.headings;
+        result.headingCounts = pass.headingCounts;
+    }
+    if (options.classPrefixes) {
+        result.classCounts = pass.classCounts;
+    }
+    if (pass.markdown) {
+        result.markdown = pass.markdown.toString();
+    }
+    return result;
+};
+/** Parses a page, as text or as a `Response` such as a `fetch` result, for what `options` ask. */
+export const parseHtml = async (input, options) => {
+    const markdownOptions = options.markdown === true ? {} : options.markdown || undefined;
+    const pass = {
+        base: new URL(options.url),
+        classCounts: Object.fromEntries((options.classPrefixes ?? []).map((prefix) => [prefix, 0])),
+        contextFound: false,
+        heading: undefined,
+        headingCounts: [0, 0, 0, 0, 0, 0],
+        headings: [],
+        hidden: 0,
+        images: new Map(),
+        inContext: options.context ? 0 : 1,
+        link: undefined,
+        linkFilter: options.links === true ? {} : options.links || undefined,
+        links: new Map(),
+        markdown: markdownOptions
+            ? new MarkdownWriter(markdownOptions.maxLength ?? 100_000)
+            : undefined,
+        markdownOptions,
+        meta: {},
+        options,
+        resources: { frames: [], links: [], scripts: [] },
+        title: undefined,
+    };
+    let rewriter = onImages(onLinks(onStructure(new HTMLRewriter(), pass), pass), pass);
+    if (options.resources) {
+        rewriter = onResources(rewriter, pass);
+    }
+    if (options.meta) {
+        rewriter = onMeta(rewriter, pass);
+    }
+    if (options.headings) {
+        rewriter = onHeadings(rewriter, pass);
+    }
+    const response = input instanceof Response
+        ? input
+        : new Response(input, { headers: { "content-type": "text/html" } });
+    // The rewritten page is discarded; the handlers are what this pass is for.
+    await onText(rewriter, pass).transform(response).arrayBuffer();
+    // Like a selector that matches nothing in a DOM library, a missing context means the whole page.
+    // A response can be read only once, so this second pass needs the page as text.
+    if (options.context && !pass.contextFound && !(input instanceof Response)) {
+        return parseHtml(input, { ...options, context: undefined });
+    }
+    return resultOf(pass);
+};

package/dist/text.d.ts

@@ -0,0 +1,16 @@
+/**
+ * Text as HTMLRewriter delivers it: raw source text in chunks, with entities still encoded. The
+ * parser collects chunks per run of text and decodes them here, so an entity split across two
+ * chunks is still decoded correctly.
+ */
+/** Decodes numeric and common named entities; unknown names stay as written. */
+export declare const decodeEntities: (text: string) => string;
+/** Soft hyphens and zero-width characters that split words for layout only. */
+export declare const removeHyphenation: (text: string) => string;
+/** Whitespace collapsed to single spaces, as a browser renders normal text. */
+export declare const collapseWhitespace: (text: string) => string;
+/**
+ * Whether text looks like binary data or an EXIF/XMP blob rather than words. The thresholds stay
+ * conservative so umlauts and other normal characters never trigger it.
+ */
+export declare const looksBinary: (text: string) => boolean;

package/dist/text.js

@@ -0,0 +1,77 @@
+/**
+ * Text as HTMLRewriter delivers it: raw source text in chunks, with entities still encoded. The
+ * parser collects chunks per run of text and decodes them here, so an entity split across two
+ * chunks is still decoded correctly.
+ */
+const named = new Map(Object.entries({
+    Auml: "Ä",
+    Ouml: "Ö",
+    Uuml: "Ü",
+    amp: "&",
+    apos: "'",
+    auml: "ä",
+    bdquo: "„",
+    bull: "•",
+    copy: "©",
+    euro: "€",
+    gt: ">",
+    hellip: "…",
+    laquo: "«",
+    ldquo: "“",
+    lsquo: "‘",
+    lt: "<",
+    mdash: "—",
+    middot: "·",
+    nbsp: " ",
+    ndash: "–",
+    ouml: "ö",
+    quot: '"',
+    raquo: "»",
+    rdquo: "”",
+    reg: "®",
+    rsquo: "’",
+    sbquo: "‚",
+    shy: "",
+    szlig: "ß",
+    trade: "™",
+    uuml: "ü",
+}));
+const fromCodePoint = (value) => {
+    try {
+        return String.fromCodePoint(value);
+    }
+    catch {
+        return "";
+    }
+};
+/** Decodes numeric and common named entities; unknown names stay as written. */
+export const decodeEntities = (text) => text.replaceAll(/&(?:#(?<decimal>\d+)|#x(?<hex>[\da-f]+)|(?<name>[a-z]+\d?));/giu, (entity, decimal, hex, name) => {
+    if (decimal) {
+        return fromCodePoint(Number(decimal));
+    }
+    if (hex) {
+        return fromCodePoint(Number.parseInt(hex, 16));
+    }
+    return named.get(name ?? "") ?? entity;
+});
+/** Soft hyphens and zero-width characters that split words for layout only. */
+export const removeHyphenation = (text) => text.replaceAll(/[­​]/gu, "");
+/** Whitespace collapsed to single spaces, as a browser renders normal text. */
+export const collapseWhitespace = (text) => text.replaceAll(/\s+/gu, " ");
+// Control characters other than tab and line breaks, which words never contain.
+// oxlint-disable-next-line no-control-regex -- control characters are what this detects.
+const controlCharacters = /[\u0000-\u0008\u000B\u000C\u000E-\u001F]/gu;
+/**
+ * Whether text looks like binary data or an EXIF/XMP blob rather than words. The thresholds stay
+ * conservative so umlauts and other normal characters never trigger it.
+ */
+export const looksBinary = (text) => {
+    if (text.length < 16) {
+        return false;
+    }
+    if (/8BIM|Exif|Photoshop 3\.0|Adobe XMP Core|xmpmeta|rdf:RDF|xpacket/iu.test(text)) {
+        return true;
+    }
+    const control = text.match(controlCharacters)?.length ?? 0;
+    return (text.includes("\u0000") || control > 20 || control / text.length > 0.05);
+};

package/dist/urls.d.ts

@@ -0,0 +1,19 @@
+/**
+ * URL rules shared by link and image extraction: resolving against the page, telling the site's
+ * own links from others, and recognising special protocols and downloadable resources.
+ */
+export type LinkKind = "internal" | "external" | "special" | "anchor";
+/** The special protocol of a URL, such as `mailto:`, or undefined for web and relative URLs. */
+export declare const specialProtocol: (url: string) => string | undefined;
+/** The registrable part of a host, good enough to tell a site's own hosts from third parties. */
+export declare const siteOf: (host: string) => string;
+/**
+ * Resolves `url` against the page. Scheme and host are normalised by `URL`; paths keep their case,
+ * because servers treat them as case-sensitive. `mail:` becomes `mailto:` and addresses are
+ * lowercased. Special URLs other than mail stay as written.
+ */
+export declare const absoluteUrl: (url: string, base: URL) => string;
+/** Whether the URL points to a file download such as a PDF, rather than a page. */
+export declare const isResource: (url: string) => boolean;
+/** What a link is, judged from its written `href` and its resolved URL. */
+export declare const linkKind: (href: string, resolved: string, base: URL) => LinkKind;

package/dist/urls.js

@@ -0,0 +1,92 @@
+/**
+ * URL rules shared by link and image extraction: resolving against the page, telling the site's
+ * own links from others, and recognising special protocols and downloadable resources.
+ */
+const specialProtocols = [
+    "mailto:",
+    "mail:",
+    "tel:",
+    "sms:",
+    "ftp:",
+    "ftps:",
+    "ssh:",
+    "git:",
+    "svn:",
+    "news:",
+    "irc:",
+    "xmpp:",
+    "webcal:",
+    "skype:",
+    "slack:",
+    "spotify:",
+    "steam:",
+    "discord:",
+    // oxlint-disable-next-line eslint/no-script-url -- recognised so such links are not followed.
+    "javascript:",
+    "data:",
+    "blob:",
+];
+const resourceExtensions = /\.(?:pdf|docx?|xlsx?|pptx?|zip|rar|tar|gz|mp3|mp4|avi|mov|jpe?g|png|gif|svg|webp|avif|csv)$/iu;
+// Second-level labels under which registrable domains sit one level deeper, such as example.co.uk.
+const secondLevels = /^(?:co|com|org|net|gov|gv|ac|edu)$/u;
+/** The special protocol of a URL, such as `mailto:`, or undefined for web and relative URLs. */
+export const specialProtocol = (url) => {
+    const lower = url.trim().toLowerCase();
+    return specialProtocols.find((protocol) => lower.startsWith(protocol));
+};
+/** The registrable part of a host, good enough to tell a site's own hosts from third parties. */
+export const siteOf = (host) => {
+    const parts = host
+        .toLowerCase()
+        .replace(/^www\./u, "")
+        .split(".");
+    return parts.slice(secondLevels.test(parts.at(-2) ?? "") ? -3 : -2).join(".");
+};
+/**
+ * Resolves `url` against the page. Scheme and host are normalised by `URL`; paths keep their case,
+ * because servers treat them as case-sensitive. `mail:` becomes `mailto:` and addresses are
+ * lowercased. Special URLs other than mail stay as written.
+ */
+export const absoluteUrl = (url, base) => {
+    const trimmed = url.trim();
+    const protocol = specialProtocol(trimmed);
+    if (protocol === "mail:" || protocol === "mailto:") {
+        return `mailto:${trimmed.slice(protocol.length).toLowerCase()}`;
+    }
+    if (protocol) {
+        return trimmed;
+    }
+    try {
+        return new URL(trimmed, base).href;
+    }
+    catch {
+        return trimmed;
+    }
+};
+/** Whether the URL points to a file download such as a PDF, rather than a page. */
+export const isResource = (url) => {
+    try {
+        return resourceExtensions.test(new URL(url).pathname);
+    }
+    catch {
+        return resourceExtensions.test(url);
+    }
+};
+/** What a link is, judged from its written `href` and its resolved URL. */
+export const linkKind = (href, resolved, base) => {
+    const written = href.trim();
+    if (specialProtocol(written)) {
+        return "special";
+    }
+    if (written.startsWith("#") || written.startsWith("/#")) {
+        return "anchor";
+    }
+    try {
+        return siteOf(new URL(resolved).hostname) === siteOf(base.hostname)
+            ? "internal"
+            : "external";
+    }
+    catch {
+        return "external";
+    }
+};

package/package.json

@@ -0,0 +1,59 @@
+{
+  "name": "@citation-media/html-parser",
+  "version": "0.1.0",
+  "description": "Streaming HTML parser for Cloudflare Workers: links, images, resources, and Markdown without a DOM, built on HTMLRewriter.",
+  "keywords": [
+    "cloudflare-workers",
+    "html",
+    "htmlrewriter",
+    "markdown",
+    "parser",
+    "scraping"
+  ],
+  "homepage": "https://github.com/Citation-Media/html-parser#readme",
+  "bugs": "https://github.com/Citation-Media/html-parser/issues",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Citation-Media/html-parser.git"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "LICENSE"
+  ],
+  "type": "module",
+  "sideEffects": false,
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "default": "./dist/index.js"
+    }
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "build": "tsc -p tsconfig.build.json",
+    "check": "ultracite check && tsc --noEmit",
+    "fix": "ultracite fix",
+    "prepare": "npm run build",
+    "prepublishOnly": "npm run build && npm run check && npm test",
+    "test": "vitest run",
+    "typecheck": "tsc --noEmit"
+  },
+  "devDependencies": {
+    "@cloudflare/vitest-pool-workers": "0.22.0",
+    "@cloudflare/workers-types": "5.20260929.1",
+    "oxfmt": "0.68.0",
+    "oxlint": "1.83.0",
+    "typescript": "6.0.3",
+    "ultracite": "7.12.0",
+    "vitest": "4.1.11"
+  },
+  "engines": {
+    "node": ">=22.18.0"
+  }
+}