mdream 1.0.0-beta.9 → 1.0.0

package/README.md

@@ -4,189 +4,343 @@
 [![npm downloads](https://img.shields.io/npm/dm/mdream?color=yellow)](https://npm.chart.dev/mdream)
 [![license](https://img.shields.io/github/license/harlan-zw/mdream?color=yellow)](https://github.com/harlan-zw/mdream/blob/main/LICENSE.md)
 
-> Ultra-performant HTML to Markdown Convertor Optimized for LLMs. Generate llms.txt artifacts using CLI, GitHub Actions, Vite Plugin and more.
-
-<img src="../../.github/logo.png" alt="mdream logo" width="200">
-
-<p align="center">
-<table>
-<tbody>
-<td align="center">
-<sub>Made possible by my <a href="https://github.com/sponsors/harlan-zw">Sponsor Program 💖</a><br> Follow me <a href="https://twitter.com/harlan_zw">@harlan_zw</a> 🐦 • Join <a href="https://discord.gg/275MBUBvgP">Discord</a> for help</sub><br>
-</td>
-</tbody>
-</table>
-</p>
+## Installation
 
-## Features
+```bash
+# npm
+npm install mdream
 
-- 🧠 Optimized HTML To Markdown Conversion (~50% fewer tokens with Minimal preset)
-- 🔍 Generates GitHub Flavored Markdown: Frontmatter, Nested & HTML markup support.
-- 🚀 Fast: Convert 1.8MB of HTML to markdown in ~8ms (Rust), ~62ms (JS). Up to 7.9x speedup.
-- ⚡ Tiny: 10kB gzip JS core, 45kB gzip with Rust WASM engine. Zero dependencies.
-- ⚙️ Run anywhere: CLI, edge workers, browsers, Node, etc.
-- 🔌 Extensible: Declarative plugin config for both engines, hook-based plugins via `@mdream/js`.
+# pnpm
+pnpm add mdream
 
-## What is Mdream?
+# yarn
+yarn add mdream
+```
 
-Traditional HTML to Markdown converters were not built for LLMs or humans. They tend to be slow and bloated and produce output that's poorly suited for LLMs token usage or for
-human readability.
+For the JavaScript-only engine (hook-based plugins, splitter, pure HTML parser):
 
-Other LLM specific convertors focus on supporting _all_ document formats, resulting in larger bundles and lower quality Markdown output.
+```bash
+pnpm add @mdream/js
+```
 
-Mdream produces high-quality Markdown for LLMs efficiently with no core dependencies. It includes a plugin system to customize the conversion process, allowing you to parse, extract, transform, and filter as needed.
+## Table of Contents
+
+- [API Reference](#api-reference)
+  - [htmlToMarkdown()](#htmltomarkdown)
+  - [streamHtmlToMarkdown()](#streamhtmltomarkdown)
+- [Engines](#engines)
+- [Options](#options)
+  - [MdreamOptions (Rust engine)](#mdreamoptions-rust-engine)
+  - [MdreamOptions (JS engine)](#mdreamoptions-js-engine)
+  - [CleanOptions](#cleanoptions)
+  - [FrontmatterConfig](#frontmatterconfig)
+  - [TagOverride](#tagoverride)
+  - [FilterOptions](#filteroptions)
+- [Presets](#presets)
+  - [Minimal Preset](#minimal-preset)
+- [Built-in Plugins](#built-in-plugins)
+  - [Frontmatter](#frontmatter-plugin)
+  - [Isolate Main](#isolate-main-plugin)
+  - [Tailwind](#tailwind-plugin)
+  - [Filter](#filter-plugin)
+  - [Extraction](#extraction-plugin)
+- [Hook-Based Plugins (JS Engine)](#hook-based-plugins-js-engine)
+  - [Plugin Hooks](#plugin-hooks)
+  - [createPlugin()](#createplugin)
+- [Markdown Splitting (JS Engine)](#markdown-splitting-js-engine)
+  - [Basic Chunking](#basic-chunking)
+  - [Streaming Chunks](#streaming-chunks-memory-efficient)
+  - [Splitter Options](#splitter-options)
+  - [Chunk Metadata](#chunk-metadata)
+- [Content Negotiation](#content-negotiation)
+- [Pure HTML Parser (JS Engine)](#pure-html-parser-js-engine)
+- [CLI Usage](#cli-usage)
+- [Browser and Edge Usage](#browser-and-edge-usage)
+  - [Edge / Cloudflare Workers](#edge--cloudflare-workers)
+  - [Browser CDN (IIFE)](#browser-cdn-iife)
+  - [Web Worker](#web-worker)
+- [llms.txt Generation](#llmstxt-generation)
+- [Related Packages](#related-packages)
+
+## API Reference
+
+### `htmlToMarkdown()`
+
+Converts a complete HTML string to Markdown synchronously.
+
+**Rust engine** (`mdream`):
 
-## Installation
+```ts
+import { htmlToMarkdown } from 'mdream'
 
-```bash
-pnpm add mdream
+function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
 ```
 
-## CLI Usage
+**JS engine** (`@mdream/js`):
 
-Mdream provides a CLI designed to work exclusively with Unix pipes,
-providing flexibility and freedom to integrate with other tools.
+```ts
+import { htmlToMarkdown } from '@mdream/js'
 
-**Pipe Site to Markdown**
+function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
+```
 
-Fetches the [Markdown Wikipedia page](https://en.wikipedia.org/wiki/Markdown) and converts it to Markdown preserving the original links and images.
+**Example:**
 
-```bash
-curl -s https://en.wikipedia.org/wiki/Markdown \
- | npx mdream --origin https://en.wikipedia.org --preset minimal \
-  | tee streaming.md
+```ts
+import { htmlToMarkdown } from 'mdream'
+
+const markdown = htmlToMarkdown('<h1>Hello World</h1><p>Some content.</p>')
+// # Hello World
+//
+// Some content.
 ```
 
-_Tip: The `--origin` flag will fix relative image and link paths_
+### `streamHtmlToMarkdown()`
 
-**Local File to Markdown**
+Converts an HTML `ReadableStream` to Markdown incrementally. Returns an `AsyncIterable<string>` that yields Markdown chunks as they are processed.
 
-Converts a local HTML file to a Markdown file, using `tee` to write the output to a file and display it in the terminal.
 
-```bash
-cat index.html \
- | npx mdream --preset minimal \
-  | tee streaming.md
+```ts
+import { streamHtmlToMarkdown } from 'mdream'
+
+function streamHtmlToMarkdown(
+  htmlStream: ReadableStream<Uint8Array | string> | null,
+  options?: Partial<MdreamOptions>,
+): AsyncIterable<string>
 ```
 
-### CLI Options
 
-- `--origin <url>`: Base URL for resolving relative links and images
-- `--preset <preset>`: Conversion presets: minimal
-- `--help`: Display help information
-- `--version`: Display version information
+**Example:**
 
-## API Usage
+```ts
+import { streamHtmlToMarkdown } from 'mdream'
 
-Mdream provides two main functions for working with HTML:
-- `htmlToMarkdown`: Useful if you already have the entire HTML payload you want to convert.
-- `streamHtmlToMarkdown`: Best practice if you are fetching or reading from a local file.
+const response = await fetch('https://example.com')
+const stream = response.body
 
-### Engines
+for await (const chunk of streamHtmlToMarkdown(stream, {
+  origin: 'https://example.com',
+})) {
+  process.stdout.write(chunk)
+}
+```
+
+## Engines
 
 Mdream includes two rendering engines, automatically selecting the best one for your environment:
-- **Rust Engine** (default in Node.js): Native NAPI performance, 5.6-7.9x faster. WASM build for edge/browser runtimes.
-- **JavaScript Engine** (`@mdream/js`): Zero-dependencies, supports custom hook-based plugins.
+
+| Engine | Package | Plugins | Use case |
+|--------|---------|---------|----------|
+| **Rust** (NAPI) | `mdream` | Declarative config only | Node.js (default) |
+| **Rust** (WASM) | `mdream` | Declarative config only | Edge, browser |
+| **JavaScript** | `@mdream/js` | Hook-based + declarative | Custom plugins, splitter |
 
 ```ts
-import { htmlToMarkdown } from 'mdream'
+// JavaScript engine (required for hook-based plugins)
+import { htmlToMarkdown } from '@mdream/js'
 
-// Rust NAPI engine used automatically in Node.js
-// JS engine used in browser/edge runtimes
-const markdown = htmlToMarkdown('<h1>Hello World</h1>')
+// Rust NAPI engine (auto-selected in Node.js)
+import { htmlToMarkdown } from 'mdream'
 ```
 
-## Browser & Edge Usage
+Both engines accept the same declarative plugin configuration (`origin`, `minimal`, `frontmatter`, `isolateMain`, `tailwind`, `filter`, `extraction`, `tagOverrides`, `clean`). The JS engine additionally supports `hooks` for imperative plugin transforms.
 
-For browser environments and edge runtimes (Cloudflare Workers, Vercel Edge), mdream compiles to WebAssembly. Export conditions (`workerd`, `edge-light`, `browser`) select the correct build automatically, or use `mdream/worker` directly:
+## Options
 
-```ts
-import { htmlToMarkdown } from 'mdream/worker'
+### MdreamOptions (Rust engine)
+
+Defined in `mdream`:
 
-const markdown = await htmlToMarkdown('<h1>Hello World</h1>')
+```ts
+interface MdreamOptions {
+  /** Base URL for resolving relative links and images. */
+  origin?: string
+
+  /**
+   * Enable minimal preset (frontmatter, isolateMain, tailwind, filter).
+   * Default: false
+   */
+  minimal?: boolean
+
+  /**
+   * Post-processing cleanup. Pass `true` for all cleanup, or an object for specific features.
+   * Enabled by default when `minimal` is true.
+   */
+  clean?: boolean | CleanOptions
+
+  /**
+   * Extract frontmatter from HTML <head>.
+   * - `true`: enable with defaults
+   * - `(fm) => void`: enable and receive structured data via callback
+   * - `FrontmatterConfig`: enable with config and optional callback
+   */
+  frontmatter?: boolean | ((frontmatter: Record<string, string>) => void) | FrontmatterConfig
+
+  /** Isolate main content area. Default when minimal: true */
+  isolateMain?: boolean
+
+  /** Convert Tailwind utility classes to Markdown. Default when minimal: true */
+  tailwind?: boolean
+
+  /** Filter elements by CSS selectors. Default when minimal: excludes form, nav, footer, etc. */
+  filter?: { include?: string[], exclude?: string[], processChildren?: boolean }
+
+  /** Extract elements matching CSS selectors during conversion. */
+  extraction?: Record<string, (element: ExtractedElement) => void>
+
+  /** Override tag rendering behavior. String values act as aliases. */
+  tagOverrides?: Record<string, TagOverride | string>
+}
 ```
 
-### Browser CDN Usage
+### MdreamOptions (JS engine)
 
-Use mdream directly via CDN with no build step. The IIFE bundle uses the Rust WASM engine. Call `init()` once to load the WASM binary, then use `htmlToMarkdown()` synchronously.
+The JS engine extends the shared `EngineOptions` with hook-based plugin support:
 
-```html
-<script src="https://unpkg.com/mdream/dist/iife.js"></script>
-<script>
-  // init() fetches the .wasm file from the same CDN path automatically
-  await window.mdream.init()
-  const markdown = window.mdream.htmlToMarkdown('<h1>Hello</h1><p>World</p>')
-  console.log(markdown) // # Hello\n\nWorld
-</script>
-```
+```ts
+interface MdreamOptions extends EngineOptions {
+  /** Imperative hook-based transform plugins. JS engine only. */
+  hooks?: TransformPlugin[]
+}
 
-You can also pass a custom WASM URL or `ArrayBuffer` to `init()`:
+interface EngineOptions {
+  origin?: string
+  clean?: boolean | CleanOptions
+  plugins?: BuiltinPlugins
+}
 
-```js
-await window.mdream.init('https://cdn.example.com/mdream_edge_bg.wasm')
+interface BuiltinPlugins {
+  filter?: { include?: (string | number)[], exclude?: (string | number)[], processChildren?: boolean }
+  frontmatter?: boolean | ((fm: Record<string, string>) => void) | FrontmatterConfig
+  isolateMain?: boolean
+  tailwind?: boolean
+  extraction?: Record<string, (element: ExtractedElement) => void>
+  tagOverrides?: Record<string, TagOverride | string>
+}
 ```
 
-**CDN Options:**
-- **unpkg**: `https://unpkg.com/mdream/dist/iife.js`
-- **jsDelivr**: `https://cdn.jsdelivr.net/npm/mdream/dist/iife.js`
+Note: The JS engine uses `options.plugins.filter` while the Rust engine uses `options.filter` directly.
 
-**Convert existing HTML**
+### CleanOptions
 
-```ts
-import { htmlToMarkdown } from 'mdream'
+Post-processing cleanup applied to the final Markdown output. All options default to `false` unless `clean: true` is set.
 
-// Simple conversion
-const markdown = htmlToMarkdown('<h1>Hello World</h1>')
-console.log(markdown) // # Hello World
+```ts
+interface CleanOptions {
+  /** Strip tracking query parameters (utm_*, fbclid, gclid, etc.) from URLs */
+  urls?: boolean
+  /** Strip fragment-only links that don't match any heading in the output */
+  fragments?: boolean
+  /** Strip links with meaningless hrefs (#, javascript:void(0)) to plain text */
+  emptyLinks?: boolean
+  /** Collapse 3+ consecutive blank lines to 2 */
+  blankLines?: boolean
+  /** Strip links where text equals URL: [https://x.com](https://x.com) becomes https://x.com */
+  redundantLinks?: boolean
+  /** Strip self-referencing heading anchors: ## [Title](#title) becomes ## Title */
+  selfLinkHeadings?: boolean
+  /** Strip images with no alt text (decorative/tracking pixels) */
+  emptyImages?: boolean
+  /** Drop links that produce no visible text: [](url) is removed entirely */
+  emptyLinkText?: boolean
+}
 ```
 
-**Convert from Fetch**
+**Example:**
 
 ```ts
-import { streamHtmlToMarkdown } from 'mdream'
-
-// Using fetch with streaming
-const response = await fetch('https://example.com')
-const htmlStream = response.body
-const markdownGenerator = streamHtmlToMarkdown(htmlStream, {
-  origin: 'https://example.com'
+const markdown = htmlToMarkdown(html, {
+  clean: {
+    urls: true,
+    emptyLinks: true,
+    emptyImages: true,
+  },
 })
+```
+
+### FrontmatterConfig
 
-// Process chunks as they arrive
-for await (const chunk of markdownGenerator) {
-  console.log(chunk)
+```ts
+interface FrontmatterConfig {
+  /** Additional static fields to include in frontmatter */
+  additionalFields?: Record<string, string>
+  /**
+   * Meta tag names to extract beyond the defaults.
+   * Defaults: description, keywords, author, date,
+   * og:title, og:description, twitter:title, twitter:description
+   */
+  metaFields?: string[]
+  /** Callback to receive structured frontmatter data after conversion */
+  onExtract?: (frontmatter: Record<string, string>) => void
 }
 ```
 
-**Pure HTML Parser (JS Engine)**
+### TagOverride
 
-If you only need to parse HTML into a DOM-like AST without converting to Markdown, use `parseHtml` from the JS engine:
+Override how specific HTML tags are rendered in Markdown. String values act as aliases.
 
 ```ts
-import { parseHtml } from '@mdream/js'
+interface TagOverride {
+  /** Markdown string to insert when entering this tag */
+  enter?: string
+  /** Markdown string to insert when exiting this tag */
+  exit?: string
+  /** Spacing: [newlines before, newlines after] */
+  spacing?: number[]
+  /** Whether this tag should be treated as inline */
+  isInline?: boolean
+  /** Whether this tag is self-closing */
+  isSelfClosing?: boolean
+  /** Whether whitespace inside this tag should be collapsed */
+  collapsesInnerWhiteSpace?: boolean
+  /** Alias this tag to another tag's handler */
+  alias?: string
+}
+```
 
-const html = '<div><h1>Title</h1><p>Content</p></div>'
-const { events, remainingHtml } = parseHtml(html)
+**Example:**
 
-// Process the parsed events
-events.forEach((event) => {
-  if (event.type === 'enter' && event.node.type === 'element') {
-    console.log('Entering element:', event.node.tagName)
-  }
+```ts
+const markdown = htmlToMarkdown(html, {
+  tagOverrides: {
+    // Treat <x-heading> like <h2>
+    'x-heading': 'h2',
+    // Custom rendering for <callout>
+    'callout': {
+      enter: '> **Note:** ',
+      exit: '',
+      spacing: [2, 2],
+    },
+  },
 })
 ```
 
-The `parseHtml` function provides:
-- **Pure AST parsing** - No markdown generation overhead
-- **DOM events** - Enter/exit events for each element and text node
-- **Plugin support** - Can apply plugins during parsing
-- **Streaming compatible** - Works with the same plugin system
+### FilterOptions
+
+```ts
+interface FilterOptions {
+  /** CSS selectors, tag names, or TAG_* constants for elements to include (all others excluded) */
+  include?: (string | number)[]
+  /** CSS selectors, tag names, or TAG_* constants for elements to exclude */
+  exclude?: (string | number)[]
+  /** Whether to also process children of matched elements. Default: true */
+  processChildren?: boolean
+}
+```
 
 ## Presets
 
 ### Minimal Preset
 
-The `minimal` preset optimizes for token reduction and cleaner output by removing non-essential content:
+The `minimal` preset enables the following plugins together:
+
+- **frontmatter**: Extracts metadata from HTML `<head>` into YAML frontmatter
+- **isolateMain**: Extracts the main content area, skipping navigation, headers, and footers
+- **tailwind**: Converts Tailwind utility classes to Markdown formatting
+- **filter**: Excludes `form`, `fieldset`, `object`, `embed`, `footer`, `aside`, `iframe`, `input`, `textarea`, `select`, `button`, `nav`
+- **clean**: All post-processing cleanup enabled
+
+**Rust engine:**
 
 ```ts
 import { htmlToMarkdown } from 'mdream'
@@ -197,75 +351,170 @@ const markdown = htmlToMarkdown(html, {
 })
 ```
 
-**Enables:**
-- `isolateMain` - Extracts main content area
-- `frontmatter` - Generates YAML frontmatter from meta tags
-- `tailwind` - Converts Tailwind classes to Markdown
-- `filter` - Excludes forms, navigation, buttons, footers, and other non-content elements
+**JS engine (using `withMinimalPreset`):**
 
-**CLI Usage:**
-```bash
-curl -s https://example.com | npx mdream --preset minimal --origin https://example.com
+```ts
+import { htmlToMarkdown } from '@mdream/js'
+import { withMinimalPreset } from '@mdream/js/preset/minimal'
+
+const markdown = htmlToMarkdown(html, withMinimalPreset({
+  origin: 'https://example.com',
+}))
+```
+
+`withMinimalPreset()` returns an `EngineOptions` object with all plugin defaults applied. You can override individual plugins:
+
+```ts
+const markdown = htmlToMarkdown(html, withMinimalPreset({
+  plugins: {
+    frontmatter: false,
+    filter: { exclude: ['nav'] },
+  },
+}))
 ```
 
-## Declarative Options
+## Built-in Plugins
+
+All built-in plugins work with both the Rust and JS engines through declarative configuration.
+
+### Frontmatter Plugin
 
-Both engines accept the same declarative configuration:
+Extracts metadata from the HTML `<head>` element and generates YAML frontmatter.
+
+**Extracted fields by default:** `title`, `description`, `keywords`, `author`, `date`, `og:title`, `og:description`, `twitter:title`, `twitter:description`.
 
 ```ts
-import { htmlToMarkdown } from 'mdream'
+// Enable with defaults
+htmlToMarkdown(html, { frontmatter: true })
+
+// With callback to receive structured data
+htmlToMarkdown(html, {
+  frontmatter: (fm) => {
+    console.log(fm.title)
+    console.log(fm.description)
+  },
+})
 
-const markdown = htmlToMarkdown(html, {
-  origin: 'https://example.com',
-  minimal: true, // enables frontmatter, isolateMain, tailwind, filter
-  clean: true, // enable all post-processing cleanup
-  frontmatter: fm => console.log(fm), // callback for extracted frontmatter
-  filter: { exclude: ['nav', '.sidebar'] },
-  extraction: {
-    'h2': el => console.log('Heading:', el.textContent),
-    'img[alt]': el => console.log('Image:', el.attributes.src),
+// With full config
+htmlToMarkdown(html, {
+  frontmatter: {
+    additionalFields: { source: 'https://example.com' },
+    metaFields: ['robots', 'viewport'],
+    onExtract: fm => console.log(fm),
   },
-  tagOverrides: { 'custom-tag': { alias: 'div' } },
 })
 ```
 
-### Available Options
+**Output example:**
 
-| Option | Type | Description |
-|--------|------|-------------|
-| `origin` | `string` | Base URL for resolving relative links/images |
-| `minimal` | `boolean` | Enable minimal preset (frontmatter, isolateMain, tailwind, filter) |
-| `clean` | `boolean \| CleanOptions` | Post-processing cleanup (`true` for all, or pick specific) |
-| `frontmatter` | `boolean \| (fm) => void \| FrontmatterConfig` | Extract frontmatter from HTML head |
-| `isolateMain` | `boolean` | Isolate main content area |
-| `tailwind` | `boolean` | Convert Tailwind classes to Markdown |
-| `filter` | `{ include?, exclude?, processChildren? }` | Filter elements by CSS selectors |
-| `extraction` | `Record<string, (el) => void>` | Extract elements matching CSS selectors |
-| `tagOverrides` | `Record<string, TagOverride \| string>` | Override tag rendering behavior |
+```yaml
+---
+title: My Page Title
+meta:
+  description: A page description
+  'og:title': My Page Title
+---
+```
 
-### Content Extraction with Readability
+### Isolate Main Plugin
 
-For advanced content extraction (article detection, boilerplate removal), use [@mozilla/readability](https://github.com/mozilla/readability) before mdream:
+Isolates the main content area using the following priority:
+
+1. If an explicit `<main>` element exists (within 5 depth levels), use its content exclusively
+2. Otherwise, find content between the first header tag (`h1`-`h6`) and the first `<footer>`
+3. Headings inside `<header>` tags are skipped during fallback detection
+4. The `<head>` section is always passed through for other plugins (e.g., frontmatter)
 
 ```ts
-import { Readability } from '@mozilla/readability'
-import { JSDOM } from 'jsdom'
-import { htmlToMarkdown } from 'mdream'
+htmlToMarkdown(html, { isolateMain: true })
+```
 
-const dom = new JSDOM(html, { url: 'https://example.com' })
-const article = new Readability(dom.window.document).parse()
+### Tailwind Plugin
 
-if (article) {
-  const markdown = htmlToMarkdown(article.content)
-  // article.title, article.excerpt, article.byline also available
-}
+Converts Tailwind CSS utility classes to semantic Markdown formatting:
+
+| Tailwind Class | Markdown Output |
+|---|---|
+| `font-bold`, `font-semibold`, `font-medium`, `font-extrabold`, `font-black` | `**bold**` |
+| `italic`, `font-italic` | `*italic*` |
+| `line-through` | `~~strikethrough~~` |
+| `hidden`, `invisible` | Content removed |
+| `absolute`, `fixed`, `sticky` | Content removed |
+
+Supports responsive breakpoint prefixes (`sm:`, `md:`, `lg:`, `xl:`, `2xl:`) with mobile-first resolution.
+
+```ts
+htmlToMarkdown(html, { tailwind: true })
 ```
 
-This pipeline gives you battle-tested content extraction + fast markdown conversion.
+### Filter Plugin
+
+Filters HTML elements by CSS selectors, tag names, or `TAG_*` constants.
+
+```ts
+// Exclude navigation, sidebar, footer
+htmlToMarkdown(html, {
+  filter: {
+    exclude: ['nav', '#sidebar', '.footer', 'aside'],
+  },
+})
+
+// Include only specific elements
+htmlToMarkdown(html, {
+  filter: {
+    include: ['article', 'main'],
+  },
+})
+```
+
+The JS engine also supports `TAG_*` integer constants for filtering:
+
+```ts
+import { TAG_FOOTER, TAG_NAV } from '@mdream/js'
+
+htmlToMarkdown(html, {
+  plugins: {
+    filter: { exclude: [TAG_NAV, TAG_FOOTER] },
+  },
+})
+```
+
+Elements with `style="position: absolute"` or `style="position: fixed"` are also automatically excluded when the filter plugin is active.
+
+### Extraction Plugin
+
+Extracts elements matching CSS selectors during conversion. Callbacks receive the matched element with its accumulated text content and attributes.
+
+```ts
+htmlToMarkdown(html, {
+  extraction: {
+    'h2': (el) => {
+      console.log('Heading:', el.textContent)
+    },
+    'img[alt]': (el) => {
+      console.log('Image:', el.attributes.src, el.attributes.alt)
+    },
+    'a[href]': (el) => {
+      console.log('Link:', el.textContent, el.attributes.href)
+    },
+  },
+})
+```
+
+The `ExtractedElement` interface:
+
+```ts
+interface ExtractedElement {
+  selector: string
+  tagName: string
+  textContent: string
+  attributes: Record<string, string>
+}
+```
 
 ## Hook-Based Plugins (JS Engine)
 
-For custom hook-based plugins, use `@mdream/js`:
+The JS engine (`@mdream/js`) supports imperative hook-based plugins for custom transform logic. These allow you to intercept and modify the conversion pipeline at multiple stages.
 
 ```ts
 import { htmlToMarkdown } from '@mdream/js'
@@ -280,7 +529,7 @@ const myPlugin = createPlugin({
     if (textNode.parent?.attributes?.id === 'highlight') {
       return { content: `**${textNode.value}**`, skip: false }
     }
-  }
+  },
 })
 
 const markdown = htmlToMarkdown(html, { hooks: [myPlugin] })
@@ -288,15 +537,91 @@ const markdown = htmlToMarkdown(html, { hooks: [myPlugin] })
 
 ### Plugin Hooks
 
-- `beforeNodeProcess`: Called before any node processing, can skip nodes
-- `onNodeEnter`: Called when entering an element node
-- `onNodeExit`: Called when exiting an element node
-- `processTextNode`: Called for each text node
-- `processAttributes`: Called to process element attributes
+```ts
+interface TransformPlugin {
+  /**
+   * Called before any node processing. Return { skip: true } to skip the node.
+   */
+  beforeNodeProcess?: (
+    event: NodeEvent,
+    state: MdreamRuntimeState,
+  ) => undefined | void | { skip: boolean }
+
+  /**
+   * Called when entering an element node.
+   * Return a string to prepend to the output.
+   */
+  onNodeEnter?: (
+    node: ElementNode,
+    state: MdreamRuntimeState,
+  ) => string | undefined | void
+
+  /**
+   * Called when exiting an element node.
+   * Return a string to append to the output.
+   */
+  onNodeExit?: (
+    node: ElementNode,
+    state: MdreamRuntimeState,
+  ) => string | undefined | void
+
+  /**
+   * Called to process element attributes (e.g., extracting Tailwind classes).
+   */
+  processAttributes?: (
+    node: ElementNode,
+    state: MdreamRuntimeState,
+  ) => void
+
+  /**
+   * Called for each text node. Return { content, skip } to transform text.
+   * Return undefined for no transformation.
+   */
+  processTextNode?: (
+    node: TextNode,
+    state: MdreamRuntimeState,
+  ) => { content: string, skip: boolean } | undefined
+}
+```
+
+### `createPlugin()`
+
+A typed identity function for creating plugins with full TypeScript inference:
+
+```ts
+import { createPlugin } from '@mdream/js/plugins'
+
+const plugin = createPlugin({
+  beforeNodeProcess({ node }) {
+    // Skip all div elements with class "ad"
+    if (node.type === 1 && node.attributes?.class?.includes('ad')) {
+      return { skip: true }
+    }
+  },
+})
+```
+
+### Built-in Plugin Functions (JS Engine)
+
+The following plugin factory functions are available from `@mdream/js/plugins`:
+
+```ts
+import {
+  createPlugin,
+  extractionCollectorPlugin,
+  extractionPlugin,
+  filterPlugin,
+  frontmatterPlugin,
+  isolateMainPlugin,
+  tailwindPlugin,
+} from '@mdream/js/plugins'
+```
+
+## Markdown Splitting (JS Engine)
 
-## Markdown Splitting
+Split HTML into Markdown chunks during conversion. Compatible with the LangChain `Document` structure.
 
-Split HTML into chunks during conversion for LLM context windows, vector databases, or document processing.
+Available from `@mdream/js/splitter`.
 
 ### Basic Chunking
 
@@ -313,18 +638,17 @@ const html = `
 `
 
 const chunks = htmlToMarkdownSplitChunks(html, {
-  headersToSplitOn: [TAG_H2], // Split on h2 headers
-  chunkSize: 1000, // Max chars per chunk
-  chunkOverlap: 200, // Overlap for context
-  stripHeaders: true // Remove headers from content
+  headersToSplitOn: [TAG_H2],
+  chunkSize: 1000,
+  chunkOverlap: 200,
+  stripHeaders: true,
 })
 
-// Each chunk includes content and metadata
 chunks.forEach((chunk) => {
   console.log(chunk.content)
   console.log(chunk.metadata.headers) // { h1: "Documentation", h2: "Installation" }
   console.log(chunk.metadata.code) // Language if chunk contains code
-  console.log(chunk.metadata.loc) // Line numbers
+  console.log(chunk.metadata.loc) // { lines: { from: 1, to: 5 } }
 })
 ```
 
@@ -335,54 +659,82 @@ For large documents, use the generator version to process chunks one at a time:
 ```ts
 import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'
 
-// Process chunks incrementally - lower memory usage
 for (const chunk of htmlToMarkdownSplitChunksStream(html, options)) {
-  await processChunk(chunk) // Handle each chunk as it's generated
+  await processChunk(chunk)
 
-  // Can break early if you found what you need
+  // Early termination supported
   if (foundTarget)
     break
 }
 ```
 
-**Benefits of streaming:**
-- Lower memory usage - chunks aren't stored in an array
-- Early termination - stop processing when you find what you need
-- Better for large documents
-
-### Splitting Options
+### Splitter Options
 
 ```ts
 interface SplitterOptions {
-  // Structural splitting
-  headersToSplitOn?: number[] // TAG_H1, TAG_H2, etc. Default: [TAG_H2-TAG_H6]
+  // --- Structural splitting ---
+
+  /**
+   * Header tag IDs to split on (TAG_H1 through TAG_H6).
+   * Default: [TAG_H2, TAG_H3, TAG_H4, TAG_H5, TAG_H6]
+   */
+  headersToSplitOn?: number[]
+
+  // --- Size-based splitting ---
+
+  /** Maximum chunk size in characters. Default: 1000 */
+  chunkSize?: number
+
+  /** Overlap between chunks for context preservation. Default: 200 */
+  chunkOverlap?: number
+
+  /**
+   * Custom length function (e.g., a token counter for LLM applications).
+   * Default: (text) => text.length
+   */
+  lengthFunction?: (text: string) => number
+
+  // --- Output formatting ---
+
+  /** Remove headers from chunk content. Default: true */
+  stripHeaders?: boolean
 
-  // Size-based splitting
-  chunkSize?: number // Max chunk size. Default: 1000
-  chunkOverlap?: number // Overlap between chunks. Default: 200
-  lengthFunction?: (text: string) => number // Custom length (e.g., token count)
+  /** Split into individual lines. Default: false */
+  returnEachLine?: boolean
 
-  // Output formatting
-  stripHeaders?: boolean // Remove headers from content. Default: true
-  returnEachLine?: boolean // Split into individual lines. Default: false
+  /** Keep separators in the split chunks. Default: false */
+  keepSeparator?: boolean
 
-  // Standard options
-  origin?: string // Base URL for links/images
-  hooks?: TransformPlugin[] // Apply hook-based plugins during conversion (@mdream/js only)
+  // --- Standard options ---
+
+  /** Base URL for resolving relative links/images */
+  origin?: string
+
+  /** Declarative built-in plugin config */
+  plugins?: BuiltinPlugins
+
+  /** Hook-based plugins (JS engine only) */
+  hooks?: TransformPlugin[]
+
+  /** Post-processing cleanup */
+  clean?: boolean | CleanOptions
 }
 ```
 
 ### Chunk Metadata
 
-Each chunk includes rich metadata for context:
+Each chunk includes metadata for context:
 
 ```ts
 interface MarkdownChunk {
   content: string
   metadata: {
-    headers?: Record<string, string> // Header hierarchy: { h1: "Title", h2: "Section" }
-    code?: string // Code block language if present
-    loc?: { // Line number range
+    /** Header hierarchy at this chunk position */
+    headers?: Record<string, string> // { h1: "Title", h2: "Section" }
+    /** Code block language if chunk contains code */
+    code?: string
+    /** Line number range in the original document */
+    loc?: {
       lines: { from: number, to: number }
     }
   }
@@ -391,7 +743,7 @@ interface MarkdownChunk {
 
 ### Use with Presets
 
-Combine splitting with presets for optimized output:
+Combine splitting with presets:
 
 ```ts
 import { TAG_H2 } from '@mdream/js'
@@ -401,13 +753,189 @@ import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
 const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
   headersToSplitOn: [TAG_H2],
   chunkSize: 500,
-  origin: 'https://example.com'
+  origin: 'https://example.com',
 }))
 ```
 
+## Content Negotiation
+
+The `@mdream/js/negotiate` module provides HTTP content negotiation utilities for serving Markdown to LLM clients:
+
+```ts
+import { parseAcceptHeader, shouldServeMarkdown } from '@mdream/js/negotiate'
+
+// Check if client prefers markdown
+const serveMarkdown = shouldServeMarkdown(
+  request.headers.get('accept'),
+  request.headers.get('sec-fetch-dest'),
+)
+
+if (serveMarkdown) {
+  return new Response(markdown, {
+    headers: { 'Content-Type': 'text/markdown' },
+  })
+}
+```
+
+`shouldServeMarkdown()` uses Accept header quality weights and position ordering. It returns `true` when `text/markdown` or `text/plain` has higher priority than `text/html`. Browser navigation requests (`sec-fetch-dest: document`) always return `false`.
+
+## Pure HTML Parser (JS Engine)
+
+If you only need to parse HTML into a DOM-like event stream without converting to Markdown, use `parseHtml` from the JS engine:
+
+```ts
+import { parseHtml } from '@mdream/js/parse'
+
+const html = '<div><h1>Title</h1><p>Content</p></div>'
+const { events, remainingHtml } = parseHtml(html)
+
+events.forEach((event) => {
+  if (event.type === 0 && event.node.type === 1) { // Enter + Element
+    console.log('Entering element:', event.node.name)
+  }
+})
+```
+
+The parser provides:
+- Pure AST event stream with no markdown generation overhead
+- Enter/exit events for each element and text node
+- Plugin support during parsing
+- Streaming compatible via `parseHtmlStream()`
+
+## CLI Usage
+
+Mdream provides a CLI that works with Unix pipes.
+
+**Pipe site to Markdown:**
+
+```bash
+curl -s https://en.wikipedia.org/wiki/Markdown \
+  | npx mdream --origin https://en.wikipedia.org --preset minimal \
+  | tee output.md
+```
+
+**Local file to Markdown:**
+
+```bash
+cat index.html \
+  | npx mdream --preset minimal \
+  | tee output.md
+```
+
+### CLI Options
+
+| Option | Description |
+|--------|-------------|
+| `--origin <url>` | Base URL for resolving relative links and images |
+| `--preset minimal` | Enable the minimal preset |
+| `-h`, `--help` | Display help information |
+
+The CLI reads HTML from stdin and writes Markdown to stdout. It uses the streaming API internally.
+
+## Browser and Edge Usage
+
+### Edge / Cloudflare Workers
+
+For edge runtimes (Cloudflare Workers, Vercel Edge), `mdream` automatically selects the WASM build via export conditions (`workerd`, `edge-light`). Both `htmlToMarkdown` and `streamHtmlToMarkdown` are available:
+
+```ts
+import { htmlToMarkdown, streamHtmlToMarkdown } from 'mdream'
+
+// WASM engine auto-selected via export conditions
+const markdown = htmlToMarkdown('<h1>Hello World</h1>')
+
+// Streaming works the same as Node.js
+const response = await fetch('https://example.com')
+for await (const chunk of streamHtmlToMarkdown(response.body)) {
+  // process chunk
+}
+```
+
+You can also import the edge entry point directly:
+
+```ts
+import { htmlToMarkdown } from 'mdream/worker'
+```
+
+The `mdream/worker` entry provides an async API since WASM must be initialized first:
+
+```ts
+import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
+
+// Initialize once with the WASM URL
+await initWorker('https://cdn.example.com/mdream_edge_bg.wasm')
+
+// Convert (returns Promise<string>)
+const markdown = await htmlToMarkdown('<h1>Hello</h1>')
+
+// Clean up when done
+terminateWorker()
+```
+
+### Browser CDN (IIFE)
+
+Use mdream directly via CDN with no build step. Call `init()` once to load the WASM binary, then use `htmlToMarkdown()` synchronously.
+
+```html
+<script src="https://unpkg.com/mdream/dist/iife.js"></script>
+<script>
+  await window.mdream.init()
+  const markdown = window.mdream.htmlToMarkdown('<h1>Hello</h1><p>World</p>')
+  console.log(markdown) // # Hello\n\nWorld
+</script>
+```
+
+You can pass a custom WASM URL or `ArrayBuffer` to `init()`:
+
+```js
+// Custom URL
+await window.mdream.init('https://cdn.example.com/mdream_edge_bg.wasm')
+
+// Pre-loaded ArrayBuffer
+const wasmBytes = await fetch('/wasm/mdream_edge_bg.wasm').then(r => r.arrayBuffer())
+await window.mdream.init(wasmBytes)
+```
+
+**CDN Options:**
+- **unpkg**: `https://unpkg.com/mdream/dist/iife.js`
+- **jsDelivr**: `https://cdn.jsdelivr.net/npm/mdream/dist/iife.js`
+
+### Web Worker
+
+For browser environments, `mdream/worker` runs conversions off the main thread using a Web Worker:
+
+```ts
+import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
+
+await initWorker('/path/to/mdream_edge_bg.wasm')
+
+const markdown = await htmlToMarkdown('<h1>Hello</h1>')
+
+// Clean up
+terminateWorker()
+```
+
+## Content Extraction with Readability
+
+For advanced content extraction (article detection, boilerplate removal), use [@mozilla/readability](https://github.com/mozilla/readability) before mdream:
+
+```ts
+import { Readability } from '@mozilla/readability'
+import { JSDOM } from 'jsdom'
+import { htmlToMarkdown } from 'mdream'
+
+const dom = new JSDOM(html, { url: 'https://example.com' })
+const article = new Readability(dom.window.document).parse()
+
+if (article) {
+  const markdown = htmlToMarkdown(article.content)
+  // article.title, article.excerpt, article.byline also available
+}
+```
+
 ## llms.txt Generation
 
-For llms.txt artifact generation, use `@mdream/llms-txt`. It accepts pre-converted markdown and generates `llms.txt` and `llms-full.txt` artifacts.
+For llms.txt artifact generation, use the separate `@mdream/llms-txt` package. It accepts pre-converted Markdown and generates `llms.txt` and `llms-full.txt` artifacts.
 
 ```ts
 import { generateLlmsTxtArtifacts } from '@mdream/llms-txt'
@@ -427,11 +955,18 @@ console.log(result.llmsTxt) // llms.txt content
 console.log(result.llmsFullTxt) // llms-full.txt content
 ```
 
-## Credits
+## Related Packages
 
-- [ultrahtml](https://github.com/natemoo-re/ultrahtml): HTML parsing inspiration
+| Package | Description |
+|---------|-------------|
+| [`mdream`](https://npmjs.com/package/mdream) | Core HTML to Markdown converter (Rust + WASM engine) |
+| [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with hook-based plugins and splitter |
+| [`@mdream/llms-txt`](https://github.com/harlan-zw/mdream/tree/main/packages/llms-txt) | Engine-agnostic llms.txt artifact generation |
+| [`@mdream/crawl`](https://github.com/harlan-zw/mdream/tree/main/packages/crawl) | Site-wide crawler for llms.txt generation |
+| [`@mdream/vite`](https://github.com/harlan-zw/mdream/tree/main/packages/vite) | Vite plugin integration |
+| [`@mdream/nuxt`](https://github.com/harlan-zw/mdream/tree/main/packages/nuxt) | Nuxt module integration |
+| [`@mdream/action`](https://github.com/harlan-zw/mdream/tree/main/packages/action) | GitHub Actions integration |
 
 ## License
 
 Licensed under the [MIT license](https://github.com/harlan-zw/mdream/blob/main/LICENSE.md).
-