mdream 1.7.2 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -3,6 +3,18 @@
3
3
  [![npm version](https://img.shields.io/npm/v/mdream?color=yellow)](https://npmjs.com/package/mdream)
4
4
  [![npm downloads](https://img.shields.io/npm/dm/mdream?color=yellow)](https://npm.chart.dev/mdream)
5
5
  [![license](https://img.shields.io/github/license/harlan-zw/mdream?color=yellow)](https://github.com/harlan-zw/mdream/blob/main/LICENSE.md)
6
+ <a href="https://skilld.dev/gh/harlan-zw/mdream">
7
+ <picture>
8
+ <source media="(prefers-color-scheme: dark)" srcset="https://skilld.dev/b/harlan-zw/mdream?theme=dark">
9
+ <source media="(prefers-color-scheme: light)" srcset="https://skilld.dev/b/harlan-zw/mdream?theme=light">
10
+ <img alt="Skill repository on skilld.dev" src="https://skilld.dev/b/harlan-zw/mdream?theme=light">
11
+ </picture>
12
+ </a>
13
+
14
+ Powering Cloudflare Browser Run's [/markdown](https://developers.cloudflare.com/browser-run/quick-actions/markdown-endpoint/) and [/crawl](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/) endpoints.
15
+
16
+ > [!TIP]
17
+ > 🎉 **Upgrading to Mdream v2?** Follow [Migrating from v1](#migrating-from-v1).
6
18
 
7
19
  ## Installation
8
20
 
@@ -17,7 +29,10 @@ pnpm add mdream
17
29
  yarn add mdream
18
30
  ```
19
31
 
20
- For the JavaScript-only engine (hook-based plugins, splitter, pure HTML parser):
32
+ > [!TIP]
33
+ > Using an AI agent? Get the mdream Skill on [skilld.dev/gh/harlan-zw/mdream](https://skilld.dev/gh/harlan-zw/mdream).
34
+
35
+ For the JavaScript-only engine with explicit plugins, formats, splitter, and parser:
21
36
 
22
37
  ```bash
23
38
  pnpm add @mdream/js
@@ -48,6 +63,7 @@ externals: ['mdream']
48
63
 
49
64
  ## Table of Contents
50
65
 
66
+ - [Migrating from v1](#migrating-from-v1)
51
67
  - [API Reference](#api-reference)
52
68
  - [htmlToMarkdown()](#htmltomarkdown)
53
69
  - [streamHtmlToMarkdown()](#streamhtmltomarkdown)
@@ -67,7 +83,7 @@ externals: ['mdream']
67
83
  - [Tailwind](#tailwind-plugin)
68
84
  - [Filter](#filter-plugin)
69
85
  - [Extraction](#extraction-plugin)
70
- - [Hook-Based Plugins (JS Engine)](#hook-based-plugins-js-engine)
86
+ - [Plugins (JS Engine)](#plugins-js-engine)
71
87
  - [Plugin Hooks](#plugin-hooks)
72
88
  - [createPlugin()](#createplugin)
73
89
  - [Markdown Splitting (JS Engine)](#markdown-splitting-js-engine)
@@ -81,30 +97,172 @@ externals: ['mdream']
81
97
  - [Browser and Edge Usage](#browser-and-edge-usage)
82
98
  - [Edge / Cloudflare Workers](#edge--cloudflare-workers)
83
99
  - [Browser CDN (IIFE)](#browser-cdn-iife)
84
- - [Web Worker](#web-worker)
85
100
  - [llms.txt Generation](#llmstxt-generation)
86
101
  - [Related Packages](#related-packages)
87
102
 
103
+ ## Migrating from v1
104
+
105
+ Using a coding agent? Give it the [v1 to v2 migration prompt](https://github.com/harlan-zw/mdream/blob/main/MIGRATION_PROMPT.md).
106
+
107
+ Install `mdream` for the Rust engine.
108
+ This guide covers v2.
109
+ For an earlier release, read the README at its release tag.
110
+ Node and edge conversions still return a string synchronously.
111
+ The `format`, `clean`, and top-level plugin options remain available.
112
+ Custom hook plugins use `@mdream/js`; follow its [migration guide](../js/README.md#migrating-from-v1).
113
+ That guide also covers upgrades from an earlier v2 beta.
114
+
115
+ ### Unknown options
116
+
117
+ Every entry throws a `TypeError` for an option that it does not read.
118
+ In v1, mdream ignored an unknown option, so a typo gave no error.
119
+ The error names the fix.
120
+ An option set to `undefined` does not throw.
121
+
122
+ | Old option | v2 option |
123
+ |------------|-----------|
124
+ | `preset: 'minimal'` | `minimal: true` |
125
+ | `cleanUrls: true` | `clean: { urls: true }` |
126
+ | `plugins: { frontmatter: true }` | `frontmatter: true`, at the top level |
127
+ | `plugins: [myPlugin]` | `@mdream/js` with `plugins: [myPlugin]` |
128
+
129
+ `clean.blankLines` is removed. It had no effect in either engine, so delete it.
130
+
131
+ The `mdream` CLI exits with code 1 on an unknown flag, a flag without a value, or an unknown `--preset` value.
132
+ In v1, the CLI ignored them.
133
+
134
+ ### Browser imports and returns
135
+
136
+ In a browser bundle, import from `mdream/browser`.
137
+ Its `htmlToMarkdown` returns `Promise<string>`.
138
+ Remove `.markdown` and await the conversion:
139
+
140
+ ```diff
141
+ - import { htmlToMarkdown } from 'mdream'
142
+ - const markdown = (await htmlToMarkdown(html)).markdown
143
+ + import { htmlToMarkdown } from 'mdream/browser'
144
+ + const markdown = await htmlToMarkdown(html)
145
+ ```
146
+
147
+ In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
148
+ The import itself does not throw.
149
+ A module that runs on the server and the client can import `mdream` and call it only on the server.
150
+ The `mdream/browser` types need no TypeScript `customConditions` setting.
151
+
152
+ For a CDN script:
153
+
154
+ ```diff
155
+ - const markdown = window.mdream.htmlToMarkdown(html).markdown
156
+ + const markdown = await window.mdream.htmlToMarkdown(html)
157
+ ```
158
+
159
+ Use the CDN example inside an async function or a `<script type="module">` block.
160
+ See [Browser and Edge Usage](#browser-and-edge-usage) for edge runtimes and the CDN script.
161
+
162
+ ### Web Worker
163
+
164
+ v2 removes `mdream/worker`.
165
+ To convert off the main thread, write a module worker that imports `mdream/browser`:
166
+
167
+ ```ts
168
+ // md.worker.ts
169
+ import { htmlToMarkdown } from 'mdream/browser'
170
+
171
+ addEventListener('message', async (event: MessageEvent<string>) => {
172
+ postMessage(await htmlToMarkdown(event.data, { minimal: true }))
173
+ })
174
+ ```
175
+
176
+ ```ts
177
+ // main.ts
178
+ const worker = new Worker(new URL('./md.worker.ts', import.meta.url), { type: 'module' })
179
+ worker.onmessage = (event: MessageEvent<string>) => console.log(event.data)
180
+ worker.postMessage('<h1>Hello</h1>')
181
+ ```
182
+
183
+ Your bundler builds the worker and copies the WASM binary.
184
+ Functions cannot cross `postMessage`.
185
+ To use callbacks such as `frontmatter`, set them inside the worker.
186
+
187
+ ### Minimal filtering
188
+
189
+ With `minimal: true`, a custom `filter` adds to the preset's excludes.
190
+ In v1, it replaced them and could restore forms and navigation.
191
+ To keep those elements in v2, disable the preset filter.
192
+ To apply only your own excludes, use `minimal: false` and compose the options:
193
+
194
+ ```ts
195
+ import { htmlToMarkdown } from 'mdream'
196
+
197
+ htmlToMarkdown(html, { minimal: true, filter: false })
198
+
199
+ htmlToMarkdown(html, {
200
+ frontmatter: true,
201
+ isolateMain: true,
202
+ tailwind: true,
203
+ clean: true,
204
+ filter: { exclude: ['footer'] },
205
+ })
206
+ ```
207
+
208
+ ### Page titles
209
+
210
+ `<title>` text no longer appears in the body, in any output format.
211
+ To include it in Markdown metadata, enable `frontmatter: true`.
212
+ To read it separately, use `extraction: { title: element => console.log(element.textContent) }`.
213
+ Review saved output or snapshots that relied on a leading title line.
214
+ The converter also removes trailing spaces from link text before empty elements. See [#318](https://github.com/harlan-zw/mdream/pull/318).
215
+ Stream boundaries can change. Join the output before comparing snapshots.
216
+
217
+ ### Markdown splitting
218
+
219
+ The JS splitter changes its chunk text, line ranges, overlap, and first-chunk timing.
220
+ If you store chunks or embeddings, rebuild them from the original HTML.
221
+ See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for the upgrade steps.
222
+
223
+ ### Direct native binding access
224
+
225
+ The package's `exports` map defines its public API. It does not expose `napi/`.
226
+ If you loaded the native binding by file path, replace these removed exports:
227
+
228
+ | Removed native API | Public API |
229
+ |---|---|
230
+ | `splitMarkdown`, `htmlToMarkdownChunks` | `htmlToMarkdownSplitChunks` from `@mdream/js/splitter`, with the original HTML input |
231
+ | `htmlToMarkdownBytes` | Decode complete HTML bytes with `TextDecoder`, then call `htmlToMarkdown` from `mdream` |
232
+ | `MarkdownStream.processChunkBytes` | `streamHtmlToMarkdown` from `mdream`, which still accepts byte streams |
233
+
234
+ For direct Markdown splitting in Rust, use `mdream::splitter::split_markdown`.
235
+
88
236
  ## API Reference
89
237
 
90
238
  ### `htmlToMarkdown()`
91
239
 
92
- Converts a complete HTML string to Markdown, plain text, or HTML synchronously.
240
+ Converts a complete HTML string to Markdown, plain text, or HTML. It is synchronous in Node and on edge runtimes.
93
241
 
94
242
  **Rust engine** (`mdream`):
95
243
 
96
244
  ```ts
97
- import { htmlToMarkdown } from 'mdream'
245
+ import type { MdreamOptions } from 'mdream'
246
+
247
+ declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
98
248
 
99
- function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
249
+ // `mdream/browser` and the CDN script return Promise<string> instead.
100
250
  ```
101
251
 
102
- **JS engine** (`@mdream/js`):
252
+ **JS engine** (`@mdream/js`) uses a separate entry point for each format:
103
253
 
104
254
  ```ts
105
255
  import { htmlToMarkdown } from '@mdream/js'
256
+ import { htmlToSafeHtml } from '@mdream/js/html'
257
+ import { htmlToText } from '@mdream/js/text'
258
+ ```
106
259
 
107
- function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
260
+ ```ts
261
+ import type { MdreamOptions } from '@mdream/js'
262
+
263
+ declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
264
+ declare function htmlToText(html: string, options?: Partial<MdreamOptions>): string
265
+ declare function htmlToSafeHtml(html: string, options?: Partial<MdreamOptions>): string
108
266
  ```
109
267
 
110
268
  **Example:**
@@ -132,11 +290,13 @@ const html = htmlToMarkdown('<h1>Hello <strong>World</strong></h1>', {
132
290
 
133
291
  Converts an HTML `ReadableStream` to Markdown, plain text, or HTML incrementally. Returns an `AsyncIterable<string>` that yields output chunks as they are processed.
134
292
 
293
+ It takes the same options as `htmlToMarkdown()`. The `frontmatter` callback and the `extraction` handlers run once, after the stream ends and before the last chunk is yielded.
294
+
135
295
 
136
296
  ```ts
137
- import { streamHtmlToMarkdown } from 'mdream'
297
+ import type { MdreamOptions } from 'mdream'
138
298
 
139
- function streamHtmlToMarkdown(
299
+ declare function streamHtmlToMarkdown(
140
300
  htmlStream: ReadableStream<Uint8Array | string> | null,
141
301
  options?: Partial<MdreamOptions>,
142
302
  ): AsyncIterable<string>
@@ -160,23 +320,31 @@ for await (const chunk of streamHtmlToMarkdown(stream, {
160
320
 
161
321
  ## Engines
162
322
 
163
- Mdream includes two rendering engines, automatically selecting the best one for your environment:
323
+ Choose the Rust or JavaScript engine.
324
+ The `mdream` package selects its Rust binding by runtime.
325
+ Import `@mdream/js` to use the JavaScript engine:
164
326
 
165
327
  | Engine | Package | Plugins | Use case |
166
328
  |--------|---------|---------|----------|
167
329
  | **Rust** (NAPI) | `mdream` | Declarative config only | Node.js (default) |
168
- | **Rust** (WASM) | `mdream` | Declarative config only | Edge, browser |
169
- | **JavaScript** | `@mdream/js` | Hook-based + declarative | Custom plugins, splitter |
330
+ | **Rust** (WASM) | `mdream`, `mdream/browser` | Declarative config only | Edge, browser |
331
+ | **JavaScript** | `@mdream/js` | Explicit plugin arrays | Small bundles, custom plugins, splitter |
170
332
 
171
333
  ```ts
172
- // JavaScript engine (required for hook-based plugins)
334
+ // JavaScript engine for explicit plugins and smaller bundles
173
335
  import { htmlToMarkdown } from '@mdream/js'
174
336
 
175
337
  // Rust NAPI engine (auto-selected in Node.js)
176
- import { htmlToMarkdown } from 'mdream'
338
+ import { htmlToMarkdown as htmlToMarkdownRust } from 'mdream'
177
339
  ```
178
340
 
179
- Both engines accept the same declarative plugin configuration (`origin`, `minimal`, `frontmatter`, `isolateMain`, `tailwind`, `filter`, `extraction`, `tagOverrides`, `clean`). The JS engine additionally supports `hooks` for imperative plugin transforms.
341
+ The Rust engine takes declarative options: `minimal`, `format`, `frontmatter`, `isolateMain`, `tailwind`, `filter`, and `extraction`.
342
+ Pass them at the top level.
343
+
344
+ The JS engine takes an array of plugin instances in `options.plugins`.
345
+ Each output format has its own entry point.
346
+ For its minimal preset, use `withMinimalPreset()` from `@mdream/js/preset/minimal`.
347
+ If you pass Rust-only options to the JS engine, it throws a `TypeError` that names the fix.
180
348
 
181
349
  ## Options
182
350
 
@@ -216,7 +384,7 @@ interface MdreamOptions {
216
384
  tailwind?: boolean
217
385
 
218
386
  /** Filter elements by CSS selectors. Default when minimal: excludes form, nav, footer, etc. */
219
- filter?: { include?: string[], exclude?: string[], processChildren?: boolean }
387
+ filter?: false | { include?: string[], exclude?: string[], processChildren?: boolean }
220
388
 
221
389
  /** Extract elements matching CSS selectors during conversion. */
222
390
  extraction?: Record<string, (element: ExtractedElement) => void>
@@ -236,20 +404,23 @@ interface MdreamOptions {
236
404
  }
237
405
  ```
238
406
 
407
+ Any other key throws a `TypeError` that names the key and the fix.
408
+
239
409
  ### MdreamOptions (JS engine)
240
410
 
241
- The JS engine extends the shared `EngineOptions` with hook-based plugin support:
411
+ The JS engine uses explicit plugins. Each output format has its own entry point.
242
412
 
243
413
  ```ts
244
414
  interface MdreamOptions extends EngineOptions {
245
- /** Imperative hook-based transform plugins. JS engine only. */
246
- hooks?: TransformPlugin[]
415
+ /** Explicit plugins, applied in array order. */
416
+ plugins?: Plugin[]
247
417
  }
248
418
 
249
419
  interface EngineOptions {
250
420
  origin?: string
251
- clean?: boolean | CleanOptions
252
- plugins?: BuiltinPlugins
421
+ /** Cleanup rules from clean() in @mdream/js/clean */
422
+ clean?: Cleaner
423
+ tagOverrides?: Record<string, TagOverride | string>
253
424
 
254
425
  /**
255
426
  * Hard-wrap prose at this many characters, breaking on word boundaries.
@@ -258,26 +429,18 @@ interface EngineOptions {
258
429
  */
259
430
  wrapWidth?: number
260
431
 
261
- /** Output Markdown, plain text, or HTML. Default: 'markdown' */
262
- format?: 'markdown' | 'text' | 'html'
263
- }
264
-
265
- interface BuiltinPlugins {
266
- filter?: { include?: (string | number)[], exclude?: (string | number)[], processChildren?: boolean }
267
- frontmatter?: boolean | ((fm: Record<string, string>) => void) | FrontmatterConfig
268
- isolateMain?: boolean
269
- tailwind?: boolean
270
- extraction?: Record<string, (element: ExtractedElement) => void>
271
- tagOverrides?: Record<string, TagOverride | string>
272
432
  }
273
433
  ```
274
434
 
275
- Note: The JS engine uses `options.plugins.filter` while the Rust engine uses `options.filter` directly.
435
+ Use `@mdream/js/text` for plain text. Use `@mdream/js/html` for safe HTML.
436
+ Any other key throws a `TypeError` that names the key and the fix.
276
437
 
277
438
  ### CleanOptions
278
439
 
279
440
  Post-processing cleanup applied to the final Markdown output. All options default to `false` unless `clean: true` is set.
280
441
 
442
+ With `@mdream/js`, pass these rules to `clean()` from `@mdream/js/clean`, for example `clean: clean({ urls: true })`.
443
+
281
444
  ```ts
282
445
  interface CleanOptions {
283
446
  /** Strip tracking query parameters (utm_*, fbclid, gclid, etc.) from URLs */
@@ -286,8 +449,6 @@ interface CleanOptions {
286
449
  fragments?: boolean
287
450
  /** Strip links with meaningless hrefs (#, javascript:void(0)) to plain text */
288
451
  emptyLinks?: boolean
289
- /** Collapse 3+ consecutive blank lines to 2 */
290
- blankLines?: boolean
291
452
  /** Strip links where text equals URL: [https://x.com](https://x.com) becomes https://x.com */
292
453
  redundantLinks?: boolean
293
454
  /** Strip self-referencing heading anchors: ## [Title](#title) becomes ## Title */
@@ -302,6 +463,8 @@ interface CleanOptions {
302
463
  **Example:**
303
464
 
304
465
  ```ts
466
+ import { htmlToMarkdown } from 'mdream'
467
+
305
468
  const markdown = htmlToMarkdown(html, {
306
469
  clean: {
307
470
  urls: true,
@@ -336,7 +499,7 @@ Override how specific HTML tags are rendered in Markdown. String values act as a
336
499
  >
337
500
  > ```ts
338
501
  > htmlToMarkdown('<my-em>hi</my-em>', { tagOverrides: { 'my-em': 'em' } })
339
- > // → "_hi_"
502
+ > // → "*hi*"
340
503
  > ```
341
504
 
342
505
  ```ts
@@ -361,6 +524,8 @@ interface TagOverride {
361
524
  **Example:**
362
525
 
363
526
  ```ts
527
+ import { htmlToMarkdown } from 'mdream'
528
+
364
529
  const markdown = htmlToMarkdown(html, {
365
530
  tagOverrides: {
366
531
  // Treat <x-heading> like <h2>
@@ -375,12 +540,14 @@ const markdown = htmlToMarkdown(html, {
375
540
  })
376
541
  ```
377
542
 
378
- **Output is GitHub Flavored Markdown.** Mdream emits a fixed GFM dialect tuned for LLM input: ATX headings (`#`), fenced code blocks (` ``` `), `-` bullets, `_` emphasis, `**` strong, `---` horizontal rules, inline links. These are not configurable. For simple delimiter swaps you can use `tagOverrides`:
543
+ **Output is GitHub Flavored Markdown.** Mdream emits a fixed GFM dialect tuned for LLM input: ATX headings (`#`), fenced code blocks (` ``` `), `-` bullets, `*` emphasis, `**` strong, `---` horizontal rules, inline links. These are not configurable. For simple delimiter swaps you can use `tagOverrides`:
379
544
 
380
545
  ```ts
546
+ import { htmlToMarkdown } from 'mdream'
547
+
381
548
  htmlToMarkdown(html, {
382
549
  tagOverrides: {
383
- em: { enter: '*', exit: '*', isInline: true }, // _x_ → *x*
550
+ em: { enter: '_', exit: '_', isInline: true }, // *x* → _x_
384
551
  strong: { enter: '__', exit: '__', isInline: true }, // **x** → __x__
385
552
  hr: { enter: '* * *', exit: '' }, // --- → * * *
386
553
  },
@@ -393,10 +560,10 @@ Structural style differences (setext headings, indented code blocks, reference-s
393
560
 
394
561
  ```ts
395
562
  interface FilterOptions {
396
- /** CSS selectors, tag names, or TAG_* constants for elements to include (all others excluded) */
397
- include?: (string | number)[]
398
- /** CSS selectors, tag names, or TAG_* constants for elements to exclude */
399
- exclude?: (string | number)[]
563
+ /** CSS selectors or tag names for elements to include (all others excluded) */
564
+ include?: string[]
565
+ /** CSS selectors or tag names for elements to exclude */
566
+ exclude?: string[]
400
567
  /** Whether to also process children of matched elements. Default: true */
401
568
  processChildren?: boolean
402
569
  }
@@ -414,6 +581,18 @@ The `minimal` preset enables the following plugins together:
414
581
  - **filter**: Excludes `form`, `fieldset`, `object`, `embed`, `footer`, `aside`, `iframe`, `input`, `textarea`, `select`, `button`, `nav`
415
582
  - **clean**: All post-processing cleanup enabled
416
583
 
584
+ A `filter` passed with `minimal` adds to the preset's excludes. It does not replace them. To turn the preset's filter off, pass `filter: false`. In the JS engine, a `filterPlugin()` appended to `withMinimalPreset()` also adds to the excludes.
585
+
586
+ ```ts
587
+ import { htmlToMarkdown } from 'mdream'
588
+
589
+ // Excludes h1 as well as form, nav, footer, and the rest
590
+ htmlToMarkdown(html, { minimal: true, filter: { exclude: ['h1'] } })
591
+
592
+ // Keeps forms and nav
593
+ htmlToMarkdown(html, { minimal: true, filter: false })
594
+ ```
595
+
417
596
  **Rust engine:**
418
597
 
419
598
  ```ts
@@ -436,20 +615,21 @@ const markdown = htmlToMarkdown(html, withMinimalPreset({
436
615
  }))
437
616
  ```
438
617
 
439
- `withMinimalPreset()` returns an `EngineOptions` object with all plugin defaults applied. You can override individual plugins:
618
+ `withMinimalPreset()` returns explicit plugin defaults. You can append custom plugins. An appended `filterPlugin()` adds to the preset's excludes:
440
619
 
441
620
  ```ts
621
+ import { htmlToMarkdown } from '@mdream/js'
622
+ import { filterPlugin } from '@mdream/js/plugins'
623
+ import { withMinimalPreset } from '@mdream/js/preset/minimal'
624
+
442
625
  const markdown = htmlToMarkdown(html, withMinimalPreset({
443
- plugins: {
444
- frontmatter: false,
445
- filter: { exclude: ['nav'] },
446
- },
626
+ plugins: [filterPlugin({ exclude: ['.cookie-banner'] }), myPlugin],
447
627
  }))
448
628
  ```
449
629
 
450
630
  ## Built-in Plugins
451
631
 
452
- All built-in plugins work with both the Rust and JS engines through declarative configuration.
632
+ The Rust engine uses declarative options. The JS engine exports matching plugin factories.
453
633
 
454
634
  ### Frontmatter Plugin
455
635
 
@@ -458,6 +638,8 @@ Extracts metadata from the HTML `<head>` element and generates YAML frontmatter.
458
638
  **Extracted fields by default:** `title`, `description`, `keywords`, `author`, `date`, `og:title`, `og:description`, `twitter:title`, `twitter:description`.
459
639
 
460
640
  ```ts
641
+ import { htmlToMarkdown } from 'mdream'
642
+
461
643
  // Enable with defaults
462
644
  htmlToMarkdown(html, { frontmatter: true })
463
645
 
@@ -500,6 +682,8 @@ Isolates the main content area using the following priority:
500
682
  4. The `<head>` section is always passed through for other plugins (e.g., frontmatter)
501
683
 
502
684
  ```ts
685
+ import { htmlToMarkdown } from 'mdream'
686
+
503
687
  htmlToMarkdown(html, { isolateMain: true })
504
688
  ```
505
689
 
@@ -518,14 +702,18 @@ Converts Tailwind CSS utility classes to semantic Markdown formatting:
518
702
  Supports responsive breakpoint prefixes (`sm:`, `md:`, `lg:`, `xl:`, `2xl:`) with mobile-first resolution.
519
703
 
520
704
  ```ts
705
+ import { htmlToMarkdown } from 'mdream'
706
+
521
707
  htmlToMarkdown(html, { tailwind: true })
522
708
  ```
523
709
 
524
710
  ### Filter Plugin
525
711
 
526
- Filters HTML elements by CSS selectors, tag names, or `TAG_*` constants.
712
+ Filters HTML elements by CSS selectors or tag names.
527
713
 
528
714
  ```ts
715
+ import { htmlToMarkdown } from 'mdream'
716
+
529
717
  // Exclude navigation, sidebar, footer
530
718
  htmlToMarkdown(html, {
531
719
  filter: {
@@ -541,15 +729,14 @@ htmlToMarkdown(html, {
541
729
  })
542
730
  ```
543
731
 
544
- The JS engine also supports `TAG_*` integer constants for filtering:
732
+ The JS engine takes the same strings in `filterPlugin`. If an entry is not a string, `filterPlugin` throws a `TypeError`.
545
733
 
546
734
  ```ts
547
- import { TAG_FOOTER, TAG_NAV } from '@mdream/js'
735
+ import { htmlToMarkdown } from '@mdream/js'
736
+ import { filterPlugin } from '@mdream/js/plugins'
548
737
 
549
738
  htmlToMarkdown(html, {
550
- plugins: {
551
- filter: { exclude: [TAG_NAV, TAG_FOOTER] },
552
- },
739
+ plugins: [filterPlugin({ exclude: ['nav', 'footer'] })],
553
740
  })
554
741
  ```
555
742
 
@@ -560,6 +747,8 @@ Elements with `style="position: absolute"` or `style="position: fixed"` are also
560
747
  Extracts elements matching CSS selectors during conversion. Callbacks receive the matched element with its accumulated text content and attributes.
561
748
 
562
749
  ```ts
750
+ import { htmlToMarkdown } from 'mdream'
751
+
563
752
  htmlToMarkdown(html, {
564
753
  extraction: {
565
754
  'h2': (el) => {
@@ -586,9 +775,9 @@ interface ExtractedElement {
586
775
  }
587
776
  ```
588
777
 
589
- ## Hook-Based Plugins (JS Engine)
778
+ ## Plugins (JS Engine)
590
779
 
591
- The JS engine (`@mdream/js`) supports imperative hook-based plugins for custom transform logic. These allow you to intercept and modify the conversion pipeline at multiple stages.
780
+ The JS engine composes plugins in an explicit array.
592
781
 
593
782
  ```ts
594
783
  import { htmlToMarkdown } from '@mdream/js'
@@ -606,19 +795,30 @@ const myPlugin = createPlugin({
606
795
  },
607
796
  })
608
797
 
609
- const markdown = htmlToMarkdown(html, { hooks: [myPlugin] })
798
+ const markdown = htmlToMarkdown(html, { plugins: [myPlugin] })
610
799
  ```
611
800
 
612
801
  ### Plugin Hooks
613
802
 
803
+ Each hook gets a read-only `PluginState`. To keep data for a node, set it on `node.context`.
804
+
614
805
  ```ts
806
+ interface PluginState {
807
+ /** Options passed to the converter. */
808
+ readonly options: Readonly<EngineOptions>
809
+ /** Output that the running converter writes. */
810
+ readonly outputFormat: 'markdown' | 'text' | 'html'
811
+ /** Nesting depth of the current node. It is 0 before the first node. */
812
+ readonly depth: number
813
+ }
814
+
615
815
  interface TransformPlugin {
616
816
  /**
617
817
  * Called before any node processing. Return { skip: true } to skip the node.
618
818
  */
619
819
  beforeNodeProcess?: (
620
820
  event: NodeEvent,
621
- state: MdreamRuntimeState,
821
+ state: PluginState,
622
822
  ) => undefined | void | { skip: boolean }
623
823
 
624
824
  /**
@@ -627,7 +827,7 @@ interface TransformPlugin {
627
827
  */
628
828
  onNodeEnter?: (
629
829
  node: ElementNode,
630
- state: MdreamRuntimeState,
830
+ state: PluginState,
631
831
  ) => string | undefined | void
632
832
 
633
833
  /**
@@ -636,7 +836,7 @@ interface TransformPlugin {
636
836
  */
637
837
  onNodeExit?: (
638
838
  node: ElementNode,
639
- state: MdreamRuntimeState,
839
+ state: PluginState,
640
840
  ) => string | undefined | void
641
841
 
642
842
  /**
@@ -644,7 +844,7 @@ interface TransformPlugin {
644
844
  */
645
845
  processAttributes?: (
646
846
  node: ElementNode,
647
- state: MdreamRuntimeState,
847
+ state: PluginState,
648
848
  ) => void
649
849
 
650
850
  /**
@@ -653,8 +853,13 @@ interface TransformPlugin {
653
853
  */
654
854
  processTextNode?: (
655
855
  node: TextNode,
656
- state: MdreamRuntimeState,
856
+ state: PluginState,
657
857
  ) => { content: string, skip: boolean } | undefined
858
+
859
+ /**
860
+ * Called once after the whole document is converted, including for streams.
861
+ */
862
+ onDocumentEnd?: (state: PluginState) => void
658
863
  }
659
864
  ```
660
865
 
@@ -663,13 +868,18 @@ interface TransformPlugin {
663
868
  A typed identity function for creating plugins with full TypeScript inference:
664
869
 
665
870
  ```ts
871
+ import type { ElementNode } from '@mdream/js'
872
+ import { ELEMENT_NODE } from '@mdream/js'
666
873
  import { createPlugin } from '@mdream/js/plugins'
667
874
 
668
875
  const plugin = createPlugin({
669
876
  beforeNodeProcess({ node }) {
670
877
  // Skip all div elements with class "ad"
671
- if (node.type === 1 && node.attributes?.class?.includes('ad')) {
672
- return { skip: true }
878
+ if (node.type === ELEMENT_NODE) {
879
+ const element = node as ElementNode
880
+ if (element.name === 'div' && element.attributes.class?.split(/\s+/).includes('ad')) {
881
+ return { skip: true }
882
+ }
673
883
  }
674
884
  },
675
885
  })
@@ -682,7 +892,6 @@ The following plugin factory functions are available from `@mdream/js/plugins`:
682
892
  ```ts
683
893
  import {
684
894
  createPlugin,
685
- extractionCollectorPlugin,
686
895
  extractionPlugin,
687
896
  filterPlugin,
688
897
  frontmatterPlugin,
@@ -700,7 +909,6 @@ Available from `@mdream/js/splitter`.
700
909
  ### Basic Chunking
701
910
 
702
911
  ```ts
703
- import { TAG_H2 } from '@mdream/js'
704
912
  import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
705
913
 
706
914
  const html = `
@@ -712,7 +920,7 @@ const html = `
712
920
  `
713
921
 
714
922
  const chunks = htmlToMarkdownSplitChunks(html, {
715
- headersToSplitOn: [TAG_H2],
923
+ headersToSplitOn: [2],
716
924
  chunkSize: 1000,
717
925
  chunkOverlap: 200,
718
926
  stripHeaders: true,
@@ -728,7 +936,10 @@ chunks.forEach((chunk) => {
728
936
 
729
937
  ### Streaming Chunks (Memory Efficient)
730
938
 
731
- For large documents, use the generator version to process chunks one at a time:
939
+ The generator converts the whole document before yielding its first chunk.
940
+ It keeps the converted Markdown in memory, but avoids collecting the chunk array.
941
+ Stopping early skips the remaining chunk work. HTML conversion has already finished.
942
+ See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for migration details.
732
943
 
733
944
  ```ts
734
945
  import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'
@@ -749,14 +960,14 @@ interface SplitterOptions {
749
960
  // --- Structural splitting ---
750
961
 
751
962
  /**
752
- * Header tag IDs to split on (TAG_H1 through TAG_H6).
753
- * Default: [TAG_H2, TAG_H3, TAG_H4, TAG_H5, TAG_H6]
963
+ * Heading levels that start a new chunk, from 1 (<h1>) to 6 (<h6>).
964
+ * Another value throws a TypeError. Default: [2, 3, 4, 5, 6]
754
965
  */
755
- headersToSplitOn?: number[]
966
+ headersToSplitOn?: (1 | 2 | 3 | 4 | 5 | 6)[]
756
967
 
757
968
  // --- Size-based splitting ---
758
969
 
759
- /** Maximum chunk size in characters. Default: 1000 */
970
+ /** Target chunk size in characters. Complete code fences can exceed it. Default: 1000 */
760
971
  chunkSize?: number
761
972
 
762
973
  /** Overlap between chunks for context preservation. Default: 200 */
@@ -776,7 +987,7 @@ interface SplitterOptions {
776
987
  /** Split into individual lines. Default: false */
777
988
  returnEachLine?: boolean
778
989
 
779
- /** Keep separators in the split chunks. Default: false */
990
+ /** The splitter drops whitespace between chunks for either value. */
780
991
  keepSeparator?: boolean
781
992
 
782
993
  // --- Standard options ---
@@ -784,14 +995,14 @@ interface SplitterOptions {
784
995
  /** Base URL for resolving relative links/images */
785
996
  origin?: string
786
997
 
787
- /** Declarative built-in plugin config */
788
- plugins?: BuiltinPlugins
998
+ /** Explicit plugins, applied in array order */
999
+ plugins?: Plugin[]
789
1000
 
790
- /** Hook-based plugins (JS engine only) */
791
- hooks?: TransformPlugin[]
1001
+ /** Custom tag output or aliases */
1002
+ tagOverrides?: Record<string, TagOverride | string>
792
1003
 
793
- /** Post-processing cleanup */
794
- clean?: boolean | CleanOptions
1004
+ /** Cleanup rules from clean() in @mdream/js/clean */
1005
+ clean?: Cleaner
795
1006
  }
796
1007
  ```
797
1008
 
@@ -807,7 +1018,7 @@ interface MarkdownChunk {
807
1018
  headers?: Record<string, string> // { h1: "Title", h2: "Section" }
808
1019
  /** Code block language if chunk contains code */
809
1020
  code?: string
810
- /** Line number range in the original document */
1021
+ /** Inclusive, one-based line range in converted Markdown, excluding stripped headings */
811
1022
  loc?: {
812
1023
  lines: { from: number, to: number }
813
1024
  }
@@ -820,12 +1031,11 @@ interface MarkdownChunk {
820
1031
  Combine splitting with presets:
821
1032
 
822
1033
  ```ts
823
- import { TAG_H2 } from '@mdream/js'
824
1034
  import { withMinimalPreset } from '@mdream/js/preset/minimal'
825
1035
  import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
826
1036
 
827
1037
  const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
828
- headersToSplitOn: [TAG_H2],
1038
+ headersToSplitOn: [2],
829
1039
  chunkSize: 500,
830
1040
  origin: 'https://example.com',
831
1041
  }))
@@ -836,18 +1046,17 @@ const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
836
1046
  The `@mdream/js/negotiate` module provides HTTP content negotiation utilities for serving Markdown to LLM clients:
837
1047
 
838
1048
  ```ts
839
- import { parseAcceptHeader, shouldServeMarkdown } from '@mdream/js/negotiate'
840
-
841
- // Check if client prefers markdown
842
- const serveMarkdown = shouldServeMarkdown(
843
- request.headers.get('accept'),
844
- request.headers.get('sec-fetch-dest'),
845
- )
846
-
847
- if (serveMarkdown) {
848
- return new Response(markdown, {
849
- headers: { 'Content-Type': 'text/markdown' },
850
- })
1049
+ import { shouldServeMarkdown } from '@mdream/js/negotiate'
1050
+
1051
+ export function markdownResponse(request: Request, markdown: string): Response | undefined {
1052
+ if (shouldServeMarkdown(
1053
+ request.headers.get('accept') ?? undefined,
1054
+ request.headers.get('sec-fetch-dest') ?? undefined,
1055
+ )) {
1056
+ return new Response(markdown, {
1057
+ headers: { 'Content-Type': 'text/markdown' },
1058
+ })
1059
+ }
851
1060
  }
852
1061
  ```
853
1062
 
@@ -858,14 +1067,16 @@ if (serveMarkdown) {
858
1067
  If you only need to parse HTML into a DOM-like event stream without converting to Markdown, use `parseHtml` from the JS engine:
859
1068
 
860
1069
  ```ts
1070
+ import type { ElementNode } from '@mdream/js'
1071
+ import { ELEMENT_NODE, NodeEventEnter } from '@mdream/js'
861
1072
  import { parseHtml } from '@mdream/js/parse'
862
1073
 
863
1074
  const html = '<div><h1>Title</h1><p>Content</p></div>'
864
1075
  const { events, remainingHtml } = parseHtml(html)
865
1076
 
866
1077
  events.forEach((event) => {
867
- if (event.type === 0 && event.node.type === 1) { // Enter + Element
868
- console.log('Entering element:', event.node.name)
1078
+ if (event.type === NodeEventEnter && event.node.type === ELEMENT_NODE) {
1079
+ console.log('Entering element:', (event.node as ElementNode).name)
869
1080
  }
870
1081
  })
871
1082
  ```
@@ -924,9 +1135,27 @@ cat index.html \
924
1135
  | `-h`, `--help` | Display help information |
925
1136
 
926
1137
  The CLI reads HTML from stdin and writes Markdown, plain text, or HTML to stdout. It uses the streaming API internally.
1138
+ A value option also accepts the `--origin=<url>` form.
1139
+ If you pass an unknown flag, a flag without a value, or an unknown preset, the CLI writes one line to stderr and exits with code 1.
927
1140
 
928
1141
  ## Browser and Edge Usage
929
1142
 
1143
+ Every entry takes the same `MdreamOptions` and produces the same Markdown string.
1144
+ In Node and edge runtimes, `htmlToMarkdown` from `mdream` is synchronous.
1145
+ In a browser bundle, import from `mdream/browser`.
1146
+ Its `htmlToMarkdown` returns `Promise<string>`, because the WASM binary loads first.
1147
+ The CDN script also returns `Promise<string>`.
1148
+
1149
+ ```ts
1150
+ import { htmlToMarkdown } from 'mdream/browser'
1151
+
1152
+ const markdown = await htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
1153
+ ```
1154
+
1155
+ In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
1156
+ The `mdream` types always describe the synchronous API.
1157
+ To convert off the main thread, see [Web Worker](#web-worker).
1158
+
930
1159
  ### Edge / Cloudflare Workers
931
1160
 
932
1161
  For edge runtimes (Cloudflare Workers, Vercel Edge), `mdream` automatically selects the WASM build via export conditions (`workerd`, `edge-light`). Both `htmlToMarkdown` and `streamHtmlToMarkdown` are available:
@@ -944,93 +1173,59 @@ for await (const chunk of streamHtmlToMarkdown(response.body)) {
944
1173
  }
945
1174
  ```
946
1175
 
947
- Conversion errors are catchable. WASM has no unwinding, so an internal Rust panic aborts the call, but the message is preserved and re-thrown as a normal `Error` (with the raw WASM trap as `cause`), and the instance keeps working for later requests:
1176
+ You can catch conversion errors.
1177
+ An internal Rust panic aborts the call because WASM has no unwinding.
1178
+ The wrapper preserves the panic message in an `Error`, with the raw WASM trap as `cause`.
1179
+ The instance remains usable for later requests:
948
1180
 
949
1181
  ```ts
950
- try {
951
- return new Response(htmlToMarkdown(html))
952
- }
953
- catch (error) {
954
- // "mdream WASM panic, please report this at ...\npanicked at ..."
955
- console.error(error)
956
- return new Response('conversion failed', { status: 500 })
1182
+ import { htmlToMarkdown } from 'mdream'
1183
+
1184
+ export function convertResponse(html: string): Response {
1185
+ try {
1186
+ return new Response(htmlToMarkdown(html))
1187
+ }
1188
+ catch (error) {
1189
+ console.error(error)
1190
+ return new Response('Conversion failed', { status: 500 })
1191
+ }
957
1192
  }
958
1193
  ```
959
1194
 
960
- If your toolchain doesn't resolve the export conditions, the raw wasm-bindgen build (web target) is exposed at `mdream/wasm` for manual initialization:
1195
+ If your toolchain does not resolve export conditions, import the web-target wasm-bindgen build from `mdream/wasm`.
1196
+ Initialize it manually before converting.
1197
+ Its `htmlToMarkdown` takes the same `MdreamOptions` as `mdream`.
1198
+ Its `MarkdownStream` runs callbacks in `finish()`.
1199
+
1200
+ Your bundler must support `.wasm` imports and provide their TypeScript declarations.
961
1201
 
962
1202
  ```ts
963
1203
  import init, { htmlToMarkdown } from 'mdream/wasm'
964
1204
  import wasmModule from 'mdream/wasm/mdream_edge_bg.wasm'
965
1205
 
966
1206
  await init({ module_or_path: wasmModule })
967
- const markdown = htmlToMarkdown('<h1>Hello</h1>', {})
968
- ```
969
-
970
- You can also import the edge entry point directly:
971
-
972
- ```ts
973
- import { htmlToMarkdown } from 'mdream/worker'
974
- ```
975
-
976
- The `mdream/worker` entry provides an async API since WASM must be initialized first:
977
-
978
- ```ts
979
- import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
980
-
981
- // Initialize once with the WASM URL
982
- await initWorker('https://cdn.example.com/mdream_edge_bg.wasm')
983
-
984
- // Convert (returns Promise<string>)
985
- const markdown = await htmlToMarkdown('<h1>Hello</h1>')
986
-
987
- // Clean up when done
988
- terminateWorker()
1207
+ const markdown = htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
989
1208
  ```
990
1209
 
991
1210
  ### Browser CDN (IIFE)
992
1211
 
993
- Use mdream directly via CDN with no build step. Call `init()` once to load the WASM binary, then use `htmlToMarkdown()` synchronously.
1212
+ Use mdream from a CDN without a build step.
1213
+ The script includes the WASM binary and initializes it on load.
1214
+ Call `window.mdream.htmlToMarkdown()` with the same `MdreamOptions` as `mdream`.
1215
+ Await its `Promise<string>`, as with `mdream/browser`.
994
1216
 
995
1217
  ```html
996
1218
  <script src="https://unpkg.com/mdream/dist/iife.js"></script>
997
1219
  <script>
998
- await window.mdream.init()
999
- const markdown = window.mdream.htmlToMarkdown('<h1>Hello</h1><p>World</p>')
1000
- console.log(markdown) // # Hello\n\nWorld
1220
+ window.mdream.htmlToMarkdown('<h1>Hello</h1><p>World</p>')
1221
+ .then(markdown => console.log(markdown)) // # Hello\n\nWorld
1001
1222
  </script>
1002
1223
  ```
1003
1224
 
1004
- You can pass a custom WASM URL or `ArrayBuffer` to `init()`:
1005
-
1006
- ```js
1007
- // Custom URL
1008
- await window.mdream.init('https://cdn.example.com/mdream_edge_bg.wasm')
1009
-
1010
- // Pre-loaded ArrayBuffer
1011
- const wasmBytes = await fetch('/wasm/mdream_edge_bg.wasm').then(r => r.arrayBuffer())
1012
- await window.mdream.init(wasmBytes)
1013
- ```
1014
-
1015
1225
  **CDN Options:**
1016
1226
  - **unpkg**: `https://unpkg.com/mdream/dist/iife.js`
1017
1227
  - **jsDelivr**: `https://cdn.jsdelivr.net/npm/mdream/dist/iife.js`
1018
1228
 
1019
- ### Web Worker
1020
-
1021
- For browser environments, `mdream/worker` runs conversions off the main thread using a Web Worker:
1022
-
1023
- ```ts
1024
- import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
1025
-
1026
- await initWorker('/path/to/mdream_edge_bg.wasm')
1027
-
1028
- const markdown = await htmlToMarkdown('<h1>Hello</h1>')
1029
-
1030
- // Clean up
1031
- terminateWorker()
1032
- ```
1033
-
1034
1229
  ## Content Extraction with Readability
1035
1230
 
1036
1231
  For advanced content extraction (article detection, boilerplate removal), use [@mozilla/readability](https://github.com/mozilla/readability) before mdream:
@@ -1043,7 +1238,7 @@ import { htmlToMarkdown } from 'mdream'
1043
1238
  const dom = new JSDOM(html, { url: 'https://example.com' })
1044
1239
  const article = new Readability(dom.window.document).parse()
1045
1240
 
1046
- if (article) {
1241
+ if (article?.content) {
1047
1242
  const markdown = htmlToMarkdown(article.content)
1048
1243
  // article.title, article.excerpt, article.byline also available
1049
1244
  }
@@ -1051,10 +1246,10 @@ if (article) {
1051
1246
 
1052
1247
  ## llms.txt Generation
1053
1248
 
1054
- For llms.txt artifact generation, use the separate `@mdream/llms-txt` package. It accepts pre-converted Markdown and generates `llms.txt` and `llms-full.txt` artifacts.
1249
+ For llms.txt artifact generation, use `@mdream/js/llms-txt` from the `@mdream/js` package. It accepts pre-converted Markdown and generates `llms.txt` and `llms-full.txt` artifacts.
1055
1250
 
1056
1251
  ```ts
1057
- import { generateLlmsTxtArtifacts } from '@mdream/llms-txt'
1252
+ import { generateLlmsTxtArtifacts } from '@mdream/js/llms-txt'
1058
1253
  import { htmlToMarkdown } from 'mdream'
1059
1254
 
1060
1255
  const result = await generateLlmsTxtArtifacts({
@@ -1076,8 +1271,7 @@ console.log(result.llmsFullTxt) // llms-full.txt content
1076
1271
  | Package | Description |
1077
1272
  |---------|-------------|
1078
1273
  | [`mdream`](https://npmjs.com/package/mdream) | Core HTML to Markdown converter (Rust + WASM engine) |
1079
- | [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with hook-based plugins and splitter |
1080
- | [`@mdream/llms-txt`](https://github.com/harlan-zw/mdream/tree/main/packages/llms-txt) | Engine-agnostic llms.txt artifact generation |
1274
+ | [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with tree-shakable formats, plugins, splitter, and llms.txt generation (`@mdream/js/llms-txt`) |
1081
1275
  | [`@mdream/crawl`](https://github.com/harlan-zw/mdream/tree/main/packages/crawl) | Site-wide crawler for llms.txt generation |
1082
1276
  | [`@mdream/vite`](https://github.com/harlan-zw/mdream/tree/main/packages/vite) | Vite plugin integration |
1083
1277
  | [`@mdream/nuxt`](https://github.com/harlan-zw/mdream/tree/main/packages/nuxt) | Nuxt module integration |