mdream 2.0.0-beta.1 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -11,6 +11,11 @@
11
11
  </picture>
12
12
  </a>
13
13
 
14
+ Powering Cloudflare Browser Run's [/markdown](https://developers.cloudflare.com/browser-run/quick-actions/markdown-endpoint/) and [/crawl](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/) endpoints.
15
+
16
+ > [!TIP]
17
+ > 🎉 **Upgrading to Mdream v2?** Follow [Migrating from v1](#migrating-from-v1).
18
+
14
19
  ## Installation
15
20
 
16
21
  ```bash
@@ -27,7 +32,7 @@ yarn add mdream
27
32
  > [!TIP]
28
33
  > Using an AI agent? Get the mdream Skill on [skilld.dev/gh/harlan-zw/mdream](https://skilld.dev/gh/harlan-zw/mdream).
29
34
 
30
- For the JavaScript-only engine (hook-based plugins, splitter, pure HTML parser):
35
+ For the JavaScript-only engine with explicit plugins, formats, splitter, and parser:
31
36
 
32
37
  ```bash
33
38
  pnpm add @mdream/js
@@ -58,6 +63,7 @@ externals: ['mdream']
58
63
 
59
64
  ## Table of Contents
60
65
 
66
+ - [Migrating from v1](#migrating-from-v1)
61
67
  - [API Reference](#api-reference)
62
68
  - [htmlToMarkdown()](#htmltomarkdown)
63
69
  - [streamHtmlToMarkdown()](#streamhtmltomarkdown)
@@ -77,7 +83,7 @@ externals: ['mdream']
77
83
  - [Tailwind](#tailwind-plugin)
78
84
  - [Filter](#filter-plugin)
79
85
  - [Extraction](#extraction-plugin)
80
- - [Hook-Based Plugins (JS Engine)](#hook-based-plugins-js-engine)
86
+ - [Plugins (JS Engine)](#plugins-js-engine)
81
87
  - [Plugin Hooks](#plugin-hooks)
82
88
  - [createPlugin()](#createplugin)
83
89
  - [Markdown Splitting (JS Engine)](#markdown-splitting-js-engine)
@@ -91,10 +97,142 @@ externals: ['mdream']
91
97
  - [Browser and Edge Usage](#browser-and-edge-usage)
92
98
  - [Edge / Cloudflare Workers](#edge--cloudflare-workers)
93
99
  - [Browser CDN (IIFE)](#browser-cdn-iife)
94
- - [Web Worker](#web-worker)
95
100
  - [llms.txt Generation](#llmstxt-generation)
96
101
  - [Related Packages](#related-packages)
97
102
 
103
+ ## Migrating from v1
104
+
105
+ Using a coding agent? Give it the [v1 to v2 migration prompt](https://github.com/harlan-zw/mdream/blob/main/MIGRATION_PROMPT.md).
106
+
107
+ Install `mdream` for the Rust engine.
108
+ This guide covers v2.
109
+ For an earlier release, read the README at its release tag.
110
+ Node and edge conversions still return a string synchronously.
111
+ The `format`, `clean`, and top-level plugin options remain available.
112
+ Custom hook plugins use `@mdream/js`; follow its [migration guide](../js/README.md#migrating-from-v1).
113
+ That guide also covers upgrades from an earlier v2 beta.
114
+
115
+ ### Unknown options
116
+
117
+ Every entry throws a `TypeError` for an option that it does not read.
118
+ In v1, mdream ignored an unknown option, so a typo gave no error.
119
+ The error names the fix.
120
+ An option set to `undefined` does not throw.
121
+
122
+ | Old option | v2 option |
123
+ |------------|-----------|
124
+ | `preset: 'minimal'` | `minimal: true` |
125
+ | `cleanUrls: true` | `clean: { urls: true }` |
126
+ | `plugins: { frontmatter: true }` | `frontmatter: true`, at the top level |
127
+ | `plugins: [myPlugin]` | `@mdream/js` with `plugins: [myPlugin]` |
128
+
129
+ `clean.blankLines` is removed. It had no effect in either engine, so delete it.
130
+
131
+ The `mdream` CLI exits with code 1 on an unknown flag, a flag without a value, or an unknown `--preset` value.
132
+ In v1, the CLI ignored them.
133
+
134
+ ### Browser imports and returns
135
+
136
+ In a browser bundle, import from `mdream/browser`.
137
+ Its `htmlToMarkdown` returns `Promise<string>`.
138
+ Remove `.markdown` and await the conversion:
139
+
140
+ ```diff
141
+ - import { htmlToMarkdown } from 'mdream'
142
+ - const markdown = (await htmlToMarkdown(html)).markdown
143
+ + import { htmlToMarkdown } from 'mdream/browser'
144
+ + const markdown = await htmlToMarkdown(html)
145
+ ```
146
+
147
+ In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
148
+ The import itself does not throw.
149
+ A module that runs on the server and the client can import `mdream` and call it only on the server.
150
+ The `mdream/browser` types need no TypeScript `customConditions` setting.
151
+
152
+ For a CDN script:
153
+
154
+ ```diff
155
+ - const markdown = window.mdream.htmlToMarkdown(html).markdown
156
+ + const markdown = await window.mdream.htmlToMarkdown(html)
157
+ ```
158
+
159
+ Use the CDN example inside an async function or a `<script type="module">` block.
160
+ See [Browser and Edge Usage](#browser-and-edge-usage) for edge runtimes and the CDN script.
161
+
162
+ ### Web Worker
163
+
164
+ v2 removes `mdream/worker`.
165
+ To convert off the main thread, write a module worker that imports `mdream/browser`:
166
+
167
+ ```ts
168
+ // md.worker.ts
169
+ import { htmlToMarkdown } from 'mdream/browser'
170
+
171
+ addEventListener('message', async (event: MessageEvent<string>) => {
172
+ postMessage(await htmlToMarkdown(event.data, { minimal: true }))
173
+ })
174
+ ```
175
+
176
+ ```ts
177
+ // main.ts
178
+ const worker = new Worker(new URL('./md.worker.ts', import.meta.url), { type: 'module' })
179
+ worker.onmessage = (event: MessageEvent<string>) => console.log(event.data)
180
+ worker.postMessage('<h1>Hello</h1>')
181
+ ```
182
+
183
+ Your bundler builds the worker and copies the WASM binary.
184
+ Functions cannot cross `postMessage`.
185
+ To use callbacks such as `frontmatter`, set them inside the worker.
186
+
187
+ ### Minimal filtering
188
+
189
+ With `minimal: true`, a custom `filter` adds to the preset's excludes.
190
+ In v1, it replaced them and could restore forms and navigation.
191
+ To keep those elements in v2, disable the preset filter.
192
+ To apply only your own excludes, use `minimal: false` and compose the options:
193
+
194
+ ```ts
195
+ import { htmlToMarkdown } from 'mdream'
196
+
197
+ htmlToMarkdown(html, { minimal: true, filter: false })
198
+
199
+ htmlToMarkdown(html, {
200
+ frontmatter: true,
201
+ isolateMain: true,
202
+ tailwind: true,
203
+ clean: true,
204
+ filter: { exclude: ['footer'] },
205
+ })
206
+ ```
207
+
208
+ ### Page titles
209
+
210
+ `<title>` text no longer appears in the body, in any output format.
211
+ To include it in Markdown metadata, enable `frontmatter: true`.
212
+ To read it separately, use `extraction: { title: element => console.log(element.textContent) }`.
213
+ Review saved output or snapshots that relied on a leading title line.
214
+ The converter also removes trailing spaces from link text before empty elements. See [#318](https://github.com/harlan-zw/mdream/pull/318).
215
+ Stream boundaries can change. Join the output before comparing snapshots.
216
+
217
+ ### Markdown splitting
218
+
219
+ The JS splitter changes its chunk text, line ranges, overlap, and first-chunk timing.
220
+ If you store chunks or embeddings, rebuild them from the original HTML.
221
+ See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for the upgrade steps.
222
+
223
+ ### Direct native binding access
224
+
225
+ The package's `exports` map defines its public API. It does not expose `napi/`.
226
+ If you loaded the native binding by file path, replace these removed exports:
227
+
228
+ | Removed native API | Public API |
229
+ |---|---|
230
+ | `splitMarkdown`, `htmlToMarkdownChunks` | `htmlToMarkdownSplitChunks` from `@mdream/js/splitter`, with the original HTML input |
231
+ | `htmlToMarkdownBytes` | Decode complete HTML bytes with `TextDecoder`, then call `htmlToMarkdown` from `mdream` |
232
+ | `MarkdownStream.processChunkBytes` | `streamHtmlToMarkdown` from `mdream`, which still accepts byte streams |
233
+
234
+ For direct Markdown splitting in Rust, use `mdream::splitter::split_markdown`.
235
+
98
236
  ## API Reference
99
237
 
100
238
  ### `htmlToMarkdown()`
@@ -104,20 +242,27 @@ Converts a complete HTML string to Markdown, plain text, or HTML. It is synchron
104
242
  **Rust engine** (`mdream`):
105
243
 
106
244
  ```ts
107
- import { htmlToMarkdown } from 'mdream'
245
+ import type { MdreamOptions } from 'mdream'
108
246
 
109
- function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
247
+ declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
110
248
 
111
- // Browser bundles and the CDN script load the WASM binary first
112
- function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): Promise<string>
249
+ // `mdream/browser` and the CDN script return Promise<string> instead.
113
250
  ```
114
251
 
115
- **JS engine** (`@mdream/js`):
252
+ **JS engine** (`@mdream/js`) uses a separate entry point for each format:
116
253
 
117
254
  ```ts
118
255
  import { htmlToMarkdown } from '@mdream/js'
256
+ import { htmlToSafeHtml } from '@mdream/js/html'
257
+ import { htmlToText } from '@mdream/js/text'
258
+ ```
119
259
 
120
- function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
260
+ ```ts
261
+ import type { MdreamOptions } from '@mdream/js'
262
+
263
+ declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
264
+ declare function htmlToText(html: string, options?: Partial<MdreamOptions>): string
265
+ declare function htmlToSafeHtml(html: string, options?: Partial<MdreamOptions>): string
121
266
  ```
122
267
 
123
268
  **Example:**
@@ -149,9 +294,9 @@ It takes the same options as `htmlToMarkdown()`. The `frontmatter` callback and
149
294
 
150
295
 
151
296
  ```ts
152
- import { streamHtmlToMarkdown } from 'mdream'
297
+ import type { MdreamOptions } from 'mdream'
153
298
 
154
- function streamHtmlToMarkdown(
299
+ declare function streamHtmlToMarkdown(
155
300
  htmlStream: ReadableStream<Uint8Array | string> | null,
156
301
  options?: Partial<MdreamOptions>,
157
302
  ): AsyncIterable<string>
@@ -175,23 +320,31 @@ for await (const chunk of streamHtmlToMarkdown(stream, {
175
320
 
176
321
  ## Engines
177
322
 
178
- Mdream includes two rendering engines, automatically selecting the best one for your environment:
323
+ Choose the Rust or JavaScript engine.
324
+ The `mdream` package selects its Rust binding by runtime.
325
+ Import `@mdream/js` to use the JavaScript engine:
179
326
 
180
327
  | Engine | Package | Plugins | Use case |
181
328
  |--------|---------|---------|----------|
182
329
  | **Rust** (NAPI) | `mdream` | Declarative config only | Node.js (default) |
183
- | **Rust** (WASM) | `mdream` | Declarative config only | Edge, browser |
184
- | **JavaScript** | `@mdream/js` | Hook-based + declarative | Custom plugins, splitter |
330
+ | **Rust** (WASM) | `mdream`, `mdream/browser` | Declarative config only | Edge, browser |
331
+ | **JavaScript** | `@mdream/js` | Explicit plugin arrays | Small bundles, custom plugins, splitter |
185
332
 
186
333
  ```ts
187
- // JavaScript engine (required for hook-based plugins)
334
+ // JavaScript engine for explicit plugins and smaller bundles
188
335
  import { htmlToMarkdown } from '@mdream/js'
189
336
 
190
337
  // Rust NAPI engine (auto-selected in Node.js)
191
- import { htmlToMarkdown } from 'mdream'
338
+ import { htmlToMarkdown as htmlToMarkdownRust } from 'mdream'
192
339
  ```
193
340
 
194
- Both engines support the same declarative plugins, but the option shape differs. The Rust engine takes `minimal`, `frontmatter`, `isolateMain`, `tailwind`, `filter`, `extraction`, and `tagOverrides` at the top level. The JS engine reads the plugins from `options.plugins` and has no `minimal` option: use `withMinimalPreset()` from `@mdream/js/preset/minimal`. If the JS engine gets one of these options at the top level, it throws a `TypeError` that names the fix. The JS engine also supports `hooks` for imperative plugin transforms.
341
+ The Rust engine takes declarative options: `minimal`, `format`, `frontmatter`, `isolateMain`, `tailwind`, `filter`, and `extraction`.
342
+ Pass them at the top level.
343
+
344
+ The JS engine takes an array of plugin instances in `options.plugins`.
345
+ Each output format has its own entry point.
346
+ For its minimal preset, use `withMinimalPreset()` from `@mdream/js/preset/minimal`.
347
+ If you pass Rust-only options to the JS engine, it throws a `TypeError` that names the fix.
195
348
 
196
349
  ## Options
197
350
 
@@ -251,20 +404,23 @@ interface MdreamOptions {
251
404
  }
252
405
  ```
253
406
 
407
+ Any other key throws a `TypeError` that names the key and the fix.
408
+
254
409
  ### MdreamOptions (JS engine)
255
410
 
256
- The JS engine extends the shared `EngineOptions` with hook-based plugin support:
411
+ The JS engine uses explicit plugins. Each output format has its own entry point.
257
412
 
258
413
  ```ts
259
414
  interface MdreamOptions extends EngineOptions {
260
- /** Imperative hook-based transform plugins. JS engine only. */
261
- hooks?: TransformPlugin[]
415
+ /** Explicit plugins, applied in array order. */
416
+ plugins?: Plugin[]
262
417
  }
263
418
 
264
419
  interface EngineOptions {
265
420
  origin?: string
266
- clean?: boolean | CleanOptions
267
- plugins?: BuiltinPlugins
421
+ /** Cleanup rules from clean() in @mdream/js/clean */
422
+ clean?: Cleaner
423
+ tagOverrides?: Record<string, TagOverride | string>
268
424
 
269
425
  /**
270
426
  * Hard-wrap prose at this many characters, breaking on word boundaries.
@@ -273,26 +429,18 @@ interface EngineOptions {
273
429
  */
274
430
  wrapWidth?: number
275
431
 
276
- /** Output Markdown, plain text, or HTML. Default: 'markdown' */
277
- format?: 'markdown' | 'text' | 'html'
278
- }
279
-
280
- interface BuiltinPlugins {
281
- filter?: false | { include?: (string | number)[], exclude?: (string | number)[], processChildren?: boolean }
282
- frontmatter?: boolean | ((fm: Record<string, string>) => void) | FrontmatterConfig
283
- isolateMain?: boolean
284
- tailwind?: boolean
285
- extraction?: Record<string, (element: ExtractedElement) => void>
286
- tagOverrides?: Record<string, TagOverride | string>
287
432
  }
288
433
  ```
289
434
 
290
- Note: The JS engine uses `options.plugins.filter` while the Rust engine uses `options.filter` directly.
435
+ Use `@mdream/js/text` for plain text. Use `@mdream/js/html` for safe HTML.
436
+ Any other key throws a `TypeError` that names the key and the fix.
291
437
 
292
438
  ### CleanOptions
293
439
 
294
440
  Post-processing cleanup applied to the final Markdown output. All options default to `false` unless `clean: true` is set.
295
441
 
442
+ With `@mdream/js`, pass these rules to `clean()` from `@mdream/js/clean`, for example `clean: clean({ urls: true })`.
443
+
296
444
  ```ts
297
445
  interface CleanOptions {
298
446
  /** Strip tracking query parameters (utm_*, fbclid, gclid, etc.) from URLs */
@@ -301,8 +449,6 @@ interface CleanOptions {
301
449
  fragments?: boolean
302
450
  /** Strip links with meaningless hrefs (#, javascript:void(0)) to plain text */
303
451
  emptyLinks?: boolean
304
- /** Collapse 3+ consecutive blank lines to 2 */
305
- blankLines?: boolean
306
452
  /** Strip links where text equals URL: [https://x.com](https://x.com) becomes https://x.com */
307
453
  redundantLinks?: boolean
308
454
  /** Strip self-referencing heading anchors: ## [Title](#title) becomes ## Title */
@@ -317,6 +463,8 @@ interface CleanOptions {
317
463
  **Example:**
318
464
 
319
465
  ```ts
466
+ import { htmlToMarkdown } from 'mdream'
467
+
320
468
  const markdown = htmlToMarkdown(html, {
321
469
  clean: {
322
470
  urls: true,
@@ -376,6 +524,8 @@ interface TagOverride {
376
524
  **Example:**
377
525
 
378
526
  ```ts
527
+ import { htmlToMarkdown } from 'mdream'
528
+
379
529
  const markdown = htmlToMarkdown(html, {
380
530
  tagOverrides: {
381
531
  // Treat <x-heading> like <h2>
@@ -393,6 +543,8 @@ const markdown = htmlToMarkdown(html, {
393
543
  **Output is GitHub Flavored Markdown.** Mdream emits a fixed GFM dialect tuned for LLM input: ATX headings (`#`), fenced code blocks (` ``` `), `-` bullets, `*` emphasis, `**` strong, `---` horizontal rules, inline links. These are not configurable. For simple delimiter swaps you can use `tagOverrides`:
394
544
 
395
545
  ```ts
546
+ import { htmlToMarkdown } from 'mdream'
547
+
396
548
  htmlToMarkdown(html, {
397
549
  tagOverrides: {
398
550
  em: { enter: '_', exit: '_', isInline: true }, // *x* → _x_
@@ -408,10 +560,10 @@ Structural style differences (setext headings, indented code blocks, reference-s
408
560
 
409
561
  ```ts
410
562
  interface FilterOptions {
411
- /** CSS selectors, tag names, or TAG_* constants for elements to include (all others excluded) */
412
- include?: (string | number)[]
413
- /** CSS selectors, tag names, or TAG_* constants for elements to exclude */
414
- exclude?: (string | number)[]
563
+ /** CSS selectors or tag names for elements to include (all others excluded) */
564
+ include?: string[]
565
+ /** CSS selectors or tag names for elements to exclude */
566
+ exclude?: string[]
415
567
  /** Whether to also process children of matched elements. Default: true */
416
568
  processChildren?: boolean
417
569
  }
@@ -429,9 +581,11 @@ The `minimal` preset enables the following plugins together:
429
581
  - **filter**: Excludes `form`, `fieldset`, `object`, `embed`, `footer`, `aside`, `iframe`, `input`, `textarea`, `select`, `button`, `nav`
430
582
  - **clean**: All post-processing cleanup enabled
431
583
 
432
- A `filter` passed with `minimal` adds to the preset's excludes. It does not replace them. To turn the preset's filter off, pass `filter: false`. `withMinimalPreset()` in the JS engine works the same way.
584
+ A `filter` passed with `minimal` adds to the preset's excludes. It does not replace them. To turn the preset's filter off, pass `filter: false`. In the JS engine, a `filterPlugin()` appended to `withMinimalPreset()` also adds to the excludes.
433
585
 
434
586
  ```ts
587
+ import { htmlToMarkdown } from 'mdream'
588
+
435
589
  // Excludes h1 as well as form, nav, footer, and the rest
436
590
  htmlToMarkdown(html, { minimal: true, filter: { exclude: ['h1'] } })
437
591
 
@@ -461,20 +615,21 @@ const markdown = htmlToMarkdown(html, withMinimalPreset({
461
615
  }))
462
616
  ```
463
617
 
464
- `withMinimalPreset()` returns an `EngineOptions` object with all plugin defaults applied. You can override individual plugins. A `filter` adds to the preset's excludes:
618
+ `withMinimalPreset()` returns explicit plugin defaults. You can append custom plugins. An appended `filterPlugin()` adds to the preset's excludes:
465
619
 
466
620
  ```ts
621
+ import { htmlToMarkdown } from '@mdream/js'
622
+ import { filterPlugin } from '@mdream/js/plugins'
623
+ import { withMinimalPreset } from '@mdream/js/preset/minimal'
624
+
467
625
  const markdown = htmlToMarkdown(html, withMinimalPreset({
468
- plugins: {
469
- frontmatter: false,
470
- filter: { exclude: ['.cookie-banner'] },
471
- },
626
+ plugins: [filterPlugin({ exclude: ['.cookie-banner'] }), myPlugin],
472
627
  }))
473
628
  ```
474
629
 
475
630
  ## Built-in Plugins
476
631
 
477
- All built-in plugins work with both the Rust and JS engines through declarative configuration.
632
+ The Rust engine uses declarative options. The JS engine exports matching plugin factories.
478
633
 
479
634
  ### Frontmatter Plugin
480
635
 
@@ -483,6 +638,8 @@ Extracts metadata from the HTML `<head>` element and generates YAML frontmatter.
483
638
  **Extracted fields by default:** `title`, `description`, `keywords`, `author`, `date`, `og:title`, `og:description`, `twitter:title`, `twitter:description`.
484
639
 
485
640
  ```ts
641
+ import { htmlToMarkdown } from 'mdream'
642
+
486
643
  // Enable with defaults
487
644
  htmlToMarkdown(html, { frontmatter: true })
488
645
 
@@ -525,6 +682,8 @@ Isolates the main content area using the following priority:
525
682
  4. The `<head>` section is always passed through for other plugins (e.g., frontmatter)
526
683
 
527
684
  ```ts
685
+ import { htmlToMarkdown } from 'mdream'
686
+
528
687
  htmlToMarkdown(html, { isolateMain: true })
529
688
  ```
530
689
 
@@ -543,14 +702,18 @@ Converts Tailwind CSS utility classes to semantic Markdown formatting:
543
702
  Supports responsive breakpoint prefixes (`sm:`, `md:`, `lg:`, `xl:`, `2xl:`) with mobile-first resolution.
544
703
 
545
704
  ```ts
705
+ import { htmlToMarkdown } from 'mdream'
706
+
546
707
  htmlToMarkdown(html, { tailwind: true })
547
708
  ```
548
709
 
549
710
  ### Filter Plugin
550
711
 
551
- Filters HTML elements by CSS selectors, tag names, or `TAG_*` constants.
712
+ Filters HTML elements by CSS selectors or tag names.
552
713
 
553
714
  ```ts
715
+ import { htmlToMarkdown } from 'mdream'
716
+
554
717
  // Exclude navigation, sidebar, footer
555
718
  htmlToMarkdown(html, {
556
719
  filter: {
@@ -566,15 +729,14 @@ htmlToMarkdown(html, {
566
729
  })
567
730
  ```
568
731
 
569
- The JS engine also supports `TAG_*` integer constants for filtering:
732
+ The JS engine takes the same strings in `filterPlugin`. If an entry is not a string, `filterPlugin` throws a `TypeError`.
570
733
 
571
734
  ```ts
572
- import { TAG_FOOTER, TAG_NAV } from '@mdream/js'
735
+ import { htmlToMarkdown } from '@mdream/js'
736
+ import { filterPlugin } from '@mdream/js/plugins'
573
737
 
574
738
  htmlToMarkdown(html, {
575
- plugins: {
576
- filter: { exclude: [TAG_NAV, TAG_FOOTER] },
577
- },
739
+ plugins: [filterPlugin({ exclude: ['nav', 'footer'] })],
578
740
  })
579
741
  ```
580
742
 
@@ -585,6 +747,8 @@ Elements with `style="position: absolute"` or `style="position: fixed"` are also
585
747
  Extracts elements matching CSS selectors during conversion. Callbacks receive the matched element with its accumulated text content and attributes.
586
748
 
587
749
  ```ts
750
+ import { htmlToMarkdown } from 'mdream'
751
+
588
752
  htmlToMarkdown(html, {
589
753
  extraction: {
590
754
  'h2': (el) => {
@@ -611,9 +775,9 @@ interface ExtractedElement {
611
775
  }
612
776
  ```
613
777
 
614
- ## Hook-Based Plugins (JS Engine)
778
+ ## Plugins (JS Engine)
615
779
 
616
- The JS engine (`@mdream/js`) supports imperative hook-based plugins for custom transform logic. These allow you to intercept and modify the conversion pipeline at multiple stages.
780
+ The JS engine composes plugins in an explicit array.
617
781
 
618
782
  ```ts
619
783
  import { htmlToMarkdown } from '@mdream/js'
@@ -631,19 +795,30 @@ const myPlugin = createPlugin({
631
795
  },
632
796
  })
633
797
 
634
- const markdown = htmlToMarkdown(html, { hooks: [myPlugin] })
798
+ const markdown = htmlToMarkdown(html, { plugins: [myPlugin] })
635
799
  ```
636
800
 
637
801
  ### Plugin Hooks
638
802
 
803
+ Each hook gets a read-only `PluginState`. To keep data for a node, set it on `node.context`.
804
+
639
805
  ```ts
806
+ interface PluginState {
807
+ /** Options passed to the converter. */
808
+ readonly options: Readonly<EngineOptions>
809
+ /** Output that the running converter writes. */
810
+ readonly outputFormat: 'markdown' | 'text' | 'html'
811
+ /** Nesting depth of the current node. It is 0 before the first node. */
812
+ readonly depth: number
813
+ }
814
+
640
815
  interface TransformPlugin {
641
816
  /**
642
817
  * Called before any node processing. Return { skip: true } to skip the node.
643
818
  */
644
819
  beforeNodeProcess?: (
645
820
  event: NodeEvent,
646
- state: MdreamRuntimeState,
821
+ state: PluginState,
647
822
  ) => undefined | void | { skip: boolean }
648
823
 
649
824
  /**
@@ -652,7 +827,7 @@ interface TransformPlugin {
652
827
  */
653
828
  onNodeEnter?: (
654
829
  node: ElementNode,
655
- state: MdreamRuntimeState,
830
+ state: PluginState,
656
831
  ) => string | undefined | void
657
832
 
658
833
  /**
@@ -661,7 +836,7 @@ interface TransformPlugin {
661
836
  */
662
837
  onNodeExit?: (
663
838
  node: ElementNode,
664
- state: MdreamRuntimeState,
839
+ state: PluginState,
665
840
  ) => string | undefined | void
666
841
 
667
842
  /**
@@ -669,7 +844,7 @@ interface TransformPlugin {
669
844
  */
670
845
  processAttributes?: (
671
846
  node: ElementNode,
672
- state: MdreamRuntimeState,
847
+ state: PluginState,
673
848
  ) => void
674
849
 
675
850
  /**
@@ -678,8 +853,13 @@ interface TransformPlugin {
678
853
  */
679
854
  processTextNode?: (
680
855
  node: TextNode,
681
- state: MdreamRuntimeState,
856
+ state: PluginState,
682
857
  ) => { content: string, skip: boolean } | undefined
858
+
859
+ /**
860
+ * Called once after the whole document is converted, including for streams.
861
+ */
862
+ onDocumentEnd?: (state: PluginState) => void
683
863
  }
684
864
  ```
685
865
 
@@ -688,13 +868,18 @@ interface TransformPlugin {
688
868
  A typed identity function for creating plugins with full TypeScript inference:
689
869
 
690
870
  ```ts
871
+ import type { ElementNode } from '@mdream/js'
872
+ import { ELEMENT_NODE } from '@mdream/js'
691
873
  import { createPlugin } from '@mdream/js/plugins'
692
874
 
693
875
  const plugin = createPlugin({
694
876
  beforeNodeProcess({ node }) {
695
877
  // Skip all div elements with class "ad"
696
- if (node.type === 1 && node.attributes?.class?.includes('ad')) {
697
- return { skip: true }
878
+ if (node.type === ELEMENT_NODE) {
879
+ const element = node as ElementNode
880
+ if (element.name === 'div' && element.attributes.class?.split(/\s+/).includes('ad')) {
881
+ return { skip: true }
882
+ }
698
883
  }
699
884
  },
700
885
  })
@@ -707,7 +892,6 @@ The following plugin factory functions are available from `@mdream/js/plugins`:
707
892
  ```ts
708
893
  import {
709
894
  createPlugin,
710
- extractionCollectorPlugin,
711
895
  extractionPlugin,
712
896
  filterPlugin,
713
897
  frontmatterPlugin,
@@ -725,7 +909,6 @@ Available from `@mdream/js/splitter`.
725
909
  ### Basic Chunking
726
910
 
727
911
  ```ts
728
- import { TAG_H2 } from '@mdream/js'
729
912
  import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
730
913
 
731
914
  const html = `
@@ -737,7 +920,7 @@ const html = `
737
920
  `
738
921
 
739
922
  const chunks = htmlToMarkdownSplitChunks(html, {
740
- headersToSplitOn: [TAG_H2],
923
+ headersToSplitOn: [2],
741
924
  chunkSize: 1000,
742
925
  chunkOverlap: 200,
743
926
  stripHeaders: true,
@@ -753,7 +936,10 @@ chunks.forEach((chunk) => {
753
936
 
754
937
  ### Streaming Chunks (Memory Efficient)
755
938
 
756
- For large documents, use the generator version to process chunks one at a time:
939
+ The generator converts the whole document before yielding its first chunk.
940
+ It keeps the converted Markdown in memory, but avoids collecting the chunk array.
941
+ Stopping early skips the remaining chunk work. HTML conversion has already finished.
942
+ See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for migration details.
757
943
 
758
944
  ```ts
759
945
  import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'
@@ -774,14 +960,14 @@ interface SplitterOptions {
774
960
  // --- Structural splitting ---
775
961
 
776
962
  /**
777
- * Header tag IDs to split on (TAG_H1 through TAG_H6).
778
- * Default: [TAG_H2, TAG_H3, TAG_H4, TAG_H5, TAG_H6]
963
+ * Heading levels that start a new chunk, from 1 (<h1>) to 6 (<h6>).
964
+ * Another value throws a TypeError. Default: [2, 3, 4, 5, 6]
779
965
  */
780
- headersToSplitOn?: number[]
966
+ headersToSplitOn?: (1 | 2 | 3 | 4 | 5 | 6)[]
781
967
 
782
968
  // --- Size-based splitting ---
783
969
 
784
- /** Maximum chunk size in characters. Default: 1000 */
970
+ /** Target chunk size in characters. Complete code fences can exceed it. Default: 1000 */
785
971
  chunkSize?: number
786
972
 
787
973
  /** Overlap between chunks for context preservation. Default: 200 */
@@ -801,7 +987,7 @@ interface SplitterOptions {
801
987
  /** Split into individual lines. Default: false */
802
988
  returnEachLine?: boolean
803
989
 
804
- /** Keep separators in the split chunks. Default: false */
990
+ /** The splitter drops whitespace between chunks for either value. */
805
991
  keepSeparator?: boolean
806
992
 
807
993
  // --- Standard options ---
@@ -809,14 +995,14 @@ interface SplitterOptions {
809
995
  /** Base URL for resolving relative links/images */
810
996
  origin?: string
811
997
 
812
- /** Declarative built-in plugin config */
813
- plugins?: BuiltinPlugins
998
+ /** Explicit plugins, applied in array order */
999
+ plugins?: Plugin[]
814
1000
 
815
- /** Hook-based plugins (JS engine only) */
816
- hooks?: TransformPlugin[]
1001
+ /** Custom tag output or aliases */
1002
+ tagOverrides?: Record<string, TagOverride | string>
817
1003
 
818
- /** Post-processing cleanup */
819
- clean?: boolean | CleanOptions
1004
+ /** Cleanup rules from clean() in @mdream/js/clean */
1005
+ clean?: Cleaner
820
1006
  }
821
1007
  ```
822
1008
 
@@ -832,7 +1018,7 @@ interface MarkdownChunk {
832
1018
  headers?: Record<string, string> // { h1: "Title", h2: "Section" }
833
1019
  /** Code block language if chunk contains code */
834
1020
  code?: string
835
- /** Line number range in the original document */
1021
+ /** Inclusive, one-based line range in converted Markdown, excluding stripped headings */
836
1022
  loc?: {
837
1023
  lines: { from: number, to: number }
838
1024
  }
@@ -845,12 +1031,11 @@ interface MarkdownChunk {
845
1031
  Combine splitting with presets:
846
1032
 
847
1033
  ```ts
848
- import { TAG_H2 } from '@mdream/js'
849
1034
  import { withMinimalPreset } from '@mdream/js/preset/minimal'
850
1035
  import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
851
1036
 
852
1037
  const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
853
- headersToSplitOn: [TAG_H2],
1038
+ headersToSplitOn: [2],
854
1039
  chunkSize: 500,
855
1040
  origin: 'https://example.com',
856
1041
  }))
@@ -861,18 +1046,17 @@ const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
861
1046
  The `@mdream/js/negotiate` module provides HTTP content negotiation utilities for serving Markdown to LLM clients:
862
1047
 
863
1048
  ```ts
864
- import { parseAcceptHeader, shouldServeMarkdown } from '@mdream/js/negotiate'
865
-
866
- // Check if client prefers markdown
867
- const serveMarkdown = shouldServeMarkdown(
868
- request.headers.get('accept'),
869
- request.headers.get('sec-fetch-dest'),
870
- )
871
-
872
- if (serveMarkdown) {
873
- return new Response(markdown, {
874
- headers: { 'Content-Type': 'text/markdown' },
875
- })
1049
+ import { shouldServeMarkdown } from '@mdream/js/negotiate'
1050
+
1051
+ export function markdownResponse(request: Request, markdown: string): Response | undefined {
1052
+ if (shouldServeMarkdown(
1053
+ request.headers.get('accept') ?? undefined,
1054
+ request.headers.get('sec-fetch-dest') ?? undefined,
1055
+ )) {
1056
+ return new Response(markdown, {
1057
+ headers: { 'Content-Type': 'text/markdown' },
1058
+ })
1059
+ }
876
1060
  }
877
1061
  ```
878
1062
 
@@ -883,14 +1067,16 @@ if (serveMarkdown) {
883
1067
  If you only need to parse HTML into a DOM-like event stream without converting to Markdown, use `parseHtml` from the JS engine:
884
1068
 
885
1069
  ```ts
1070
+ import type { ElementNode } from '@mdream/js'
1071
+ import { ELEMENT_NODE, NodeEventEnter } from '@mdream/js'
886
1072
  import { parseHtml } from '@mdream/js/parse'
887
1073
 
888
1074
  const html = '<div><h1>Title</h1><p>Content</p></div>'
889
1075
  const { events, remainingHtml } = parseHtml(html)
890
1076
 
891
1077
  events.forEach((event) => {
892
- if (event.type === 0 && event.node.type === 1) { // Enter + Element
893
- console.log('Entering element:', event.node.name)
1078
+ if (event.type === NodeEventEnter && event.node.type === ELEMENT_NODE) {
1079
+ console.log('Entering element:', (event.node as ElementNode).name)
894
1080
  }
895
1081
  })
896
1082
  ```
@@ -949,18 +1135,27 @@ cat index.html \
949
1135
  | `-h`, `--help` | Display help information |
950
1136
 
951
1137
  The CLI reads HTML from stdin and writes Markdown, plain text, or HTML to stdout. It uses the streaming API internally.
1138
+ A value option also accepts the `--origin=<url>` form.
1139
+ If you pass an unknown flag, a flag without a value, or an unknown preset, the CLI writes one line to stderr and exits with code 1.
952
1140
 
953
1141
  ## Browser and Edge Usage
954
1142
 
955
- Every entry takes the same `MdreamOptions` and produces the same Markdown string. In Node and edge runtimes, `htmlToMarkdown` is synchronous. In browser bundles, the CDN script, and `mdream/worker`, it returns `Promise<string>`, because the WASM binary loads first. Each export condition ships its own types, so TypeScript shows the right return type when it resolves the `browser` condition (for example, with `customConditions: ["browser"]`).
1143
+ Every entry takes the same `MdreamOptions` and produces the same Markdown string.
1144
+ In Node and edge runtimes, `htmlToMarkdown` from `mdream` is synchronous.
1145
+ In a browser bundle, import from `mdream/browser`.
1146
+ Its `htmlToMarkdown` returns `Promise<string>`, because the WASM binary loads first.
1147
+ The CDN script also returns `Promise<string>`.
956
1148
 
957
1149
  ```ts
958
- import { htmlToMarkdown } from 'mdream'
1150
+ import { htmlToMarkdown } from 'mdream/browser'
959
1151
 
960
- // In a browser bundle
961
1152
  const markdown = await htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
962
1153
  ```
963
1154
 
1155
+ In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
1156
+ The `mdream` types always describe the synchronous API.
1157
+ To convert off the main thread, see [Web Worker](#web-worker).
1158
+
964
1159
  ### Edge / Cloudflare Workers
965
1160
 
966
1161
  For edge runtimes (Cloudflare Workers, Vercel Edge), `mdream` automatically selects the WASM build via export conditions (`workerd`, `edge-light`). Both `htmlToMarkdown` and `streamHtmlToMarkdown` are available:
@@ -978,20 +1173,31 @@ for await (const chunk of streamHtmlToMarkdown(response.body)) {
978
1173
  }
979
1174
  ```
980
1175
 
981
- Conversion errors are catchable. WASM has no unwinding, so an internal Rust panic aborts the call, but the message is preserved and re-thrown as a normal `Error` (with the raw WASM trap as `cause`), and the instance keeps working for later requests:
1176
+ You can catch conversion errors.
1177
+ An internal Rust panic aborts the call because WASM has no unwinding.
1178
+ The wrapper preserves the panic message in an `Error`, with the raw WASM trap as `cause`.
1179
+ The instance remains usable for later requests:
982
1180
 
983
1181
  ```ts
984
- try {
985
- return new Response(htmlToMarkdown(html))
986
- }
987
- catch (error) {
988
- // "mdream WASM panic, please report this at ...\npanicked at ..."
989
- console.error(error)
990
- return new Response('conversion failed', { status: 500 })
1182
+ import { htmlToMarkdown } from 'mdream'
1183
+
1184
+ export function convertResponse(html: string): Response {
1185
+ try {
1186
+ return new Response(htmlToMarkdown(html))
1187
+ }
1188
+ catch (error) {
1189
+ console.error(error)
1190
+ return new Response('Conversion failed', { status: 500 })
1191
+ }
991
1192
  }
992
1193
  ```
993
1194
 
994
- If your toolchain doesn't resolve the export conditions, the wasm-bindgen build (web target) is exposed at `mdream/wasm` for manual initialization. Its `htmlToMarkdown` takes the same `MdreamOptions` as the `mdream` entry. Its `MarkdownStream` runs the callbacks in `finish()`.
1195
+ If your toolchain does not resolve export conditions, import the web-target wasm-bindgen build from `mdream/wasm`.
1196
+ Initialize it manually before converting.
1197
+ Its `htmlToMarkdown` takes the same `MdreamOptions` as `mdream`.
1198
+ Its `MarkdownStream` runs callbacks in `finish()`.
1199
+
1200
+ Your bundler must support `.wasm` imports and provide their TypeScript declarations.
995
1201
 
996
1202
  ```ts
997
1203
  import init, { htmlToMarkdown } from 'mdream/wasm'
@@ -1001,11 +1207,12 @@ await init({ module_or_path: wasmModule })
1001
1207
  const markdown = htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
1002
1208
  ```
1003
1209
 
1004
- `mdream/worker` is not an edge entry. It runs the conversion in a browser Web Worker; see [Web Worker](#web-worker).
1005
-
1006
1210
  ### Browser CDN (IIFE)
1007
1211
 
1008
- Use mdream directly via CDN with no build step. The script inlines the WASM binary and initializes it on load, so `window.mdream.htmlToMarkdown()` is ready at once. It takes the same `MdreamOptions` as the `mdream` entry and returns `Promise<string>`, the same as the browser bundle.
1212
+ Use mdream from a CDN without a build step.
1213
+ The script includes the WASM binary and initializes it on load.
1214
+ Call `window.mdream.htmlToMarkdown()` with the same `MdreamOptions` as `mdream`.
1215
+ Await its `Promise<string>`, as with `mdream/browser`.
1009
1216
 
1010
1217
  ```html
1011
1218
  <script src="https://unpkg.com/mdream/dist/iife.js"></script>
@@ -1019,22 +1226,6 @@ Use mdream directly via CDN with no build step. The script inlines the WASM bina
1019
1226
  - **unpkg**: `https://unpkg.com/mdream/dist/iife.js`
1020
1227
  - **jsDelivr**: `https://cdn.jsdelivr.net/npm/mdream/dist/iife.js`
1021
1228
 
1022
- ### Web Worker
1023
-
1024
- For browser environments, `mdream/worker` runs conversions off the main thread using a Web Worker:
1025
-
1026
- ```ts
1027
- import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
1028
-
1029
- await initWorker('/path/to/mdream_edge_bg.wasm')
1030
-
1031
- // Takes the same options as the mdream entry. Callbacks run on this thread.
1032
- const markdown = await htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
1033
-
1034
- // Clean up
1035
- terminateWorker()
1036
- ```
1037
-
1038
1229
  ## Content Extraction with Readability
1039
1230
 
1040
1231
  For advanced content extraction (article detection, boilerplate removal), use [@mozilla/readability](https://github.com/mozilla/readability) before mdream:
@@ -1047,7 +1238,7 @@ import { htmlToMarkdown } from 'mdream'
1047
1238
  const dom = new JSDOM(html, { url: 'https://example.com' })
1048
1239
  const article = new Readability(dom.window.document).parse()
1049
1240
 
1050
- if (article) {
1241
+ if (article?.content) {
1051
1242
  const markdown = htmlToMarkdown(article.content)
1052
1243
  // article.title, article.excerpt, article.byline also available
1053
1244
  }
@@ -1080,7 +1271,7 @@ console.log(result.llmsFullTxt) // llms-full.txt content
1080
1271
  | Package | Description |
1081
1272
  |---------|-------------|
1082
1273
  | [`mdream`](https://npmjs.com/package/mdream) | Core HTML to Markdown converter (Rust + WASM engine) |
1083
- | [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with hook-based plugins, splitter, and llms.txt generation (`@mdream/js/llms-txt`) |
1274
+ | [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with tree-shakable formats, plugins, splitter, and llms.txt generation (`@mdream/js/llms-txt`) |
1084
1275
  | [`@mdream/crawl`](https://github.com/harlan-zw/mdream/tree/main/packages/crawl) | Site-wide crawler for llms.txt generation |
1085
1276
  | [`@mdream/vite`](https://github.com/harlan-zw/mdream/tree/main/packages/vite) | Vite plugin integration |
1086
1277
  | [`@mdream/nuxt`](https://github.com/harlan-zw/mdream/tree/main/packages/nuxt) | Nuxt module integration |