mdream 1.7.2 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +357 -163
- package/bin/mdream.mjs +45 -24
- package/dist/browser-stub.mjs +7 -0
- package/dist/browser.d.mts +98 -5
- package/dist/browser.mjs +224 -33
- package/dist/edge.d.mts +93 -4
- package/dist/edge.mjs +151 -69
- package/dist/iife.js +1 -396
- package/dist/index.d.mts +8 -6
- package/dist/index.mjs +104 -43
- package/dist/{worker.d.mts → wasm.d.mts} +20 -20
- package/dist/wasm.mjs +150 -0
- package/napi/index.d.mts +20 -33
- package/napi/index.d.ts +20 -33
- package/napi/index.mjs +48 -36
- package/package.json +47 -37
- package/skills/mdream/SKILL.md +128 -0
- package/skills/mdream/references/runtimes.md +57 -0
- package/wasm/mdream_edge.d.ts +27 -0
- package/wasm/mdream_edge.js +94 -20
- package/wasm/mdream_edge_bg.wasm +0 -0
- package/wasm/mdream_edge_bg.wasm.d.ts +3 -0
- package/wasm/package.json +1 -1
- package/wasm-bundler/mdream_edge.d.ts +24 -0
- package/wasm-bundler/mdream_edge.js +6 -2
- package/wasm-bundler/mdream_edge_bg.js +94 -20
- package/wasm-bundler/mdream_edge_bg.wasm +0 -0
- package/wasm-bundler/mdream_edge_bg.wasm.d.ts +3 -0
- package/wasm-bundler/package.json +1 -1
- package/dist/worker.mjs +0 -100
package/README.md
CHANGED
|
@@ -3,6 +3,18 @@
|
|
|
3
3
|
[](https://npmjs.com/package/mdream)
|
|
4
4
|
[](https://npm.chart.dev/mdream)
|
|
5
5
|
[](https://github.com/harlan-zw/mdream/blob/main/LICENSE.md)
|
|
6
|
+
<a href="https://skilld.dev/gh/harlan-zw/mdream">
|
|
7
|
+
<picture>
|
|
8
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://skilld.dev/b/harlan-zw/mdream?theme=dark">
|
|
9
|
+
<source media="(prefers-color-scheme: light)" srcset="https://skilld.dev/b/harlan-zw/mdream?theme=light">
|
|
10
|
+
<img alt="Skill repository on skilld.dev" src="https://skilld.dev/b/harlan-zw/mdream?theme=light">
|
|
11
|
+
</picture>
|
|
12
|
+
</a>
|
|
13
|
+
|
|
14
|
+
Powering Cloudflare Browser Run's [/markdown](https://developers.cloudflare.com/browser-run/quick-actions/markdown-endpoint/) and [/crawl](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/) endpoints.
|
|
15
|
+
|
|
16
|
+
> [!TIP]
|
|
17
|
+
> 🎉 **Upgrading to Mdream v2?** Follow [Migrating from v1](#migrating-from-v1).
|
|
6
18
|
|
|
7
19
|
## Installation
|
|
8
20
|
|
|
@@ -17,7 +29,10 @@ pnpm add mdream
|
|
|
17
29
|
yarn add mdream
|
|
18
30
|
```
|
|
19
31
|
|
|
20
|
-
|
|
32
|
+
> [!TIP]
|
|
33
|
+
> Using an AI agent? Get the mdream Skill on [skilld.dev/gh/harlan-zw/mdream](https://skilld.dev/gh/harlan-zw/mdream).
|
|
34
|
+
|
|
35
|
+
For the JavaScript-only engine with explicit plugins, formats, splitter, and parser:
|
|
21
36
|
|
|
22
37
|
```bash
|
|
23
38
|
pnpm add @mdream/js
|
|
@@ -48,6 +63,7 @@ externals: ['mdream']
|
|
|
48
63
|
|
|
49
64
|
## Table of Contents
|
|
50
65
|
|
|
66
|
+
- [Migrating from v1](#migrating-from-v1)
|
|
51
67
|
- [API Reference](#api-reference)
|
|
52
68
|
- [htmlToMarkdown()](#htmltomarkdown)
|
|
53
69
|
- [streamHtmlToMarkdown()](#streamhtmltomarkdown)
|
|
@@ -67,7 +83,7 @@ externals: ['mdream']
|
|
|
67
83
|
- [Tailwind](#tailwind-plugin)
|
|
68
84
|
- [Filter](#filter-plugin)
|
|
69
85
|
- [Extraction](#extraction-plugin)
|
|
70
|
-
- [
|
|
86
|
+
- [Plugins (JS Engine)](#plugins-js-engine)
|
|
71
87
|
- [Plugin Hooks](#plugin-hooks)
|
|
72
88
|
- [createPlugin()](#createplugin)
|
|
73
89
|
- [Markdown Splitting (JS Engine)](#markdown-splitting-js-engine)
|
|
@@ -81,30 +97,172 @@ externals: ['mdream']
|
|
|
81
97
|
- [Browser and Edge Usage](#browser-and-edge-usage)
|
|
82
98
|
- [Edge / Cloudflare Workers](#edge--cloudflare-workers)
|
|
83
99
|
- [Browser CDN (IIFE)](#browser-cdn-iife)
|
|
84
|
-
- [Web Worker](#web-worker)
|
|
85
100
|
- [llms.txt Generation](#llmstxt-generation)
|
|
86
101
|
- [Related Packages](#related-packages)
|
|
87
102
|
|
|
103
|
+
## Migrating from v1
|
|
104
|
+
|
|
105
|
+
Using a coding agent? Give it the [v1 to v2 migration prompt](https://github.com/harlan-zw/mdream/blob/main/MIGRATION_PROMPT.md).
|
|
106
|
+
|
|
107
|
+
Install `mdream` for the Rust engine.
|
|
108
|
+
This guide covers v2.
|
|
109
|
+
For an earlier release, read the README at its release tag.
|
|
110
|
+
Node and edge conversions still return a string synchronously.
|
|
111
|
+
The `format`, `clean`, and top-level plugin options remain available.
|
|
112
|
+
Custom hook plugins use `@mdream/js`; follow its [migration guide](../js/README.md#migrating-from-v1).
|
|
113
|
+
That guide also covers upgrades from an earlier v2 beta.
|
|
114
|
+
|
|
115
|
+
### Unknown options
|
|
116
|
+
|
|
117
|
+
Every entry throws a `TypeError` for an option that it does not read.
|
|
118
|
+
In v1, mdream ignored an unknown option, so a typo gave no error.
|
|
119
|
+
The error names the fix.
|
|
120
|
+
An option set to `undefined` does not throw.
|
|
121
|
+
|
|
122
|
+
| Old option | v2 option |
|
|
123
|
+
|------------|-----------|
|
|
124
|
+
| `preset: 'minimal'` | `minimal: true` |
|
|
125
|
+
| `cleanUrls: true` | `clean: { urls: true }` |
|
|
126
|
+
| `plugins: { frontmatter: true }` | `frontmatter: true`, at the top level |
|
|
127
|
+
| `plugins: [myPlugin]` | `@mdream/js` with `plugins: [myPlugin]` |
|
|
128
|
+
|
|
129
|
+
`clean.blankLines` is removed. It had no effect in either engine, so delete it.
|
|
130
|
+
|
|
131
|
+
The `mdream` CLI exits with code 1 on an unknown flag, a flag without a value, or an unknown `--preset` value.
|
|
132
|
+
In v1, the CLI ignored them.
|
|
133
|
+
|
|
134
|
+
### Browser imports and returns
|
|
135
|
+
|
|
136
|
+
In a browser bundle, import from `mdream/browser`.
|
|
137
|
+
Its `htmlToMarkdown` returns `Promise<string>`.
|
|
138
|
+
Remove `.markdown` and await the conversion:
|
|
139
|
+
|
|
140
|
+
```diff
|
|
141
|
+
- import { htmlToMarkdown } from 'mdream'
|
|
142
|
+
- const markdown = (await htmlToMarkdown(html)).markdown
|
|
143
|
+
+ import { htmlToMarkdown } from 'mdream/browser'
|
|
144
|
+
+ const markdown = await htmlToMarkdown(html)
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
|
|
148
|
+
The import itself does not throw.
|
|
149
|
+
A module that runs on the server and the client can import `mdream` and call it only on the server.
|
|
150
|
+
The `mdream/browser` types need no TypeScript `customConditions` setting.
|
|
151
|
+
|
|
152
|
+
For a CDN script:
|
|
153
|
+
|
|
154
|
+
```diff
|
|
155
|
+
- const markdown = window.mdream.htmlToMarkdown(html).markdown
|
|
156
|
+
+ const markdown = await window.mdream.htmlToMarkdown(html)
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Use the CDN example inside an async function or a `<script type="module">` block.
|
|
160
|
+
See [Browser and Edge Usage](#browser-and-edge-usage) for edge runtimes and the CDN script.
|
|
161
|
+
|
|
162
|
+
### Web Worker
|
|
163
|
+
|
|
164
|
+
v2 removes `mdream/worker`.
|
|
165
|
+
To convert off the main thread, write a module worker that imports `mdream/browser`:
|
|
166
|
+
|
|
167
|
+
```ts
|
|
168
|
+
// md.worker.ts
|
|
169
|
+
import { htmlToMarkdown } from 'mdream/browser'
|
|
170
|
+
|
|
171
|
+
addEventListener('message', async (event: MessageEvent<string>) => {
|
|
172
|
+
postMessage(await htmlToMarkdown(event.data, { minimal: true }))
|
|
173
|
+
})
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
// main.ts
|
|
178
|
+
const worker = new Worker(new URL('./md.worker.ts', import.meta.url), { type: 'module' })
|
|
179
|
+
worker.onmessage = (event: MessageEvent<string>) => console.log(event.data)
|
|
180
|
+
worker.postMessage('<h1>Hello</h1>')
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Your bundler builds the worker and copies the WASM binary.
|
|
184
|
+
Functions cannot cross `postMessage`.
|
|
185
|
+
To use callbacks such as `frontmatter`, set them inside the worker.
|
|
186
|
+
|
|
187
|
+
### Minimal filtering
|
|
188
|
+
|
|
189
|
+
With `minimal: true`, a custom `filter` adds to the preset's excludes.
|
|
190
|
+
In v1, it replaced them and could restore forms and navigation.
|
|
191
|
+
To keep those elements in v2, disable the preset filter.
|
|
192
|
+
To apply only your own excludes, use `minimal: false` and compose the options:
|
|
193
|
+
|
|
194
|
+
```ts
|
|
195
|
+
import { htmlToMarkdown } from 'mdream'
|
|
196
|
+
|
|
197
|
+
htmlToMarkdown(html, { minimal: true, filter: false })
|
|
198
|
+
|
|
199
|
+
htmlToMarkdown(html, {
|
|
200
|
+
frontmatter: true,
|
|
201
|
+
isolateMain: true,
|
|
202
|
+
tailwind: true,
|
|
203
|
+
clean: true,
|
|
204
|
+
filter: { exclude: ['footer'] },
|
|
205
|
+
})
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
### Page titles
|
|
209
|
+
|
|
210
|
+
`<title>` text no longer appears in the body, in any output format.
|
|
211
|
+
To include it in Markdown metadata, enable `frontmatter: true`.
|
|
212
|
+
To read it separately, use `extraction: { title: element => console.log(element.textContent) }`.
|
|
213
|
+
Review saved output or snapshots that relied on a leading title line.
|
|
214
|
+
The converter also removes trailing spaces from link text before empty elements. See [#318](https://github.com/harlan-zw/mdream/pull/318).
|
|
215
|
+
Stream boundaries can change. Join the output before comparing snapshots.
|
|
216
|
+
|
|
217
|
+
### Markdown splitting
|
|
218
|
+
|
|
219
|
+
The JS splitter changes its chunk text, line ranges, overlap, and first-chunk timing.
|
|
220
|
+
If you store chunks or embeddings, rebuild them from the original HTML.
|
|
221
|
+
See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for the upgrade steps.
|
|
222
|
+
|
|
223
|
+
### Direct native binding access
|
|
224
|
+
|
|
225
|
+
The package's `exports` map defines its public API. It does not expose `napi/`.
|
|
226
|
+
If you loaded the native binding by file path, replace these removed exports:
|
|
227
|
+
|
|
228
|
+
| Removed native API | Public API |
|
|
229
|
+
|---|---|
|
|
230
|
+
| `splitMarkdown`, `htmlToMarkdownChunks` | `htmlToMarkdownSplitChunks` from `@mdream/js/splitter`, with the original HTML input |
|
|
231
|
+
| `htmlToMarkdownBytes` | Decode complete HTML bytes with `TextDecoder`, then call `htmlToMarkdown` from `mdream` |
|
|
232
|
+
| `MarkdownStream.processChunkBytes` | `streamHtmlToMarkdown` from `mdream`, which still accepts byte streams |
|
|
233
|
+
|
|
234
|
+
For direct Markdown splitting in Rust, use `mdream::splitter::split_markdown`.
|
|
235
|
+
|
|
88
236
|
## API Reference
|
|
89
237
|
|
|
90
238
|
### `htmlToMarkdown()`
|
|
91
239
|
|
|
92
|
-
Converts a complete HTML string to Markdown, plain text, or HTML
|
|
240
|
+
Converts a complete HTML string to Markdown, plain text, or HTML. It is synchronous in Node and on edge runtimes.
|
|
93
241
|
|
|
94
242
|
**Rust engine** (`mdream`):
|
|
95
243
|
|
|
96
244
|
```ts
|
|
97
|
-
import {
|
|
245
|
+
import type { MdreamOptions } from 'mdream'
|
|
246
|
+
|
|
247
|
+
declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
|
|
98
248
|
|
|
99
|
-
|
|
249
|
+
// `mdream/browser` and the CDN script return Promise<string> instead.
|
|
100
250
|
```
|
|
101
251
|
|
|
102
|
-
**JS engine** (`@mdream/js`):
|
|
252
|
+
**JS engine** (`@mdream/js`) uses a separate entry point for each format:
|
|
103
253
|
|
|
104
254
|
```ts
|
|
105
255
|
import { htmlToMarkdown } from '@mdream/js'
|
|
256
|
+
import { htmlToSafeHtml } from '@mdream/js/html'
|
|
257
|
+
import { htmlToText } from '@mdream/js/text'
|
|
258
|
+
```
|
|
106
259
|
|
|
107
|
-
|
|
260
|
+
```ts
|
|
261
|
+
import type { MdreamOptions } from '@mdream/js'
|
|
262
|
+
|
|
263
|
+
declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
|
|
264
|
+
declare function htmlToText(html: string, options?: Partial<MdreamOptions>): string
|
|
265
|
+
declare function htmlToSafeHtml(html: string, options?: Partial<MdreamOptions>): string
|
|
108
266
|
```
|
|
109
267
|
|
|
110
268
|
**Example:**
|
|
@@ -132,11 +290,13 @@ const html = htmlToMarkdown('<h1>Hello <strong>World</strong></h1>', {
|
|
|
132
290
|
|
|
133
291
|
Converts an HTML `ReadableStream` to Markdown, plain text, or HTML incrementally. Returns an `AsyncIterable<string>` that yields output chunks as they are processed.
|
|
134
292
|
|
|
293
|
+
It takes the same options as `htmlToMarkdown()`. The `frontmatter` callback and the `extraction` handlers run once, after the stream ends and before the last chunk is yielded.
|
|
294
|
+
|
|
135
295
|
|
|
136
296
|
```ts
|
|
137
|
-
import {
|
|
297
|
+
import type { MdreamOptions } from 'mdream'
|
|
138
298
|
|
|
139
|
-
function streamHtmlToMarkdown(
|
|
299
|
+
declare function streamHtmlToMarkdown(
|
|
140
300
|
htmlStream: ReadableStream<Uint8Array | string> | null,
|
|
141
301
|
options?: Partial<MdreamOptions>,
|
|
142
302
|
): AsyncIterable<string>
|
|
@@ -160,23 +320,31 @@ for await (const chunk of streamHtmlToMarkdown(stream, {
|
|
|
160
320
|
|
|
161
321
|
## Engines
|
|
162
322
|
|
|
163
|
-
|
|
323
|
+
Choose the Rust or JavaScript engine.
|
|
324
|
+
The `mdream` package selects its Rust binding by runtime.
|
|
325
|
+
Import `@mdream/js` to use the JavaScript engine:
|
|
164
326
|
|
|
165
327
|
| Engine | Package | Plugins | Use case |
|
|
166
328
|
|--------|---------|---------|----------|
|
|
167
329
|
| **Rust** (NAPI) | `mdream` | Declarative config only | Node.js (default) |
|
|
168
|
-
| **Rust** (WASM) | `mdream` | Declarative config only | Edge, browser |
|
|
169
|
-
| **JavaScript** | `@mdream/js` |
|
|
330
|
+
| **Rust** (WASM) | `mdream`, `mdream/browser` | Declarative config only | Edge, browser |
|
|
331
|
+
| **JavaScript** | `@mdream/js` | Explicit plugin arrays | Small bundles, custom plugins, splitter |
|
|
170
332
|
|
|
171
333
|
```ts
|
|
172
|
-
// JavaScript engine
|
|
334
|
+
// JavaScript engine for explicit plugins and smaller bundles
|
|
173
335
|
import { htmlToMarkdown } from '@mdream/js'
|
|
174
336
|
|
|
175
337
|
// Rust NAPI engine (auto-selected in Node.js)
|
|
176
|
-
import { htmlToMarkdown } from 'mdream'
|
|
338
|
+
import { htmlToMarkdown as htmlToMarkdownRust } from 'mdream'
|
|
177
339
|
```
|
|
178
340
|
|
|
179
|
-
|
|
341
|
+
The Rust engine takes declarative options: `minimal`, `format`, `frontmatter`, `isolateMain`, `tailwind`, `filter`, and `extraction`.
|
|
342
|
+
Pass them at the top level.
|
|
343
|
+
|
|
344
|
+
The JS engine takes an array of plugin instances in `options.plugins`.
|
|
345
|
+
Each output format has its own entry point.
|
|
346
|
+
For its minimal preset, use `withMinimalPreset()` from `@mdream/js/preset/minimal`.
|
|
347
|
+
If you pass Rust-only options to the JS engine, it throws a `TypeError` that names the fix.
|
|
180
348
|
|
|
181
349
|
## Options
|
|
182
350
|
|
|
@@ -216,7 +384,7 @@ interface MdreamOptions {
|
|
|
216
384
|
tailwind?: boolean
|
|
217
385
|
|
|
218
386
|
/** Filter elements by CSS selectors. Default when minimal: excludes form, nav, footer, etc. */
|
|
219
|
-
filter?: { include?: string[], exclude?: string[], processChildren?: boolean }
|
|
387
|
+
filter?: false | { include?: string[], exclude?: string[], processChildren?: boolean }
|
|
220
388
|
|
|
221
389
|
/** Extract elements matching CSS selectors during conversion. */
|
|
222
390
|
extraction?: Record<string, (element: ExtractedElement) => void>
|
|
@@ -236,20 +404,23 @@ interface MdreamOptions {
|
|
|
236
404
|
}
|
|
237
405
|
```
|
|
238
406
|
|
|
407
|
+
Any other key throws a `TypeError` that names the key and the fix.
|
|
408
|
+
|
|
239
409
|
### MdreamOptions (JS engine)
|
|
240
410
|
|
|
241
|
-
The JS engine
|
|
411
|
+
The JS engine uses explicit plugins. Each output format has its own entry point.
|
|
242
412
|
|
|
243
413
|
```ts
|
|
244
414
|
interface MdreamOptions extends EngineOptions {
|
|
245
|
-
/**
|
|
246
|
-
|
|
415
|
+
/** Explicit plugins, applied in array order. */
|
|
416
|
+
plugins?: Plugin[]
|
|
247
417
|
}
|
|
248
418
|
|
|
249
419
|
interface EngineOptions {
|
|
250
420
|
origin?: string
|
|
251
|
-
clean
|
|
252
|
-
|
|
421
|
+
/** Cleanup rules from clean() in @mdream/js/clean */
|
|
422
|
+
clean?: Cleaner
|
|
423
|
+
tagOverrides?: Record<string, TagOverride | string>
|
|
253
424
|
|
|
254
425
|
/**
|
|
255
426
|
* Hard-wrap prose at this many characters, breaking on word boundaries.
|
|
@@ -258,26 +429,18 @@ interface EngineOptions {
|
|
|
258
429
|
*/
|
|
259
430
|
wrapWidth?: number
|
|
260
431
|
|
|
261
|
-
/** Output Markdown, plain text, or HTML. Default: 'markdown' */
|
|
262
|
-
format?: 'markdown' | 'text' | 'html'
|
|
263
|
-
}
|
|
264
|
-
|
|
265
|
-
interface BuiltinPlugins {
|
|
266
|
-
filter?: { include?: (string | number)[], exclude?: (string | number)[], processChildren?: boolean }
|
|
267
|
-
frontmatter?: boolean | ((fm: Record<string, string>) => void) | FrontmatterConfig
|
|
268
|
-
isolateMain?: boolean
|
|
269
|
-
tailwind?: boolean
|
|
270
|
-
extraction?: Record<string, (element: ExtractedElement) => void>
|
|
271
|
-
tagOverrides?: Record<string, TagOverride | string>
|
|
272
432
|
}
|
|
273
433
|
```
|
|
274
434
|
|
|
275
|
-
|
|
435
|
+
Use `@mdream/js/text` for plain text. Use `@mdream/js/html` for safe HTML.
|
|
436
|
+
Any other key throws a `TypeError` that names the key and the fix.
|
|
276
437
|
|
|
277
438
|
### CleanOptions
|
|
278
439
|
|
|
279
440
|
Post-processing cleanup applied to the final Markdown output. All options default to `false` unless `clean: true` is set.
|
|
280
441
|
|
|
442
|
+
With `@mdream/js`, pass these rules to `clean()` from `@mdream/js/clean`, for example `clean: clean({ urls: true })`.
|
|
443
|
+
|
|
281
444
|
```ts
|
|
282
445
|
interface CleanOptions {
|
|
283
446
|
/** Strip tracking query parameters (utm_*, fbclid, gclid, etc.) from URLs */
|
|
@@ -286,8 +449,6 @@ interface CleanOptions {
|
|
|
286
449
|
fragments?: boolean
|
|
287
450
|
/** Strip links with meaningless hrefs (#, javascript:void(0)) to plain text */
|
|
288
451
|
emptyLinks?: boolean
|
|
289
|
-
/** Collapse 3+ consecutive blank lines to 2 */
|
|
290
|
-
blankLines?: boolean
|
|
291
452
|
/** Strip links where text equals URL: [https://x.com](https://x.com) becomes https://x.com */
|
|
292
453
|
redundantLinks?: boolean
|
|
293
454
|
/** Strip self-referencing heading anchors: ## [Title](#title) becomes ## Title */
|
|
@@ -302,6 +463,8 @@ interface CleanOptions {
|
|
|
302
463
|
**Example:**
|
|
303
464
|
|
|
304
465
|
```ts
|
|
466
|
+
import { htmlToMarkdown } from 'mdream'
|
|
467
|
+
|
|
305
468
|
const markdown = htmlToMarkdown(html, {
|
|
306
469
|
clean: {
|
|
307
470
|
urls: true,
|
|
@@ -336,7 +499,7 @@ Override how specific HTML tags are rendered in Markdown. String values act as a
|
|
|
336
499
|
>
|
|
337
500
|
> ```ts
|
|
338
501
|
> htmlToMarkdown('<my-em>hi</my-em>', { tagOverrides: { 'my-em': 'em' } })
|
|
339
|
-
> // → "
|
|
502
|
+
> // → "*hi*"
|
|
340
503
|
> ```
|
|
341
504
|
|
|
342
505
|
```ts
|
|
@@ -361,6 +524,8 @@ interface TagOverride {
|
|
|
361
524
|
**Example:**
|
|
362
525
|
|
|
363
526
|
```ts
|
|
527
|
+
import { htmlToMarkdown } from 'mdream'
|
|
528
|
+
|
|
364
529
|
const markdown = htmlToMarkdown(html, {
|
|
365
530
|
tagOverrides: {
|
|
366
531
|
// Treat <x-heading> like <h2>
|
|
@@ -375,12 +540,14 @@ const markdown = htmlToMarkdown(html, {
|
|
|
375
540
|
})
|
|
376
541
|
```
|
|
377
542
|
|
|
378
|
-
**Output is GitHub Flavored Markdown.** Mdream emits a fixed GFM dialect tuned for LLM input: ATX headings (`#`), fenced code blocks (` ``` `), `-` bullets,
|
|
543
|
+
**Output is GitHub Flavored Markdown.** Mdream emits a fixed GFM dialect tuned for LLM input: ATX headings (`#`), fenced code blocks (` ``` `), `-` bullets, `*` emphasis, `**` strong, `---` horizontal rules, inline links. These are not configurable. For simple delimiter swaps you can use `tagOverrides`:
|
|
379
544
|
|
|
380
545
|
```ts
|
|
546
|
+
import { htmlToMarkdown } from 'mdream'
|
|
547
|
+
|
|
381
548
|
htmlToMarkdown(html, {
|
|
382
549
|
tagOverrides: {
|
|
383
|
-
em: { enter: '
|
|
550
|
+
em: { enter: '_', exit: '_', isInline: true }, // *x* → _x_
|
|
384
551
|
strong: { enter: '__', exit: '__', isInline: true }, // **x** → __x__
|
|
385
552
|
hr: { enter: '* * *', exit: '' }, // --- → * * *
|
|
386
553
|
},
|
|
@@ -393,10 +560,10 @@ Structural style differences (setext headings, indented code blocks, reference-s
|
|
|
393
560
|
|
|
394
561
|
```ts
|
|
395
562
|
interface FilterOptions {
|
|
396
|
-
/** CSS selectors
|
|
397
|
-
include?:
|
|
398
|
-
/** CSS selectors
|
|
399
|
-
exclude?:
|
|
563
|
+
/** CSS selectors or tag names for elements to include (all others excluded) */
|
|
564
|
+
include?: string[]
|
|
565
|
+
/** CSS selectors or tag names for elements to exclude */
|
|
566
|
+
exclude?: string[]
|
|
400
567
|
/** Whether to also process children of matched elements. Default: true */
|
|
401
568
|
processChildren?: boolean
|
|
402
569
|
}
|
|
@@ -414,6 +581,18 @@ The `minimal` preset enables the following plugins together:
|
|
|
414
581
|
- **filter**: Excludes `form`, `fieldset`, `object`, `embed`, `footer`, `aside`, `iframe`, `input`, `textarea`, `select`, `button`, `nav`
|
|
415
582
|
- **clean**: All post-processing cleanup enabled
|
|
416
583
|
|
|
584
|
+
A `filter` passed with `minimal` adds to the preset's excludes. It does not replace them. To turn the preset's filter off, pass `filter: false`. In the JS engine, a `filterPlugin()` appended to `withMinimalPreset()` also adds to the excludes.
|
|
585
|
+
|
|
586
|
+
```ts
|
|
587
|
+
import { htmlToMarkdown } from 'mdream'
|
|
588
|
+
|
|
589
|
+
// Excludes h1 as well as form, nav, footer, and the rest
|
|
590
|
+
htmlToMarkdown(html, { minimal: true, filter: { exclude: ['h1'] } })
|
|
591
|
+
|
|
592
|
+
// Keeps forms and nav
|
|
593
|
+
htmlToMarkdown(html, { minimal: true, filter: false })
|
|
594
|
+
```
|
|
595
|
+
|
|
417
596
|
**Rust engine:**
|
|
418
597
|
|
|
419
598
|
```ts
|
|
@@ -436,20 +615,21 @@ const markdown = htmlToMarkdown(html, withMinimalPreset({
|
|
|
436
615
|
}))
|
|
437
616
|
```
|
|
438
617
|
|
|
439
|
-
`withMinimalPreset()` returns
|
|
618
|
+
`withMinimalPreset()` returns explicit plugin defaults. You can append custom plugins. An appended `filterPlugin()` adds to the preset's excludes:
|
|
440
619
|
|
|
441
620
|
```ts
|
|
621
|
+
import { htmlToMarkdown } from '@mdream/js'
|
|
622
|
+
import { filterPlugin } from '@mdream/js/plugins'
|
|
623
|
+
import { withMinimalPreset } from '@mdream/js/preset/minimal'
|
|
624
|
+
|
|
442
625
|
const markdown = htmlToMarkdown(html, withMinimalPreset({
|
|
443
|
-
plugins: {
|
|
444
|
-
frontmatter: false,
|
|
445
|
-
filter: { exclude: ['nav'] },
|
|
446
|
-
},
|
|
626
|
+
plugins: [filterPlugin({ exclude: ['.cookie-banner'] }), myPlugin],
|
|
447
627
|
}))
|
|
448
628
|
```
|
|
449
629
|
|
|
450
630
|
## Built-in Plugins
|
|
451
631
|
|
|
452
|
-
|
|
632
|
+
The Rust engine uses declarative options. The JS engine exports matching plugin factories.
|
|
453
633
|
|
|
454
634
|
### Frontmatter Plugin
|
|
455
635
|
|
|
@@ -458,6 +638,8 @@ Extracts metadata from the HTML `<head>` element and generates YAML frontmatter.
|
|
|
458
638
|
**Extracted fields by default:** `title`, `description`, `keywords`, `author`, `date`, `og:title`, `og:description`, `twitter:title`, `twitter:description`.
|
|
459
639
|
|
|
460
640
|
```ts
|
|
641
|
+
import { htmlToMarkdown } from 'mdream'
|
|
642
|
+
|
|
461
643
|
// Enable with defaults
|
|
462
644
|
htmlToMarkdown(html, { frontmatter: true })
|
|
463
645
|
|
|
@@ -500,6 +682,8 @@ Isolates the main content area using the following priority:
|
|
|
500
682
|
4. The `<head>` section is always passed through for other plugins (e.g., frontmatter)
|
|
501
683
|
|
|
502
684
|
```ts
|
|
685
|
+
import { htmlToMarkdown } from 'mdream'
|
|
686
|
+
|
|
503
687
|
htmlToMarkdown(html, { isolateMain: true })
|
|
504
688
|
```
|
|
505
689
|
|
|
@@ -518,14 +702,18 @@ Converts Tailwind CSS utility classes to semantic Markdown formatting:
|
|
|
518
702
|
Supports responsive breakpoint prefixes (`sm:`, `md:`, `lg:`, `xl:`, `2xl:`) with mobile-first resolution.
|
|
519
703
|
|
|
520
704
|
```ts
|
|
705
|
+
import { htmlToMarkdown } from 'mdream'
|
|
706
|
+
|
|
521
707
|
htmlToMarkdown(html, { tailwind: true })
|
|
522
708
|
```
|
|
523
709
|
|
|
524
710
|
### Filter Plugin
|
|
525
711
|
|
|
526
|
-
Filters HTML elements by CSS selectors
|
|
712
|
+
Filters HTML elements by CSS selectors or tag names.
|
|
527
713
|
|
|
528
714
|
```ts
|
|
715
|
+
import { htmlToMarkdown } from 'mdream'
|
|
716
|
+
|
|
529
717
|
// Exclude navigation, sidebar, footer
|
|
530
718
|
htmlToMarkdown(html, {
|
|
531
719
|
filter: {
|
|
@@ -541,15 +729,14 @@ htmlToMarkdown(html, {
|
|
|
541
729
|
})
|
|
542
730
|
```
|
|
543
731
|
|
|
544
|
-
The JS engine
|
|
732
|
+
The JS engine takes the same strings in `filterPlugin`. If an entry is not a string, `filterPlugin` throws a `TypeError`.
|
|
545
733
|
|
|
546
734
|
```ts
|
|
547
|
-
import {
|
|
735
|
+
import { htmlToMarkdown } from '@mdream/js'
|
|
736
|
+
import { filterPlugin } from '@mdream/js/plugins'
|
|
548
737
|
|
|
549
738
|
htmlToMarkdown(html, {
|
|
550
|
-
plugins: {
|
|
551
|
-
filter: { exclude: [TAG_NAV, TAG_FOOTER] },
|
|
552
|
-
},
|
|
739
|
+
plugins: [filterPlugin({ exclude: ['nav', 'footer'] })],
|
|
553
740
|
})
|
|
554
741
|
```
|
|
555
742
|
|
|
@@ -560,6 +747,8 @@ Elements with `style="position: absolute"` or `style="position: fixed"` are also
|
|
|
560
747
|
Extracts elements matching CSS selectors during conversion. Callbacks receive the matched element with its accumulated text content and attributes.
|
|
561
748
|
|
|
562
749
|
```ts
|
|
750
|
+
import { htmlToMarkdown } from 'mdream'
|
|
751
|
+
|
|
563
752
|
htmlToMarkdown(html, {
|
|
564
753
|
extraction: {
|
|
565
754
|
'h2': (el) => {
|
|
@@ -586,9 +775,9 @@ interface ExtractedElement {
|
|
|
586
775
|
}
|
|
587
776
|
```
|
|
588
777
|
|
|
589
|
-
##
|
|
778
|
+
## Plugins (JS Engine)
|
|
590
779
|
|
|
591
|
-
The JS engine
|
|
780
|
+
The JS engine composes plugins in an explicit array.
|
|
592
781
|
|
|
593
782
|
```ts
|
|
594
783
|
import { htmlToMarkdown } from '@mdream/js'
|
|
@@ -606,19 +795,30 @@ const myPlugin = createPlugin({
|
|
|
606
795
|
},
|
|
607
796
|
})
|
|
608
797
|
|
|
609
|
-
const markdown = htmlToMarkdown(html, {
|
|
798
|
+
const markdown = htmlToMarkdown(html, { plugins: [myPlugin] })
|
|
610
799
|
```
|
|
611
800
|
|
|
612
801
|
### Plugin Hooks
|
|
613
802
|
|
|
803
|
+
Each hook gets a read-only `PluginState`. To keep data for a node, set it on `node.context`.
|
|
804
|
+
|
|
614
805
|
```ts
|
|
806
|
+
interface PluginState {
|
|
807
|
+
/** Options passed to the converter. */
|
|
808
|
+
readonly options: Readonly<EngineOptions>
|
|
809
|
+
/** Output that the running converter writes. */
|
|
810
|
+
readonly outputFormat: 'markdown' | 'text' | 'html'
|
|
811
|
+
/** Nesting depth of the current node. It is 0 before the first node. */
|
|
812
|
+
readonly depth: number
|
|
813
|
+
}
|
|
814
|
+
|
|
615
815
|
interface TransformPlugin {
|
|
616
816
|
/**
|
|
617
817
|
* Called before any node processing. Return { skip: true } to skip the node.
|
|
618
818
|
*/
|
|
619
819
|
beforeNodeProcess?: (
|
|
620
820
|
event: NodeEvent,
|
|
621
|
-
state:
|
|
821
|
+
state: PluginState,
|
|
622
822
|
) => undefined | void | { skip: boolean }
|
|
623
823
|
|
|
624
824
|
/**
|
|
@@ -627,7 +827,7 @@ interface TransformPlugin {
|
|
|
627
827
|
*/
|
|
628
828
|
onNodeEnter?: (
|
|
629
829
|
node: ElementNode,
|
|
630
|
-
state:
|
|
830
|
+
state: PluginState,
|
|
631
831
|
) => string | undefined | void
|
|
632
832
|
|
|
633
833
|
/**
|
|
@@ -636,7 +836,7 @@ interface TransformPlugin {
|
|
|
636
836
|
*/
|
|
637
837
|
onNodeExit?: (
|
|
638
838
|
node: ElementNode,
|
|
639
|
-
state:
|
|
839
|
+
state: PluginState,
|
|
640
840
|
) => string | undefined | void
|
|
641
841
|
|
|
642
842
|
/**
|
|
@@ -644,7 +844,7 @@ interface TransformPlugin {
|
|
|
644
844
|
*/
|
|
645
845
|
processAttributes?: (
|
|
646
846
|
node: ElementNode,
|
|
647
|
-
state:
|
|
847
|
+
state: PluginState,
|
|
648
848
|
) => void
|
|
649
849
|
|
|
650
850
|
/**
|
|
@@ -653,8 +853,13 @@ interface TransformPlugin {
|
|
|
653
853
|
*/
|
|
654
854
|
processTextNode?: (
|
|
655
855
|
node: TextNode,
|
|
656
|
-
state:
|
|
856
|
+
state: PluginState,
|
|
657
857
|
) => { content: string, skip: boolean } | undefined
|
|
858
|
+
|
|
859
|
+
/**
|
|
860
|
+
* Called once after the whole document is converted, including for streams.
|
|
861
|
+
*/
|
|
862
|
+
onDocumentEnd?: (state: PluginState) => void
|
|
658
863
|
}
|
|
659
864
|
```
|
|
660
865
|
|
|
@@ -663,13 +868,18 @@ interface TransformPlugin {
|
|
|
663
868
|
A typed identity function for creating plugins with full TypeScript inference:
|
|
664
869
|
|
|
665
870
|
```ts
|
|
871
|
+
import type { ElementNode } from '@mdream/js'
|
|
872
|
+
import { ELEMENT_NODE } from '@mdream/js'
|
|
666
873
|
import { createPlugin } from '@mdream/js/plugins'
|
|
667
874
|
|
|
668
875
|
const plugin = createPlugin({
|
|
669
876
|
beforeNodeProcess({ node }) {
|
|
670
877
|
// Skip all div elements with class "ad"
|
|
671
|
-
if (node.type ===
|
|
672
|
-
|
|
878
|
+
if (node.type === ELEMENT_NODE) {
|
|
879
|
+
const element = node as ElementNode
|
|
880
|
+
if (element.name === 'div' && element.attributes.class?.split(/\s+/).includes('ad')) {
|
|
881
|
+
return { skip: true }
|
|
882
|
+
}
|
|
673
883
|
}
|
|
674
884
|
},
|
|
675
885
|
})
|
|
@@ -682,7 +892,6 @@ The following plugin factory functions are available from `@mdream/js/plugins`:
|
|
|
682
892
|
```ts
|
|
683
893
|
import {
|
|
684
894
|
createPlugin,
|
|
685
|
-
extractionCollectorPlugin,
|
|
686
895
|
extractionPlugin,
|
|
687
896
|
filterPlugin,
|
|
688
897
|
frontmatterPlugin,
|
|
@@ -700,7 +909,6 @@ Available from `@mdream/js/splitter`.
|
|
|
700
909
|
### Basic Chunking
|
|
701
910
|
|
|
702
911
|
```ts
|
|
703
|
-
import { TAG_H2 } from '@mdream/js'
|
|
704
912
|
import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
|
|
705
913
|
|
|
706
914
|
const html = `
|
|
@@ -712,7 +920,7 @@ const html = `
|
|
|
712
920
|
`
|
|
713
921
|
|
|
714
922
|
const chunks = htmlToMarkdownSplitChunks(html, {
|
|
715
|
-
headersToSplitOn: [
|
|
923
|
+
headersToSplitOn: [2],
|
|
716
924
|
chunkSize: 1000,
|
|
717
925
|
chunkOverlap: 200,
|
|
718
926
|
stripHeaders: true,
|
|
@@ -728,7 +936,10 @@ chunks.forEach((chunk) => {
|
|
|
728
936
|
|
|
729
937
|
### Streaming Chunks (Memory Efficient)
|
|
730
938
|
|
|
731
|
-
|
|
939
|
+
The generator converts the whole document before yielding its first chunk.
|
|
940
|
+
It keeps the converted Markdown in memory, but avoids collecting the chunk array.
|
|
941
|
+
Stopping early skips the remaining chunk work. HTML conversion has already finished.
|
|
942
|
+
See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for migration details.
|
|
732
943
|
|
|
733
944
|
```ts
|
|
734
945
|
import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'
|
|
@@ -749,14 +960,14 @@ interface SplitterOptions {
|
|
|
749
960
|
// --- Structural splitting ---
|
|
750
961
|
|
|
751
962
|
/**
|
|
752
|
-
*
|
|
753
|
-
* Default: [
|
|
963
|
+
* Heading levels that start a new chunk, from 1 (<h1>) to 6 (<h6>).
|
|
964
|
+
* Another value throws a TypeError. Default: [2, 3, 4, 5, 6]
|
|
754
965
|
*/
|
|
755
|
-
headersToSplitOn?:
|
|
966
|
+
headersToSplitOn?: (1 | 2 | 3 | 4 | 5 | 6)[]
|
|
756
967
|
|
|
757
968
|
// --- Size-based splitting ---
|
|
758
969
|
|
|
759
|
-
/**
|
|
970
|
+
/** Target chunk size in characters. Complete code fences can exceed it. Default: 1000 */
|
|
760
971
|
chunkSize?: number
|
|
761
972
|
|
|
762
973
|
/** Overlap between chunks for context preservation. Default: 200 */
|
|
@@ -776,7 +987,7 @@ interface SplitterOptions {
|
|
|
776
987
|
/** Split into individual lines. Default: false */
|
|
777
988
|
returnEachLine?: boolean
|
|
778
989
|
|
|
779
|
-
/**
|
|
990
|
+
/** The splitter drops whitespace between chunks for either value. */
|
|
780
991
|
keepSeparator?: boolean
|
|
781
992
|
|
|
782
993
|
// --- Standard options ---
|
|
@@ -784,14 +995,14 @@ interface SplitterOptions {
|
|
|
784
995
|
/** Base URL for resolving relative links/images */
|
|
785
996
|
origin?: string
|
|
786
997
|
|
|
787
|
-
/**
|
|
788
|
-
plugins?:
|
|
998
|
+
/** Explicit plugins, applied in array order */
|
|
999
|
+
plugins?: Plugin[]
|
|
789
1000
|
|
|
790
|
-
/**
|
|
791
|
-
|
|
1001
|
+
/** Custom tag output or aliases */
|
|
1002
|
+
tagOverrides?: Record<string, TagOverride | string>
|
|
792
1003
|
|
|
793
|
-
/**
|
|
794
|
-
clean?:
|
|
1004
|
+
/** Cleanup rules from clean() in @mdream/js/clean */
|
|
1005
|
+
clean?: Cleaner
|
|
795
1006
|
}
|
|
796
1007
|
```
|
|
797
1008
|
|
|
@@ -807,7 +1018,7 @@ interface MarkdownChunk {
|
|
|
807
1018
|
headers?: Record<string, string> // { h1: "Title", h2: "Section" }
|
|
808
1019
|
/** Code block language if chunk contains code */
|
|
809
1020
|
code?: string
|
|
810
|
-
/**
|
|
1021
|
+
/** Inclusive, one-based line range in converted Markdown, excluding stripped headings */
|
|
811
1022
|
loc?: {
|
|
812
1023
|
lines: { from: number, to: number }
|
|
813
1024
|
}
|
|
@@ -820,12 +1031,11 @@ interface MarkdownChunk {
|
|
|
820
1031
|
Combine splitting with presets:
|
|
821
1032
|
|
|
822
1033
|
```ts
|
|
823
|
-
import { TAG_H2 } from '@mdream/js'
|
|
824
1034
|
import { withMinimalPreset } from '@mdream/js/preset/minimal'
|
|
825
1035
|
import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
|
|
826
1036
|
|
|
827
1037
|
const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
|
|
828
|
-
headersToSplitOn: [
|
|
1038
|
+
headersToSplitOn: [2],
|
|
829
1039
|
chunkSize: 500,
|
|
830
1040
|
origin: 'https://example.com',
|
|
831
1041
|
}))
|
|
@@ -836,18 +1046,17 @@ const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
|
|
|
836
1046
|
The `@mdream/js/negotiate` module provides HTTP content negotiation utilities for serving Markdown to LLM clients:
|
|
837
1047
|
|
|
838
1048
|
```ts
|
|
839
|
-
import {
|
|
840
|
-
|
|
841
|
-
|
|
842
|
-
|
|
843
|
-
|
|
844
|
-
|
|
845
|
-
)
|
|
846
|
-
|
|
847
|
-
|
|
848
|
-
|
|
849
|
-
|
|
850
|
-
})
|
|
1049
|
+
import { shouldServeMarkdown } from '@mdream/js/negotiate'
|
|
1050
|
+
|
|
1051
|
+
export function markdownResponse(request: Request, markdown: string): Response | undefined {
|
|
1052
|
+
if (shouldServeMarkdown(
|
|
1053
|
+
request.headers.get('accept') ?? undefined,
|
|
1054
|
+
request.headers.get('sec-fetch-dest') ?? undefined,
|
|
1055
|
+
)) {
|
|
1056
|
+
return new Response(markdown, {
|
|
1057
|
+
headers: { 'Content-Type': 'text/markdown' },
|
|
1058
|
+
})
|
|
1059
|
+
}
|
|
851
1060
|
}
|
|
852
1061
|
```
|
|
853
1062
|
|
|
@@ -858,14 +1067,16 @@ if (serveMarkdown) {
|
|
|
858
1067
|
If you only need to parse HTML into a DOM-like event stream without converting to Markdown, use `parseHtml` from the JS engine:
|
|
859
1068
|
|
|
860
1069
|
```ts
|
|
1070
|
+
import type { ElementNode } from '@mdream/js'
|
|
1071
|
+
import { ELEMENT_NODE, NodeEventEnter } from '@mdream/js'
|
|
861
1072
|
import { parseHtml } from '@mdream/js/parse'
|
|
862
1073
|
|
|
863
1074
|
const html = '<div><h1>Title</h1><p>Content</p></div>'
|
|
864
1075
|
const { events, remainingHtml } = parseHtml(html)
|
|
865
1076
|
|
|
866
1077
|
events.forEach((event) => {
|
|
867
|
-
if (event.type ===
|
|
868
|
-
console.log('Entering element:', event.node.name)
|
|
1078
|
+
if (event.type === NodeEventEnter && event.node.type === ELEMENT_NODE) {
|
|
1079
|
+
console.log('Entering element:', (event.node as ElementNode).name)
|
|
869
1080
|
}
|
|
870
1081
|
})
|
|
871
1082
|
```
|
|
@@ -924,9 +1135,27 @@ cat index.html \
|
|
|
924
1135
|
| `-h`, `--help` | Display help information |
|
|
925
1136
|
|
|
926
1137
|
The CLI reads HTML from stdin and writes Markdown, plain text, or HTML to stdout. It uses the streaming API internally.
|
|
1138
|
+
A value option also accepts the `--origin=<url>` form.
|
|
1139
|
+
If you pass an unknown flag, a flag without a value, or an unknown preset, the CLI writes one line to stderr and exits with code 1.
|
|
927
1140
|
|
|
928
1141
|
## Browser and Edge Usage
|
|
929
1142
|
|
|
1143
|
+
Every entry takes the same `MdreamOptions` and produces the same Markdown string.
|
|
1144
|
+
In Node and edge runtimes, `htmlToMarkdown` from `mdream` is synchronous.
|
|
1145
|
+
In a browser bundle, import from `mdream/browser`.
|
|
1146
|
+
Its `htmlToMarkdown` returns `Promise<string>`, because the WASM binary loads first.
|
|
1147
|
+
The CDN script also returns `Promise<string>`.
|
|
1148
|
+
|
|
1149
|
+
```ts
|
|
1150
|
+
import { htmlToMarkdown } from 'mdream/browser'
|
|
1151
|
+
|
|
1152
|
+
const markdown = await htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
|
|
1153
|
+
```
|
|
1154
|
+
|
|
1155
|
+
In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
|
|
1156
|
+
The `mdream` types always describe the synchronous API.
|
|
1157
|
+
To convert off the main thread, see [Web Worker](#web-worker).
|
|
1158
|
+
|
|
930
1159
|
### Edge / Cloudflare Workers
|
|
931
1160
|
|
|
932
1161
|
For edge runtimes (Cloudflare Workers, Vercel Edge), `mdream` automatically selects the WASM build via export conditions (`workerd`, `edge-light`). Both `htmlToMarkdown` and `streamHtmlToMarkdown` are available:
|
|
@@ -944,93 +1173,59 @@ for await (const chunk of streamHtmlToMarkdown(response.body)) {
|
|
|
944
1173
|
}
|
|
945
1174
|
```
|
|
946
1175
|
|
|
947
|
-
|
|
1176
|
+
You can catch conversion errors.
|
|
1177
|
+
An internal Rust panic aborts the call because WASM has no unwinding.
|
|
1178
|
+
The wrapper preserves the panic message in an `Error`, with the raw WASM trap as `cause`.
|
|
1179
|
+
The instance remains usable for later requests:
|
|
948
1180
|
|
|
949
1181
|
```ts
|
|
950
|
-
|
|
951
|
-
|
|
952
|
-
|
|
953
|
-
|
|
954
|
-
|
|
955
|
-
|
|
956
|
-
|
|
1182
|
+
import { htmlToMarkdown } from 'mdream'
|
|
1183
|
+
|
|
1184
|
+
export function convertResponse(html: string): Response {
|
|
1185
|
+
try {
|
|
1186
|
+
return new Response(htmlToMarkdown(html))
|
|
1187
|
+
}
|
|
1188
|
+
catch (error) {
|
|
1189
|
+
console.error(error)
|
|
1190
|
+
return new Response('Conversion failed', { status: 500 })
|
|
1191
|
+
}
|
|
957
1192
|
}
|
|
958
1193
|
```
|
|
959
1194
|
|
|
960
|
-
If your toolchain
|
|
1195
|
+
If your toolchain does not resolve export conditions, import the web-target wasm-bindgen build from `mdream/wasm`.
|
|
1196
|
+
Initialize it manually before converting.
|
|
1197
|
+
Its `htmlToMarkdown` takes the same `MdreamOptions` as `mdream`.
|
|
1198
|
+
Its `MarkdownStream` runs callbacks in `finish()`.
|
|
1199
|
+
|
|
1200
|
+
Your bundler must support `.wasm` imports and provide their TypeScript declarations.
|
|
961
1201
|
|
|
962
1202
|
```ts
|
|
963
1203
|
import init, { htmlToMarkdown } from 'mdream/wasm'
|
|
964
1204
|
import wasmModule from 'mdream/wasm/mdream_edge_bg.wasm'
|
|
965
1205
|
|
|
966
1206
|
await init({ module_or_path: wasmModule })
|
|
967
|
-
const markdown = htmlToMarkdown('<h1>Hello</h1>', {})
|
|
968
|
-
```
|
|
969
|
-
|
|
970
|
-
You can also import the edge entry point directly:
|
|
971
|
-
|
|
972
|
-
```ts
|
|
973
|
-
import { htmlToMarkdown } from 'mdream/worker'
|
|
974
|
-
```
|
|
975
|
-
|
|
976
|
-
The `mdream/worker` entry provides an async API since WASM must be initialized first:
|
|
977
|
-
|
|
978
|
-
```ts
|
|
979
|
-
import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
|
|
980
|
-
|
|
981
|
-
// Initialize once with the WASM URL
|
|
982
|
-
await initWorker('https://cdn.example.com/mdream_edge_bg.wasm')
|
|
983
|
-
|
|
984
|
-
// Convert (returns Promise<string>)
|
|
985
|
-
const markdown = await htmlToMarkdown('<h1>Hello</h1>')
|
|
986
|
-
|
|
987
|
-
// Clean up when done
|
|
988
|
-
terminateWorker()
|
|
1207
|
+
const markdown = htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
|
|
989
1208
|
```
|
|
990
1209
|
|
|
991
1210
|
### Browser CDN (IIFE)
|
|
992
1211
|
|
|
993
|
-
Use mdream
|
|
1212
|
+
Use mdream from a CDN without a build step.
|
|
1213
|
+
The script includes the WASM binary and initializes it on load.
|
|
1214
|
+
Call `window.mdream.htmlToMarkdown()` with the same `MdreamOptions` as `mdream`.
|
|
1215
|
+
Await its `Promise<string>`, as with `mdream/browser`.
|
|
994
1216
|
|
|
995
1217
|
```html
|
|
996
1218
|
<script src="https://unpkg.com/mdream/dist/iife.js"></script>
|
|
997
1219
|
<script>
|
|
998
|
-
|
|
999
|
-
|
|
1000
|
-
console.log(markdown) // # Hello\n\nWorld
|
|
1220
|
+
window.mdream.htmlToMarkdown('<h1>Hello</h1><p>World</p>')
|
|
1221
|
+
.then(markdown => console.log(markdown)) // # Hello\n\nWorld
|
|
1001
1222
|
</script>
|
|
1002
1223
|
```
|
|
1003
1224
|
|
|
1004
|
-
You can pass a custom WASM URL or `ArrayBuffer` to `init()`:
|
|
1005
|
-
|
|
1006
|
-
```js
|
|
1007
|
-
// Custom URL
|
|
1008
|
-
await window.mdream.init('https://cdn.example.com/mdream_edge_bg.wasm')
|
|
1009
|
-
|
|
1010
|
-
// Pre-loaded ArrayBuffer
|
|
1011
|
-
const wasmBytes = await fetch('/wasm/mdream_edge_bg.wasm').then(r => r.arrayBuffer())
|
|
1012
|
-
await window.mdream.init(wasmBytes)
|
|
1013
|
-
```
|
|
1014
|
-
|
|
1015
1225
|
**CDN Options:**
|
|
1016
1226
|
- **unpkg**: `https://unpkg.com/mdream/dist/iife.js`
|
|
1017
1227
|
- **jsDelivr**: `https://cdn.jsdelivr.net/npm/mdream/dist/iife.js`
|
|
1018
1228
|
|
|
1019
|
-
### Web Worker
|
|
1020
|
-
|
|
1021
|
-
For browser environments, `mdream/worker` runs conversions off the main thread using a Web Worker:
|
|
1022
|
-
|
|
1023
|
-
```ts
|
|
1024
|
-
import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
|
|
1025
|
-
|
|
1026
|
-
await initWorker('/path/to/mdream_edge_bg.wasm')
|
|
1027
|
-
|
|
1028
|
-
const markdown = await htmlToMarkdown('<h1>Hello</h1>')
|
|
1029
|
-
|
|
1030
|
-
// Clean up
|
|
1031
|
-
terminateWorker()
|
|
1032
|
-
```
|
|
1033
|
-
|
|
1034
1229
|
## Content Extraction with Readability
|
|
1035
1230
|
|
|
1036
1231
|
For advanced content extraction (article detection, boilerplate removal), use [@mozilla/readability](https://github.com/mozilla/readability) before mdream:
|
|
@@ -1043,7 +1238,7 @@ import { htmlToMarkdown } from 'mdream'
|
|
|
1043
1238
|
const dom = new JSDOM(html, { url: 'https://example.com' })
|
|
1044
1239
|
const article = new Readability(dom.window.document).parse()
|
|
1045
1240
|
|
|
1046
|
-
if (article) {
|
|
1241
|
+
if (article?.content) {
|
|
1047
1242
|
const markdown = htmlToMarkdown(article.content)
|
|
1048
1243
|
// article.title, article.excerpt, article.byline also available
|
|
1049
1244
|
}
|
|
@@ -1051,10 +1246,10 @@ if (article) {
|
|
|
1051
1246
|
|
|
1052
1247
|
## llms.txt Generation
|
|
1053
1248
|
|
|
1054
|
-
For llms.txt artifact generation, use
|
|
1249
|
+
For llms.txt artifact generation, use `@mdream/js/llms-txt` from the `@mdream/js` package. It accepts pre-converted Markdown and generates `llms.txt` and `llms-full.txt` artifacts.
|
|
1055
1250
|
|
|
1056
1251
|
```ts
|
|
1057
|
-
import { generateLlmsTxtArtifacts } from '@mdream/llms-txt'
|
|
1252
|
+
import { generateLlmsTxtArtifacts } from '@mdream/js/llms-txt'
|
|
1058
1253
|
import { htmlToMarkdown } from 'mdream'
|
|
1059
1254
|
|
|
1060
1255
|
const result = await generateLlmsTxtArtifacts({
|
|
@@ -1076,8 +1271,7 @@ console.log(result.llmsFullTxt) // llms-full.txt content
|
|
|
1076
1271
|
| Package | Description |
|
|
1077
1272
|
|---------|-------------|
|
|
1078
1273
|
| [`mdream`](https://npmjs.com/package/mdream) | Core HTML to Markdown converter (Rust + WASM engine) |
|
|
1079
|
-
| [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with
|
|
1080
|
-
| [`@mdream/llms-txt`](https://github.com/harlan-zw/mdream/tree/main/packages/llms-txt) | Engine-agnostic llms.txt artifact generation |
|
|
1274
|
+
| [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with tree-shakable formats, plugins, splitter, and llms.txt generation (`@mdream/js/llms-txt`) |
|
|
1081
1275
|
| [`@mdream/crawl`](https://github.com/harlan-zw/mdream/tree/main/packages/crawl) | Site-wide crawler for llms.txt generation |
|
|
1082
1276
|
| [`@mdream/vite`](https://github.com/harlan-zw/mdream/tree/main/packages/vite) | Vite plugin integration |
|
|
1083
1277
|
| [`@mdream/nuxt`](https://github.com/harlan-zw/mdream/tree/main/packages/nuxt) | Nuxt module integration |
|