mdream 2.0.0-beta.1 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +319 -128
- package/bin/mdream.mjs +45 -24
- package/dist/browser-stub.mjs +7 -0
- package/dist/browser.d.mts +1 -3
- package/dist/browser.mjs +41 -30
- package/dist/edge.d.mts +0 -2
- package/dist/edge.mjs +41 -30
- package/dist/iife.js +1 -605
- package/dist/index.d.mts +0 -2
- package/dist/index.mjs +42 -32
- package/dist/wasm.d.mts +2 -5
- package/dist/wasm.mjs +41 -31
- package/napi/index.d.mts +0 -33
- package/napi/index.d.ts +0 -33
- package/napi/index.mjs +1 -4
- package/package.json +34 -34
- package/skills/mdream/SKILL.md +9 -5
- package/skills/mdream/references/runtimes.md +17 -9
- package/wasm/mdream_edge.js +3 -1
- package/wasm/mdream_edge_bg.wasm +0 -0
- package/wasm/package.json +1 -1
- package/wasm-bundler/mdream_edge_bg.js +3 -1
- package/wasm-bundler/mdream_edge_bg.wasm +0 -0
- package/wasm-bundler/package.json +1 -1
- package/dist/worker.d.mts +0 -101
- package/dist/worker.mjs +0 -217
package/README.md
CHANGED
|
@@ -11,6 +11,11 @@
|
|
|
11
11
|
</picture>
|
|
12
12
|
</a>
|
|
13
13
|
|
|
14
|
+
Powering Cloudflare Browser Run's [/markdown](https://developers.cloudflare.com/browser-run/quick-actions/markdown-endpoint/) and [/crawl](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/) endpoints.
|
|
15
|
+
|
|
16
|
+
> [!TIP]
|
|
17
|
+
> 🎉 **Upgrading to Mdream v2?** Follow [Migrating from v1](#migrating-from-v1).
|
|
18
|
+
|
|
14
19
|
## Installation
|
|
15
20
|
|
|
16
21
|
```bash
|
|
@@ -27,7 +32,7 @@ yarn add mdream
|
|
|
27
32
|
> [!TIP]
|
|
28
33
|
> Using an AI agent? Get the mdream Skill on [skilld.dev/gh/harlan-zw/mdream](https://skilld.dev/gh/harlan-zw/mdream).
|
|
29
34
|
|
|
30
|
-
For the JavaScript-only engine
|
|
35
|
+
For the JavaScript-only engine with explicit plugins, formats, splitter, and parser:
|
|
31
36
|
|
|
32
37
|
```bash
|
|
33
38
|
pnpm add @mdream/js
|
|
@@ -58,6 +63,7 @@ externals: ['mdream']
|
|
|
58
63
|
|
|
59
64
|
## Table of Contents
|
|
60
65
|
|
|
66
|
+
- [Migrating from v1](#migrating-from-v1)
|
|
61
67
|
- [API Reference](#api-reference)
|
|
62
68
|
- [htmlToMarkdown()](#htmltomarkdown)
|
|
63
69
|
- [streamHtmlToMarkdown()](#streamhtmltomarkdown)
|
|
@@ -77,7 +83,7 @@ externals: ['mdream']
|
|
|
77
83
|
- [Tailwind](#tailwind-plugin)
|
|
78
84
|
- [Filter](#filter-plugin)
|
|
79
85
|
- [Extraction](#extraction-plugin)
|
|
80
|
-
- [
|
|
86
|
+
- [Plugins (JS Engine)](#plugins-js-engine)
|
|
81
87
|
- [Plugin Hooks](#plugin-hooks)
|
|
82
88
|
- [createPlugin()](#createplugin)
|
|
83
89
|
- [Markdown Splitting (JS Engine)](#markdown-splitting-js-engine)
|
|
@@ -91,10 +97,142 @@ externals: ['mdream']
|
|
|
91
97
|
- [Browser and Edge Usage](#browser-and-edge-usage)
|
|
92
98
|
- [Edge / Cloudflare Workers](#edge--cloudflare-workers)
|
|
93
99
|
- [Browser CDN (IIFE)](#browser-cdn-iife)
|
|
94
|
-
- [Web Worker](#web-worker)
|
|
95
100
|
- [llms.txt Generation](#llmstxt-generation)
|
|
96
101
|
- [Related Packages](#related-packages)
|
|
97
102
|
|
|
103
|
+
## Migrating from v1
|
|
104
|
+
|
|
105
|
+
Using a coding agent? Give it the [v1 to v2 migration prompt](https://github.com/harlan-zw/mdream/blob/main/MIGRATION_PROMPT.md).
|
|
106
|
+
|
|
107
|
+
Install `mdream` for the Rust engine.
|
|
108
|
+
This guide covers v2.
|
|
109
|
+
For an earlier release, read the README at its release tag.
|
|
110
|
+
Node and edge conversions still return a string synchronously.
|
|
111
|
+
The `format`, `clean`, and top-level plugin options remain available.
|
|
112
|
+
Custom hook plugins use `@mdream/js`; follow its [migration guide](../js/README.md#migrating-from-v1).
|
|
113
|
+
That guide also covers upgrades from an earlier v2 beta.
|
|
114
|
+
|
|
115
|
+
### Unknown options
|
|
116
|
+
|
|
117
|
+
Every entry throws a `TypeError` for an option that it does not read.
|
|
118
|
+
In v1, mdream ignored an unknown option, so a typo gave no error.
|
|
119
|
+
The error names the fix.
|
|
120
|
+
An option set to `undefined` does not throw.
|
|
121
|
+
|
|
122
|
+
| Old option | v2 option |
|
|
123
|
+
|------------|-----------|
|
|
124
|
+
| `preset: 'minimal'` | `minimal: true` |
|
|
125
|
+
| `cleanUrls: true` | `clean: { urls: true }` |
|
|
126
|
+
| `plugins: { frontmatter: true }` | `frontmatter: true`, at the top level |
|
|
127
|
+
| `plugins: [myPlugin]` | `@mdream/js` with `plugins: [myPlugin]` |
|
|
128
|
+
|
|
129
|
+
`clean.blankLines` is removed. It had no effect in either engine, so delete it.
|
|
130
|
+
|
|
131
|
+
The `mdream` CLI exits with code 1 on an unknown flag, a flag without a value, or an unknown `--preset` value.
|
|
132
|
+
In v1, the CLI ignored them.
|
|
133
|
+
|
|
134
|
+
### Browser imports and returns
|
|
135
|
+
|
|
136
|
+
In a browser bundle, import from `mdream/browser`.
|
|
137
|
+
Its `htmlToMarkdown` returns `Promise<string>`.
|
|
138
|
+
Remove `.markdown` and await the conversion:
|
|
139
|
+
|
|
140
|
+
```diff
|
|
141
|
+
- import { htmlToMarkdown } from 'mdream'
|
|
142
|
+
- const markdown = (await htmlToMarkdown(html)).markdown
|
|
143
|
+
+ import { htmlToMarkdown } from 'mdream/browser'
|
|
144
|
+
+ const markdown = await htmlToMarkdown(html)
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
|
|
148
|
+
The import itself does not throw.
|
|
149
|
+
A module that runs on the server and the client can import `mdream` and call it only on the server.
|
|
150
|
+
The `mdream/browser` types need no TypeScript `customConditions` setting.
|
|
151
|
+
|
|
152
|
+
For a CDN script:
|
|
153
|
+
|
|
154
|
+
```diff
|
|
155
|
+
- const markdown = window.mdream.htmlToMarkdown(html).markdown
|
|
156
|
+
+ const markdown = await window.mdream.htmlToMarkdown(html)
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Use the CDN example inside an async function or a `<script type="module">` block.
|
|
160
|
+
See [Browser and Edge Usage](#browser-and-edge-usage) for edge runtimes and the CDN script.
|
|
161
|
+
|
|
162
|
+
### Web Worker
|
|
163
|
+
|
|
164
|
+
v2 removes `mdream/worker`.
|
|
165
|
+
To convert off the main thread, write a module worker that imports `mdream/browser`:
|
|
166
|
+
|
|
167
|
+
```ts
|
|
168
|
+
// md.worker.ts
|
|
169
|
+
import { htmlToMarkdown } from 'mdream/browser'
|
|
170
|
+
|
|
171
|
+
addEventListener('message', async (event: MessageEvent<string>) => {
|
|
172
|
+
postMessage(await htmlToMarkdown(event.data, { minimal: true }))
|
|
173
|
+
})
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
// main.ts
|
|
178
|
+
const worker = new Worker(new URL('./md.worker.ts', import.meta.url), { type: 'module' })
|
|
179
|
+
worker.onmessage = (event: MessageEvent<string>) => console.log(event.data)
|
|
180
|
+
worker.postMessage('<h1>Hello</h1>')
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Your bundler builds the worker and copies the WASM binary.
|
|
184
|
+
Functions cannot cross `postMessage`.
|
|
185
|
+
To use callbacks such as `frontmatter`, set them inside the worker.
|
|
186
|
+
|
|
187
|
+
### Minimal filtering
|
|
188
|
+
|
|
189
|
+
With `minimal: true`, a custom `filter` adds to the preset's excludes.
|
|
190
|
+
In v1, it replaced them and could restore forms and navigation.
|
|
191
|
+
To keep those elements in v2, disable the preset filter.
|
|
192
|
+
To apply only your own excludes, use `minimal: false` and compose the options:
|
|
193
|
+
|
|
194
|
+
```ts
|
|
195
|
+
import { htmlToMarkdown } from 'mdream'
|
|
196
|
+
|
|
197
|
+
htmlToMarkdown(html, { minimal: true, filter: false })
|
|
198
|
+
|
|
199
|
+
htmlToMarkdown(html, {
|
|
200
|
+
frontmatter: true,
|
|
201
|
+
isolateMain: true,
|
|
202
|
+
tailwind: true,
|
|
203
|
+
clean: true,
|
|
204
|
+
filter: { exclude: ['footer'] },
|
|
205
|
+
})
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
### Page titles
|
|
209
|
+
|
|
210
|
+
`<title>` text no longer appears in the body, in any output format.
|
|
211
|
+
To include it in Markdown metadata, enable `frontmatter: true`.
|
|
212
|
+
To read it separately, use `extraction: { title: element => console.log(element.textContent) }`.
|
|
213
|
+
Review saved output or snapshots that relied on a leading title line.
|
|
214
|
+
The converter also removes trailing spaces from link text before empty elements. See [#318](https://github.com/harlan-zw/mdream/pull/318).
|
|
215
|
+
Stream boundaries can change. Join the output before comparing snapshots.
|
|
216
|
+
|
|
217
|
+
### Markdown splitting
|
|
218
|
+
|
|
219
|
+
The JS splitter changes its chunk text, line ranges, overlap, and first-chunk timing.
|
|
220
|
+
If you store chunks or embeddings, rebuild them from the original HTML.
|
|
221
|
+
See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for the upgrade steps.
|
|
222
|
+
|
|
223
|
+
### Direct native binding access
|
|
224
|
+
|
|
225
|
+
The package's `exports` map defines its public API. It does not expose `napi/`.
|
|
226
|
+
If you loaded the native binding by file path, replace these removed exports:
|
|
227
|
+
|
|
228
|
+
| Removed native API | Public API |
|
|
229
|
+
|---|---|
|
|
230
|
+
| `splitMarkdown`, `htmlToMarkdownChunks` | `htmlToMarkdownSplitChunks` from `@mdream/js/splitter`, with the original HTML input |
|
|
231
|
+
| `htmlToMarkdownBytes` | Decode complete HTML bytes with `TextDecoder`, then call `htmlToMarkdown` from `mdream` |
|
|
232
|
+
| `MarkdownStream.processChunkBytes` | `streamHtmlToMarkdown` from `mdream`, which still accepts byte streams |
|
|
233
|
+
|
|
234
|
+
For direct Markdown splitting in Rust, use `mdream::splitter::split_markdown`.
|
|
235
|
+
|
|
98
236
|
## API Reference
|
|
99
237
|
|
|
100
238
|
### `htmlToMarkdown()`
|
|
@@ -104,20 +242,27 @@ Converts a complete HTML string to Markdown, plain text, or HTML. It is synchron
|
|
|
104
242
|
**Rust engine** (`mdream`):
|
|
105
243
|
|
|
106
244
|
```ts
|
|
107
|
-
import {
|
|
245
|
+
import type { MdreamOptions } from 'mdream'
|
|
108
246
|
|
|
109
|
-
function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
|
|
247
|
+
declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
|
|
110
248
|
|
|
111
|
-
//
|
|
112
|
-
function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): Promise<string>
|
|
249
|
+
// `mdream/browser` and the CDN script return Promise<string> instead.
|
|
113
250
|
```
|
|
114
251
|
|
|
115
|
-
**JS engine** (`@mdream/js`):
|
|
252
|
+
**JS engine** (`@mdream/js`) uses a separate entry point for each format:
|
|
116
253
|
|
|
117
254
|
```ts
|
|
118
255
|
import { htmlToMarkdown } from '@mdream/js'
|
|
256
|
+
import { htmlToSafeHtml } from '@mdream/js/html'
|
|
257
|
+
import { htmlToText } from '@mdream/js/text'
|
|
258
|
+
```
|
|
119
259
|
|
|
120
|
-
|
|
260
|
+
```ts
|
|
261
|
+
import type { MdreamOptions } from '@mdream/js'
|
|
262
|
+
|
|
263
|
+
declare function htmlToMarkdown(html: string, options?: Partial<MdreamOptions>): string
|
|
264
|
+
declare function htmlToText(html: string, options?: Partial<MdreamOptions>): string
|
|
265
|
+
declare function htmlToSafeHtml(html: string, options?: Partial<MdreamOptions>): string
|
|
121
266
|
```
|
|
122
267
|
|
|
123
268
|
**Example:**
|
|
@@ -149,9 +294,9 @@ It takes the same options as `htmlToMarkdown()`. The `frontmatter` callback and
|
|
|
149
294
|
|
|
150
295
|
|
|
151
296
|
```ts
|
|
152
|
-
import {
|
|
297
|
+
import type { MdreamOptions } from 'mdream'
|
|
153
298
|
|
|
154
|
-
function streamHtmlToMarkdown(
|
|
299
|
+
declare function streamHtmlToMarkdown(
|
|
155
300
|
htmlStream: ReadableStream<Uint8Array | string> | null,
|
|
156
301
|
options?: Partial<MdreamOptions>,
|
|
157
302
|
): AsyncIterable<string>
|
|
@@ -175,23 +320,31 @@ for await (const chunk of streamHtmlToMarkdown(stream, {
|
|
|
175
320
|
|
|
176
321
|
## Engines
|
|
177
322
|
|
|
178
|
-
|
|
323
|
+
Choose the Rust or JavaScript engine.
|
|
324
|
+
The `mdream` package selects its Rust binding by runtime.
|
|
325
|
+
Import `@mdream/js` to use the JavaScript engine:
|
|
179
326
|
|
|
180
327
|
| Engine | Package | Plugins | Use case |
|
|
181
328
|
|--------|---------|---------|----------|
|
|
182
329
|
| **Rust** (NAPI) | `mdream` | Declarative config only | Node.js (default) |
|
|
183
|
-
| **Rust** (WASM) | `mdream` | Declarative config only | Edge, browser |
|
|
184
|
-
| **JavaScript** | `@mdream/js` |
|
|
330
|
+
| **Rust** (WASM) | `mdream`, `mdream/browser` | Declarative config only | Edge, browser |
|
|
331
|
+
| **JavaScript** | `@mdream/js` | Explicit plugin arrays | Small bundles, custom plugins, splitter |
|
|
185
332
|
|
|
186
333
|
```ts
|
|
187
|
-
// JavaScript engine
|
|
334
|
+
// JavaScript engine for explicit plugins and smaller bundles
|
|
188
335
|
import { htmlToMarkdown } from '@mdream/js'
|
|
189
336
|
|
|
190
337
|
// Rust NAPI engine (auto-selected in Node.js)
|
|
191
|
-
import { htmlToMarkdown } from 'mdream'
|
|
338
|
+
import { htmlToMarkdown as htmlToMarkdownRust } from 'mdream'
|
|
192
339
|
```
|
|
193
340
|
|
|
194
|
-
|
|
341
|
+
The Rust engine takes declarative options: `minimal`, `format`, `frontmatter`, `isolateMain`, `tailwind`, `filter`, and `extraction`.
|
|
342
|
+
Pass them at the top level.
|
|
343
|
+
|
|
344
|
+
The JS engine takes an array of plugin instances in `options.plugins`.
|
|
345
|
+
Each output format has its own entry point.
|
|
346
|
+
For its minimal preset, use `withMinimalPreset()` from `@mdream/js/preset/minimal`.
|
|
347
|
+
If you pass Rust-only options to the JS engine, it throws a `TypeError` that names the fix.
|
|
195
348
|
|
|
196
349
|
## Options
|
|
197
350
|
|
|
@@ -251,20 +404,23 @@ interface MdreamOptions {
|
|
|
251
404
|
}
|
|
252
405
|
```
|
|
253
406
|
|
|
407
|
+
Any other key throws a `TypeError` that names the key and the fix.
|
|
408
|
+
|
|
254
409
|
### MdreamOptions (JS engine)
|
|
255
410
|
|
|
256
|
-
The JS engine
|
|
411
|
+
The JS engine uses explicit plugins. Each output format has its own entry point.
|
|
257
412
|
|
|
258
413
|
```ts
|
|
259
414
|
interface MdreamOptions extends EngineOptions {
|
|
260
|
-
/**
|
|
261
|
-
|
|
415
|
+
/** Explicit plugins, applied in array order. */
|
|
416
|
+
plugins?: Plugin[]
|
|
262
417
|
}
|
|
263
418
|
|
|
264
419
|
interface EngineOptions {
|
|
265
420
|
origin?: string
|
|
266
|
-
clean
|
|
267
|
-
|
|
421
|
+
/** Cleanup rules from clean() in @mdream/js/clean */
|
|
422
|
+
clean?: Cleaner
|
|
423
|
+
tagOverrides?: Record<string, TagOverride | string>
|
|
268
424
|
|
|
269
425
|
/**
|
|
270
426
|
* Hard-wrap prose at this many characters, breaking on word boundaries.
|
|
@@ -273,26 +429,18 @@ interface EngineOptions {
|
|
|
273
429
|
*/
|
|
274
430
|
wrapWidth?: number
|
|
275
431
|
|
|
276
|
-
/** Output Markdown, plain text, or HTML. Default: 'markdown' */
|
|
277
|
-
format?: 'markdown' | 'text' | 'html'
|
|
278
|
-
}
|
|
279
|
-
|
|
280
|
-
interface BuiltinPlugins {
|
|
281
|
-
filter?: false | { include?: (string | number)[], exclude?: (string | number)[], processChildren?: boolean }
|
|
282
|
-
frontmatter?: boolean | ((fm: Record<string, string>) => void) | FrontmatterConfig
|
|
283
|
-
isolateMain?: boolean
|
|
284
|
-
tailwind?: boolean
|
|
285
|
-
extraction?: Record<string, (element: ExtractedElement) => void>
|
|
286
|
-
tagOverrides?: Record<string, TagOverride | string>
|
|
287
432
|
}
|
|
288
433
|
```
|
|
289
434
|
|
|
290
|
-
|
|
435
|
+
Use `@mdream/js/text` for plain text. Use `@mdream/js/html` for safe HTML.
|
|
436
|
+
Any other key throws a `TypeError` that names the key and the fix.
|
|
291
437
|
|
|
292
438
|
### CleanOptions
|
|
293
439
|
|
|
294
440
|
Post-processing cleanup applied to the final Markdown output. All options default to `false` unless `clean: true` is set.
|
|
295
441
|
|
|
442
|
+
With `@mdream/js`, pass these rules to `clean()` from `@mdream/js/clean`, for example `clean: clean({ urls: true })`.
|
|
443
|
+
|
|
296
444
|
```ts
|
|
297
445
|
interface CleanOptions {
|
|
298
446
|
/** Strip tracking query parameters (utm_*, fbclid, gclid, etc.) from URLs */
|
|
@@ -301,8 +449,6 @@ interface CleanOptions {
|
|
|
301
449
|
fragments?: boolean
|
|
302
450
|
/** Strip links with meaningless hrefs (#, javascript:void(0)) to plain text */
|
|
303
451
|
emptyLinks?: boolean
|
|
304
|
-
/** Collapse 3+ consecutive blank lines to 2 */
|
|
305
|
-
blankLines?: boolean
|
|
306
452
|
/** Strip links where text equals URL: [https://x.com](https://x.com) becomes https://x.com */
|
|
307
453
|
redundantLinks?: boolean
|
|
308
454
|
/** Strip self-referencing heading anchors: ## [Title](#title) becomes ## Title */
|
|
@@ -317,6 +463,8 @@ interface CleanOptions {
|
|
|
317
463
|
**Example:**
|
|
318
464
|
|
|
319
465
|
```ts
|
|
466
|
+
import { htmlToMarkdown } from 'mdream'
|
|
467
|
+
|
|
320
468
|
const markdown = htmlToMarkdown(html, {
|
|
321
469
|
clean: {
|
|
322
470
|
urls: true,
|
|
@@ -376,6 +524,8 @@ interface TagOverride {
|
|
|
376
524
|
**Example:**
|
|
377
525
|
|
|
378
526
|
```ts
|
|
527
|
+
import { htmlToMarkdown } from 'mdream'
|
|
528
|
+
|
|
379
529
|
const markdown = htmlToMarkdown(html, {
|
|
380
530
|
tagOverrides: {
|
|
381
531
|
// Treat <x-heading> like <h2>
|
|
@@ -393,6 +543,8 @@ const markdown = htmlToMarkdown(html, {
|
|
|
393
543
|
**Output is GitHub Flavored Markdown.** Mdream emits a fixed GFM dialect tuned for LLM input: ATX headings (`#`), fenced code blocks (` ``` `), `-` bullets, `*` emphasis, `**` strong, `---` horizontal rules, inline links. These are not configurable. For simple delimiter swaps you can use `tagOverrides`:
|
|
394
544
|
|
|
395
545
|
```ts
|
|
546
|
+
import { htmlToMarkdown } from 'mdream'
|
|
547
|
+
|
|
396
548
|
htmlToMarkdown(html, {
|
|
397
549
|
tagOverrides: {
|
|
398
550
|
em: { enter: '_', exit: '_', isInline: true }, // *x* → _x_
|
|
@@ -408,10 +560,10 @@ Structural style differences (setext headings, indented code blocks, reference-s
|
|
|
408
560
|
|
|
409
561
|
```ts
|
|
410
562
|
interface FilterOptions {
|
|
411
|
-
/** CSS selectors
|
|
412
|
-
include?:
|
|
413
|
-
/** CSS selectors
|
|
414
|
-
exclude?:
|
|
563
|
+
/** CSS selectors or tag names for elements to include (all others excluded) */
|
|
564
|
+
include?: string[]
|
|
565
|
+
/** CSS selectors or tag names for elements to exclude */
|
|
566
|
+
exclude?: string[]
|
|
415
567
|
/** Whether to also process children of matched elements. Default: true */
|
|
416
568
|
processChildren?: boolean
|
|
417
569
|
}
|
|
@@ -429,9 +581,11 @@ The `minimal` preset enables the following plugins together:
|
|
|
429
581
|
- **filter**: Excludes `form`, `fieldset`, `object`, `embed`, `footer`, `aside`, `iframe`, `input`, `textarea`, `select`, `button`, `nav`
|
|
430
582
|
- **clean**: All post-processing cleanup enabled
|
|
431
583
|
|
|
432
|
-
A `filter` passed with `minimal` adds to the preset's excludes. It does not replace them. To turn the preset's filter off, pass `filter: false`. `
|
|
584
|
+
A `filter` passed with `minimal` adds to the preset's excludes. It does not replace them. To turn the preset's filter off, pass `filter: false`. In the JS engine, a `filterPlugin()` appended to `withMinimalPreset()` also adds to the excludes.
|
|
433
585
|
|
|
434
586
|
```ts
|
|
587
|
+
import { htmlToMarkdown } from 'mdream'
|
|
588
|
+
|
|
435
589
|
// Excludes h1 as well as form, nav, footer, and the rest
|
|
436
590
|
htmlToMarkdown(html, { minimal: true, filter: { exclude: ['h1'] } })
|
|
437
591
|
|
|
@@ -461,20 +615,21 @@ const markdown = htmlToMarkdown(html, withMinimalPreset({
|
|
|
461
615
|
}))
|
|
462
616
|
```
|
|
463
617
|
|
|
464
|
-
`withMinimalPreset()` returns
|
|
618
|
+
`withMinimalPreset()` returns explicit plugin defaults. You can append custom plugins. An appended `filterPlugin()` adds to the preset's excludes:
|
|
465
619
|
|
|
466
620
|
```ts
|
|
621
|
+
import { htmlToMarkdown } from '@mdream/js'
|
|
622
|
+
import { filterPlugin } from '@mdream/js/plugins'
|
|
623
|
+
import { withMinimalPreset } from '@mdream/js/preset/minimal'
|
|
624
|
+
|
|
467
625
|
const markdown = htmlToMarkdown(html, withMinimalPreset({
|
|
468
|
-
plugins: {
|
|
469
|
-
frontmatter: false,
|
|
470
|
-
filter: { exclude: ['.cookie-banner'] },
|
|
471
|
-
},
|
|
626
|
+
plugins: [filterPlugin({ exclude: ['.cookie-banner'] }), myPlugin],
|
|
472
627
|
}))
|
|
473
628
|
```
|
|
474
629
|
|
|
475
630
|
## Built-in Plugins
|
|
476
631
|
|
|
477
|
-
|
|
632
|
+
The Rust engine uses declarative options. The JS engine exports matching plugin factories.
|
|
478
633
|
|
|
479
634
|
### Frontmatter Plugin
|
|
480
635
|
|
|
@@ -483,6 +638,8 @@ Extracts metadata from the HTML `<head>` element and generates YAML frontmatter.
|
|
|
483
638
|
**Extracted fields by default:** `title`, `description`, `keywords`, `author`, `date`, `og:title`, `og:description`, `twitter:title`, `twitter:description`.
|
|
484
639
|
|
|
485
640
|
```ts
|
|
641
|
+
import { htmlToMarkdown } from 'mdream'
|
|
642
|
+
|
|
486
643
|
// Enable with defaults
|
|
487
644
|
htmlToMarkdown(html, { frontmatter: true })
|
|
488
645
|
|
|
@@ -525,6 +682,8 @@ Isolates the main content area using the following priority:
|
|
|
525
682
|
4. The `<head>` section is always passed through for other plugins (e.g., frontmatter)
|
|
526
683
|
|
|
527
684
|
```ts
|
|
685
|
+
import { htmlToMarkdown } from 'mdream'
|
|
686
|
+
|
|
528
687
|
htmlToMarkdown(html, { isolateMain: true })
|
|
529
688
|
```
|
|
530
689
|
|
|
@@ -543,14 +702,18 @@ Converts Tailwind CSS utility classes to semantic Markdown formatting:
|
|
|
543
702
|
Supports responsive breakpoint prefixes (`sm:`, `md:`, `lg:`, `xl:`, `2xl:`) with mobile-first resolution.
|
|
544
703
|
|
|
545
704
|
```ts
|
|
705
|
+
import { htmlToMarkdown } from 'mdream'
|
|
706
|
+
|
|
546
707
|
htmlToMarkdown(html, { tailwind: true })
|
|
547
708
|
```
|
|
548
709
|
|
|
549
710
|
### Filter Plugin
|
|
550
711
|
|
|
551
|
-
Filters HTML elements by CSS selectors
|
|
712
|
+
Filters HTML elements by CSS selectors or tag names.
|
|
552
713
|
|
|
553
714
|
```ts
|
|
715
|
+
import { htmlToMarkdown } from 'mdream'
|
|
716
|
+
|
|
554
717
|
// Exclude navigation, sidebar, footer
|
|
555
718
|
htmlToMarkdown(html, {
|
|
556
719
|
filter: {
|
|
@@ -566,15 +729,14 @@ htmlToMarkdown(html, {
|
|
|
566
729
|
})
|
|
567
730
|
```
|
|
568
731
|
|
|
569
|
-
The JS engine
|
|
732
|
+
The JS engine takes the same strings in `filterPlugin`. If an entry is not a string, `filterPlugin` throws a `TypeError`.
|
|
570
733
|
|
|
571
734
|
```ts
|
|
572
|
-
import {
|
|
735
|
+
import { htmlToMarkdown } from '@mdream/js'
|
|
736
|
+
import { filterPlugin } from '@mdream/js/plugins'
|
|
573
737
|
|
|
574
738
|
htmlToMarkdown(html, {
|
|
575
|
-
plugins: {
|
|
576
|
-
filter: { exclude: [TAG_NAV, TAG_FOOTER] },
|
|
577
|
-
},
|
|
739
|
+
plugins: [filterPlugin({ exclude: ['nav', 'footer'] })],
|
|
578
740
|
})
|
|
579
741
|
```
|
|
580
742
|
|
|
@@ -585,6 +747,8 @@ Elements with `style="position: absolute"` or `style="position: fixed"` are also
|
|
|
585
747
|
Extracts elements matching CSS selectors during conversion. Callbacks receive the matched element with its accumulated text content and attributes.
|
|
586
748
|
|
|
587
749
|
```ts
|
|
750
|
+
import { htmlToMarkdown } from 'mdream'
|
|
751
|
+
|
|
588
752
|
htmlToMarkdown(html, {
|
|
589
753
|
extraction: {
|
|
590
754
|
'h2': (el) => {
|
|
@@ -611,9 +775,9 @@ interface ExtractedElement {
|
|
|
611
775
|
}
|
|
612
776
|
```
|
|
613
777
|
|
|
614
|
-
##
|
|
778
|
+
## Plugins (JS Engine)
|
|
615
779
|
|
|
616
|
-
The JS engine
|
|
780
|
+
The JS engine composes plugins in an explicit array.
|
|
617
781
|
|
|
618
782
|
```ts
|
|
619
783
|
import { htmlToMarkdown } from '@mdream/js'
|
|
@@ -631,19 +795,30 @@ const myPlugin = createPlugin({
|
|
|
631
795
|
},
|
|
632
796
|
})
|
|
633
797
|
|
|
634
|
-
const markdown = htmlToMarkdown(html, {
|
|
798
|
+
const markdown = htmlToMarkdown(html, { plugins: [myPlugin] })
|
|
635
799
|
```
|
|
636
800
|
|
|
637
801
|
### Plugin Hooks
|
|
638
802
|
|
|
803
|
+
Each hook gets a read-only `PluginState`. To keep data for a node, set it on `node.context`.
|
|
804
|
+
|
|
639
805
|
```ts
|
|
806
|
+
interface PluginState {
|
|
807
|
+
/** Options passed to the converter. */
|
|
808
|
+
readonly options: Readonly<EngineOptions>
|
|
809
|
+
/** Output that the running converter writes. */
|
|
810
|
+
readonly outputFormat: 'markdown' | 'text' | 'html'
|
|
811
|
+
/** Nesting depth of the current node. It is 0 before the first node. */
|
|
812
|
+
readonly depth: number
|
|
813
|
+
}
|
|
814
|
+
|
|
640
815
|
interface TransformPlugin {
|
|
641
816
|
/**
|
|
642
817
|
* Called before any node processing. Return { skip: true } to skip the node.
|
|
643
818
|
*/
|
|
644
819
|
beforeNodeProcess?: (
|
|
645
820
|
event: NodeEvent,
|
|
646
|
-
state:
|
|
821
|
+
state: PluginState,
|
|
647
822
|
) => undefined | void | { skip: boolean }
|
|
648
823
|
|
|
649
824
|
/**
|
|
@@ -652,7 +827,7 @@ interface TransformPlugin {
|
|
|
652
827
|
*/
|
|
653
828
|
onNodeEnter?: (
|
|
654
829
|
node: ElementNode,
|
|
655
|
-
state:
|
|
830
|
+
state: PluginState,
|
|
656
831
|
) => string | undefined | void
|
|
657
832
|
|
|
658
833
|
/**
|
|
@@ -661,7 +836,7 @@ interface TransformPlugin {
|
|
|
661
836
|
*/
|
|
662
837
|
onNodeExit?: (
|
|
663
838
|
node: ElementNode,
|
|
664
|
-
state:
|
|
839
|
+
state: PluginState,
|
|
665
840
|
) => string | undefined | void
|
|
666
841
|
|
|
667
842
|
/**
|
|
@@ -669,7 +844,7 @@ interface TransformPlugin {
|
|
|
669
844
|
*/
|
|
670
845
|
processAttributes?: (
|
|
671
846
|
node: ElementNode,
|
|
672
|
-
state:
|
|
847
|
+
state: PluginState,
|
|
673
848
|
) => void
|
|
674
849
|
|
|
675
850
|
/**
|
|
@@ -678,8 +853,13 @@ interface TransformPlugin {
|
|
|
678
853
|
*/
|
|
679
854
|
processTextNode?: (
|
|
680
855
|
node: TextNode,
|
|
681
|
-
state:
|
|
856
|
+
state: PluginState,
|
|
682
857
|
) => { content: string, skip: boolean } | undefined
|
|
858
|
+
|
|
859
|
+
/**
|
|
860
|
+
* Called once after the whole document is converted, including for streams.
|
|
861
|
+
*/
|
|
862
|
+
onDocumentEnd?: (state: PluginState) => void
|
|
683
863
|
}
|
|
684
864
|
```
|
|
685
865
|
|
|
@@ -688,13 +868,18 @@ interface TransformPlugin {
|
|
|
688
868
|
A typed identity function for creating plugins with full TypeScript inference:
|
|
689
869
|
|
|
690
870
|
```ts
|
|
871
|
+
import type { ElementNode } from '@mdream/js'
|
|
872
|
+
import { ELEMENT_NODE } from '@mdream/js'
|
|
691
873
|
import { createPlugin } from '@mdream/js/plugins'
|
|
692
874
|
|
|
693
875
|
const plugin = createPlugin({
|
|
694
876
|
beforeNodeProcess({ node }) {
|
|
695
877
|
// Skip all div elements with class "ad"
|
|
696
|
-
if (node.type ===
|
|
697
|
-
|
|
878
|
+
if (node.type === ELEMENT_NODE) {
|
|
879
|
+
const element = node as ElementNode
|
|
880
|
+
if (element.name === 'div' && element.attributes.class?.split(/\s+/).includes('ad')) {
|
|
881
|
+
return { skip: true }
|
|
882
|
+
}
|
|
698
883
|
}
|
|
699
884
|
},
|
|
700
885
|
})
|
|
@@ -707,7 +892,6 @@ The following plugin factory functions are available from `@mdream/js/plugins`:
|
|
|
707
892
|
```ts
|
|
708
893
|
import {
|
|
709
894
|
createPlugin,
|
|
710
|
-
extractionCollectorPlugin,
|
|
711
895
|
extractionPlugin,
|
|
712
896
|
filterPlugin,
|
|
713
897
|
frontmatterPlugin,
|
|
@@ -725,7 +909,6 @@ Available from `@mdream/js/splitter`.
|
|
|
725
909
|
### Basic Chunking
|
|
726
910
|
|
|
727
911
|
```ts
|
|
728
|
-
import { TAG_H2 } from '@mdream/js'
|
|
729
912
|
import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
|
|
730
913
|
|
|
731
914
|
const html = `
|
|
@@ -737,7 +920,7 @@ const html = `
|
|
|
737
920
|
`
|
|
738
921
|
|
|
739
922
|
const chunks = htmlToMarkdownSplitChunks(html, {
|
|
740
|
-
headersToSplitOn: [
|
|
923
|
+
headersToSplitOn: [2],
|
|
741
924
|
chunkSize: 1000,
|
|
742
925
|
chunkOverlap: 200,
|
|
743
926
|
stripHeaders: true,
|
|
@@ -753,7 +936,10 @@ chunks.forEach((chunk) => {
|
|
|
753
936
|
|
|
754
937
|
### Streaming Chunks (Memory Efficient)
|
|
755
938
|
|
|
756
|
-
|
|
939
|
+
The generator converts the whole document before yielding its first chunk.
|
|
940
|
+
It keeps the converted Markdown in memory, but avoids collecting the chunk array.
|
|
941
|
+
Stopping early skips the remaining chunk work. HTML conversion has already finished.
|
|
942
|
+
See [Splitter output and timing](../js/README.md#splitter-output-and-timing) for migration details.
|
|
757
943
|
|
|
758
944
|
```ts
|
|
759
945
|
import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'
|
|
@@ -774,14 +960,14 @@ interface SplitterOptions {
|
|
|
774
960
|
// --- Structural splitting ---
|
|
775
961
|
|
|
776
962
|
/**
|
|
777
|
-
*
|
|
778
|
-
* Default: [
|
|
963
|
+
* Heading levels that start a new chunk, from 1 (<h1>) to 6 (<h6>).
|
|
964
|
+
* Another value throws a TypeError. Default: [2, 3, 4, 5, 6]
|
|
779
965
|
*/
|
|
780
|
-
headersToSplitOn?:
|
|
966
|
+
headersToSplitOn?: (1 | 2 | 3 | 4 | 5 | 6)[]
|
|
781
967
|
|
|
782
968
|
// --- Size-based splitting ---
|
|
783
969
|
|
|
784
|
-
/**
|
|
970
|
+
/** Target chunk size in characters. Complete code fences can exceed it. Default: 1000 */
|
|
785
971
|
chunkSize?: number
|
|
786
972
|
|
|
787
973
|
/** Overlap between chunks for context preservation. Default: 200 */
|
|
@@ -801,7 +987,7 @@ interface SplitterOptions {
|
|
|
801
987
|
/** Split into individual lines. Default: false */
|
|
802
988
|
returnEachLine?: boolean
|
|
803
989
|
|
|
804
|
-
/**
|
|
990
|
+
/** The splitter drops whitespace between chunks for either value. */
|
|
805
991
|
keepSeparator?: boolean
|
|
806
992
|
|
|
807
993
|
// --- Standard options ---
|
|
@@ -809,14 +995,14 @@ interface SplitterOptions {
|
|
|
809
995
|
/** Base URL for resolving relative links/images */
|
|
810
996
|
origin?: string
|
|
811
997
|
|
|
812
|
-
/**
|
|
813
|
-
plugins?:
|
|
998
|
+
/** Explicit plugins, applied in array order */
|
|
999
|
+
plugins?: Plugin[]
|
|
814
1000
|
|
|
815
|
-
/**
|
|
816
|
-
|
|
1001
|
+
/** Custom tag output or aliases */
|
|
1002
|
+
tagOverrides?: Record<string, TagOverride | string>
|
|
817
1003
|
|
|
818
|
-
/**
|
|
819
|
-
clean?:
|
|
1004
|
+
/** Cleanup rules from clean() in @mdream/js/clean */
|
|
1005
|
+
clean?: Cleaner
|
|
820
1006
|
}
|
|
821
1007
|
```
|
|
822
1008
|
|
|
@@ -832,7 +1018,7 @@ interface MarkdownChunk {
|
|
|
832
1018
|
headers?: Record<string, string> // { h1: "Title", h2: "Section" }
|
|
833
1019
|
/** Code block language if chunk contains code */
|
|
834
1020
|
code?: string
|
|
835
|
-
/**
|
|
1021
|
+
/** Inclusive, one-based line range in converted Markdown, excluding stripped headings */
|
|
836
1022
|
loc?: {
|
|
837
1023
|
lines: { from: number, to: number }
|
|
838
1024
|
}
|
|
@@ -845,12 +1031,11 @@ interface MarkdownChunk {
|
|
|
845
1031
|
Combine splitting with presets:
|
|
846
1032
|
|
|
847
1033
|
```ts
|
|
848
|
-
import { TAG_H2 } from '@mdream/js'
|
|
849
1034
|
import { withMinimalPreset } from '@mdream/js/preset/minimal'
|
|
850
1035
|
import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
|
|
851
1036
|
|
|
852
1037
|
const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
|
|
853
|
-
headersToSplitOn: [
|
|
1038
|
+
headersToSplitOn: [2],
|
|
854
1039
|
chunkSize: 500,
|
|
855
1040
|
origin: 'https://example.com',
|
|
856
1041
|
}))
|
|
@@ -861,18 +1046,17 @@ const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
|
|
|
861
1046
|
The `@mdream/js/negotiate` module provides HTTP content negotiation utilities for serving Markdown to LLM clients:
|
|
862
1047
|
|
|
863
1048
|
```ts
|
|
864
|
-
import {
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
)
|
|
871
|
-
|
|
872
|
-
|
|
873
|
-
|
|
874
|
-
|
|
875
|
-
})
|
|
1049
|
+
import { shouldServeMarkdown } from '@mdream/js/negotiate'
|
|
1050
|
+
|
|
1051
|
+
export function markdownResponse(request: Request, markdown: string): Response | undefined {
|
|
1052
|
+
if (shouldServeMarkdown(
|
|
1053
|
+
request.headers.get('accept') ?? undefined,
|
|
1054
|
+
request.headers.get('sec-fetch-dest') ?? undefined,
|
|
1055
|
+
)) {
|
|
1056
|
+
return new Response(markdown, {
|
|
1057
|
+
headers: { 'Content-Type': 'text/markdown' },
|
|
1058
|
+
})
|
|
1059
|
+
}
|
|
876
1060
|
}
|
|
877
1061
|
```
|
|
878
1062
|
|
|
@@ -883,14 +1067,16 @@ if (serveMarkdown) {
|
|
|
883
1067
|
If you only need to parse HTML into a DOM-like event stream without converting to Markdown, use `parseHtml` from the JS engine:
|
|
884
1068
|
|
|
885
1069
|
```ts
|
|
1070
|
+
import type { ElementNode } from '@mdream/js'
|
|
1071
|
+
import { ELEMENT_NODE, NodeEventEnter } from '@mdream/js'
|
|
886
1072
|
import { parseHtml } from '@mdream/js/parse'
|
|
887
1073
|
|
|
888
1074
|
const html = '<div><h1>Title</h1><p>Content</p></div>'
|
|
889
1075
|
const { events, remainingHtml } = parseHtml(html)
|
|
890
1076
|
|
|
891
1077
|
events.forEach((event) => {
|
|
892
|
-
if (event.type ===
|
|
893
|
-
console.log('Entering element:', event.node.name)
|
|
1078
|
+
if (event.type === NodeEventEnter && event.node.type === ELEMENT_NODE) {
|
|
1079
|
+
console.log('Entering element:', (event.node as ElementNode).name)
|
|
894
1080
|
}
|
|
895
1081
|
})
|
|
896
1082
|
```
|
|
@@ -949,18 +1135,27 @@ cat index.html \
|
|
|
949
1135
|
| `-h`, `--help` | Display help information |
|
|
950
1136
|
|
|
951
1137
|
The CLI reads HTML from stdin and writes Markdown, plain text, or HTML to stdout. It uses the streaming API internally.
|
|
1138
|
+
A value option also accepts the `--origin=<url>` form.
|
|
1139
|
+
If you pass an unknown flag, a flag without a value, or an unknown preset, the CLI writes one line to stderr and exits with code 1.
|
|
952
1140
|
|
|
953
1141
|
## Browser and Edge Usage
|
|
954
1142
|
|
|
955
|
-
Every entry takes the same `MdreamOptions` and produces the same Markdown string.
|
|
1143
|
+
Every entry takes the same `MdreamOptions` and produces the same Markdown string.
|
|
1144
|
+
In Node and edge runtimes, `htmlToMarkdown` from `mdream` is synchronous.
|
|
1145
|
+
In a browser bundle, import from `mdream/browser`.
|
|
1146
|
+
Its `htmlToMarkdown` returns `Promise<string>`, because the WASM binary loads first.
|
|
1147
|
+
The CDN script also returns `Promise<string>`.
|
|
956
1148
|
|
|
957
1149
|
```ts
|
|
958
|
-
import { htmlToMarkdown } from 'mdream'
|
|
1150
|
+
import { htmlToMarkdown } from 'mdream/browser'
|
|
959
1151
|
|
|
960
|
-
// In a browser bundle
|
|
961
1152
|
const markdown = await htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
|
|
962
1153
|
```
|
|
963
1154
|
|
|
1155
|
+
In a browser bundle, the root `mdream` entry throws a `TypeError` when you call it.
|
|
1156
|
+
The `mdream` types always describe the synchronous API.
|
|
1157
|
+
To convert off the main thread, see [Web Worker](#web-worker).
|
|
1158
|
+
|
|
964
1159
|
### Edge / Cloudflare Workers
|
|
965
1160
|
|
|
966
1161
|
For edge runtimes (Cloudflare Workers, Vercel Edge), `mdream` automatically selects the WASM build via export conditions (`workerd`, `edge-light`). Both `htmlToMarkdown` and `streamHtmlToMarkdown` are available:
|
|
@@ -978,20 +1173,31 @@ for await (const chunk of streamHtmlToMarkdown(response.body)) {
|
|
|
978
1173
|
}
|
|
979
1174
|
```
|
|
980
1175
|
|
|
981
|
-
|
|
1176
|
+
You can catch conversion errors.
|
|
1177
|
+
An internal Rust panic aborts the call because WASM has no unwinding.
|
|
1178
|
+
The wrapper preserves the panic message in an `Error`, with the raw WASM trap as `cause`.
|
|
1179
|
+
The instance remains usable for later requests:
|
|
982
1180
|
|
|
983
1181
|
```ts
|
|
984
|
-
|
|
985
|
-
|
|
986
|
-
|
|
987
|
-
|
|
988
|
-
|
|
989
|
-
|
|
990
|
-
|
|
1182
|
+
import { htmlToMarkdown } from 'mdream'
|
|
1183
|
+
|
|
1184
|
+
export function convertResponse(html: string): Response {
|
|
1185
|
+
try {
|
|
1186
|
+
return new Response(htmlToMarkdown(html))
|
|
1187
|
+
}
|
|
1188
|
+
catch (error) {
|
|
1189
|
+
console.error(error)
|
|
1190
|
+
return new Response('Conversion failed', { status: 500 })
|
|
1191
|
+
}
|
|
991
1192
|
}
|
|
992
1193
|
```
|
|
993
1194
|
|
|
994
|
-
If your toolchain
|
|
1195
|
+
If your toolchain does not resolve export conditions, import the web-target wasm-bindgen build from `mdream/wasm`.
|
|
1196
|
+
Initialize it manually before converting.
|
|
1197
|
+
Its `htmlToMarkdown` takes the same `MdreamOptions` as `mdream`.
|
|
1198
|
+
Its `MarkdownStream` runs callbacks in `finish()`.
|
|
1199
|
+
|
|
1200
|
+
Your bundler must support `.wasm` imports and provide their TypeScript declarations.
|
|
995
1201
|
|
|
996
1202
|
```ts
|
|
997
1203
|
import init, { htmlToMarkdown } from 'mdream/wasm'
|
|
@@ -1001,11 +1207,12 @@ await init({ module_or_path: wasmModule })
|
|
|
1001
1207
|
const markdown = htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
|
|
1002
1208
|
```
|
|
1003
1209
|
|
|
1004
|
-
`mdream/worker` is not an edge entry. It runs the conversion in a browser Web Worker; see [Web Worker](#web-worker).
|
|
1005
|
-
|
|
1006
1210
|
### Browser CDN (IIFE)
|
|
1007
1211
|
|
|
1008
|
-
Use mdream
|
|
1212
|
+
Use mdream from a CDN without a build step.
|
|
1213
|
+
The script includes the WASM binary and initializes it on load.
|
|
1214
|
+
Call `window.mdream.htmlToMarkdown()` with the same `MdreamOptions` as `mdream`.
|
|
1215
|
+
Await its `Promise<string>`, as with `mdream/browser`.
|
|
1009
1216
|
|
|
1010
1217
|
```html
|
|
1011
1218
|
<script src="https://unpkg.com/mdream/dist/iife.js"></script>
|
|
@@ -1019,22 +1226,6 @@ Use mdream directly via CDN with no build step. The script inlines the WASM bina
|
|
|
1019
1226
|
- **unpkg**: `https://unpkg.com/mdream/dist/iife.js`
|
|
1020
1227
|
- **jsDelivr**: `https://cdn.jsdelivr.net/npm/mdream/dist/iife.js`
|
|
1021
1228
|
|
|
1022
|
-
### Web Worker
|
|
1023
|
-
|
|
1024
|
-
For browser environments, `mdream/worker` runs conversions off the main thread using a Web Worker:
|
|
1025
|
-
|
|
1026
|
-
```ts
|
|
1027
|
-
import { htmlToMarkdown, initWorker, terminateWorker } from 'mdream/worker'
|
|
1028
|
-
|
|
1029
|
-
await initWorker('/path/to/mdream_edge_bg.wasm')
|
|
1030
|
-
|
|
1031
|
-
// Takes the same options as the mdream entry. Callbacks run on this thread.
|
|
1032
|
-
const markdown = await htmlToMarkdown('<h1>Hello</h1>', { minimal: true })
|
|
1033
|
-
|
|
1034
|
-
// Clean up
|
|
1035
|
-
terminateWorker()
|
|
1036
|
-
```
|
|
1037
|
-
|
|
1038
1229
|
## Content Extraction with Readability
|
|
1039
1230
|
|
|
1040
1231
|
For advanced content extraction (article detection, boilerplate removal), use [@mozilla/readability](https://github.com/mozilla/readability) before mdream:
|
|
@@ -1047,7 +1238,7 @@ import { htmlToMarkdown } from 'mdream'
|
|
|
1047
1238
|
const dom = new JSDOM(html, { url: 'https://example.com' })
|
|
1048
1239
|
const article = new Readability(dom.window.document).parse()
|
|
1049
1240
|
|
|
1050
|
-
if (article) {
|
|
1241
|
+
if (article?.content) {
|
|
1051
1242
|
const markdown = htmlToMarkdown(article.content)
|
|
1052
1243
|
// article.title, article.excerpt, article.byline also available
|
|
1053
1244
|
}
|
|
@@ -1080,7 +1271,7 @@ console.log(result.llmsFullTxt) // llms-full.txt content
|
|
|
1080
1271
|
| Package | Description |
|
|
1081
1272
|
|---------|-------------|
|
|
1082
1273
|
| [`mdream`](https://npmjs.com/package/mdream) | Core HTML to Markdown converter (Rust + WASM engine) |
|
|
1083
|
-
| [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with
|
|
1274
|
+
| [`@mdream/js`](https://npmjs.com/package/@mdream/js) | JavaScript engine with tree-shakable formats, plugins, splitter, and llms.txt generation (`@mdream/js/llms-txt`) |
|
|
1084
1275
|
| [`@mdream/crawl`](https://github.com/harlan-zw/mdream/tree/main/packages/crawl) | Site-wide crawler for llms.txt generation |
|
|
1085
1276
|
| [`@mdream/vite`](https://github.com/harlan-zw/mdream/tree/main/packages/vite) | Vite plugin integration |
|
|
1086
1277
|
| [`@mdream/nuxt`](https://github.com/harlan-zw/mdream/tree/main/packages/nuxt) | Nuxt module integration |
|