@mdream/js 1.0.0-beta.10 → 1.0.0-beta.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -4,16 +4,17 @@
4
4
  [![npm downloads][npm-downloads-src]][npm-downloads-href]
5
5
  [![License][license-src]][license-href]
6
6
 
7
- JavaScript HTML-to-Markdown engine for mdream. Use this package when you need plugin hooks, custom transform plugins, or are targeting edge runtimes where the native Rust engine cannot run.
8
-
9
- This package consolidates functionality previously split across `@mdream/core`, `@mdream/shared`, and `@mdream/llms-txt`.
10
-
11
- > For most use cases, prefer the main `mdream` package which uses the Rust engine for significantly better performance. Reach for `@mdream/js` when you need hooks, custom plugins, or edge runtime compatibility.
12
-
13
7
  ## Installation
14
8
 
15
9
  ```bash
16
- pnpm add @mdream/js
10
+ # pnpm
11
+ pnpm add @mdream/js@beta
12
+
13
+ # npm
14
+ npm install @mdream/js@beta
15
+
16
+ # yarn
17
+ yarn add @mdream/js@beta
17
18
  ```
18
19
 
19
20
  ## Entry Points
@@ -21,16 +22,18 @@ pnpm add @mdream/js
21
22
  | Import | Description |
22
23
  |---|---|
23
24
  | `@mdream/js` | Core `htmlToMarkdown` and `streamHtmlToMarkdown` APIs |
24
- | `@mdream/js/plugins` | Plugin utilities: `createPlugin`, `extractionPlugin`, `filterPlugin`, `frontmatterPlugin`, `isolateMainPlugin`, `tailwindPlugin` |
25
- | `@mdream/js/preset/minimal` | `withMinimalPreset` -- declarative config for frontmatter, isolateMain, tailwind, and filter plugins |
25
+ | `@mdream/js/plugins` | Plugin utilities: `createPlugin`, `extractionPlugin`, `extractionCollectorPlugin`, `filterPlugin`, `frontmatterPlugin`, `isolateMainPlugin`, `tailwindPlugin` |
26
+ | `@mdream/js/preset/minimal` | `withMinimalPreset` for declarative config combining frontmatter, isolateMain, tailwind, and filter plugins |
26
27
  | `@mdream/js/negotiate` | HTTP content negotiation: `shouldServeMarkdown`, `parseAcceptHeader` |
27
28
  | `@mdream/js/parse` | Low-level HTML parser: `parseHtml`, `parseHtmlStream` |
28
29
  | `@mdream/js/splitter` | Single-pass markdown splitter: `htmlToMarkdownSplitChunks`, `htmlToMarkdownSplitChunksStream` |
29
30
  | `@mdream/js/llms-txt` | llms.txt artifact generation: `generateLlmsTxtArtifacts`, `createLlmsTxtStream` |
30
31
 
31
- ## Usage
32
+ ## API Reference
33
+
34
+ ### `htmlToMarkdown(html, options?)`
32
35
 
33
- ### Basic Conversion
36
+ Converts an HTML string to Markdown synchronously.
34
37
 
35
38
  ```typescript
36
39
  import { htmlToMarkdown } from '@mdream/js'
@@ -39,7 +42,18 @@ const md = htmlToMarkdown('<h1>Hello</h1><p>World</p>')
39
42
  // # Hello\n\nWorld
40
43
  ```
41
44
 
42
- ### Streaming
45
+ **Parameters:**
46
+
47
+ | Parameter | Type | Description |
48
+ |---|---|---|
49
+ | `html` | `string` | The HTML string to convert |
50
+ | `options` | `Partial<MdreamOptions>` | Optional configuration (see [MdreamOptions](#mdreamoptions)) |
51
+
52
+ **Returns:** `string`
53
+
54
+ ### `streamHtmlToMarkdown(htmlStream, options?)`
55
+
56
+ Converts an HTML stream to Markdown incrementally. Useful for large documents or streaming HTTP responses.
43
57
 
44
58
  ```typescript
45
59
  import { streamHtmlToMarkdown } from '@mdream/js'
@@ -53,25 +67,221 @@ for await (const chunk of stream) {
53
67
  }
54
68
  ```
55
69
 
56
- ### With Plugins
70
+ **Parameters:**
71
+
72
+ | Parameter | Type | Description |
73
+ |---|---|---|
74
+ | `htmlStream` | `ReadableStream<Uint8Array \| string> \| null` | A web `ReadableStream` of HTML content |
75
+ | `options` | `Partial<MdreamOptions>` | Optional configuration (see [MdreamOptions](#mdreamoptions)) |
76
+
77
+ **Returns:** `AsyncIterable<string>`
78
+
79
+ ---
80
+
81
+ ## Options
82
+
83
+ ### `MdreamOptions`
84
+
85
+ | Option | Type | Default | Description |
86
+ |---|---|---|---|
87
+ | `origin` | `string` | `undefined` | Origin URL for resolving relative image paths and internal links |
88
+ | `plugins` | `BuiltinPlugins` | `undefined` | Declarative built-in plugin configuration (see [BuiltinPlugins](#builtinplugins)) |
89
+ | `clean` | `boolean \| CleanOptions` | `undefined` | Post-processing cleanup. Pass `true` for all cleanup rules or an object for specific features (see [CleanOptions](#cleanoptions)). Sync API only for `fragments`. |
90
+ | `hooks` | `TransformPlugin[]` | `undefined` | Imperative hook-based transform plugins for custom behavior (see [Plugins](#plugins)) |
91
+
92
+ ### `BuiltinPlugins`
93
+
94
+ Declarative configuration for built-in plugins. Works with both the JavaScript and Rust engines.
95
+
96
+ | Option | Type | Default | Description |
97
+ |---|---|---|---|
98
+ | `frontmatter` | `boolean \| ((fm: Record<string, string>) => void) \| FrontmatterConfig` | `undefined` | Extract metadata from HTML `<head>` into YAML frontmatter. Pass `true` for defaults, a callback to receive structured data, or a config object. |
99
+ | `isolateMain` | `boolean` | `undefined` | Isolate main content area. Prioritizes `<main>` elements, then falls back to header-to-footer heuristic. |
100
+ | `tailwind` | `boolean` | `undefined` | Convert Tailwind utility classes (bold, italic, hidden, etc.) to semantic Markdown formatting. |
101
+ | `filter` | `{ include?, exclude?, processChildren? }` | `undefined` | Filter elements by CSS selectors, tag names, or TAG_* constants (see [Filter Plugin](#filterpluginoptions)). |
102
+ | `extraction` | `Record<string, (element: ExtractedElement) => void>` | `undefined` | Extract elements matching CSS selectors during conversion. Each key is a CSS selector; the handler is called for every match. |
103
+ | `tagOverrides` | `Record<string, TagOverride \| string>` | `undefined` | Declarative tag overrides. String values act as aliases (e.g., `{ "x-heading": "h2" }`). Object values override specific handler properties. |
104
+
105
+ ### `FrontmatterConfig`
106
+
107
+ | Option | Type | Default | Description |
108
+ |---|---|---|---|
109
+ | `additionalFields` | `Record<string, string>` | `undefined` | Extra key-value pairs to inject into the frontmatter |
110
+ | `metaFields` | `string[]` | `['description', 'keywords', 'author', 'date', 'og:title', 'og:description', 'twitter:title', 'twitter:description']` | Meta tag names to extract beyond the standard set |
111
+ | `onExtract` | `(frontmatter: Record<string, string>) => void` | `undefined` | Callback to receive structured frontmatter data after conversion |
112
+
113
+ ### `TagOverride`
114
+
115
+ | Option | Type | Description |
116
+ |---|---|---|
117
+ | `enter` | `string` | Markdown string to emit when entering the tag |
118
+ | `exit` | `string` | Markdown string to emit when exiting the tag |
119
+ | `spacing` | `[number, number]` | Newlines to add `[before, after]` the tag |
120
+ | `isInline` | `boolean` | Whether the tag is inline (no block-level spacing) |
121
+ | `isSelfClosing` | `boolean` | Whether the tag is self-closing |
122
+ | `collapsesInnerWhiteSpace` | `boolean` | Whether inner whitespace should be collapsed |
123
+
124
+ ### `CleanOptions`
125
+
126
+ Post-processing cleanup options. Pass `true` to `clean` to enable all of these.
127
+
128
+ | Option | Type | Default | Description |
129
+ |---|---|---|---|
130
+ | `urls` | `boolean` | `false` | Strip tracking query parameters (`utm_*`, `fbclid`, `gclid`, etc.) from URLs |
131
+ | `fragments` | `boolean` | `false` | Strip fragment-only links that do not match any heading slug in the output |
132
+ | `emptyLinks` | `boolean` | `false` | Strip links with meaningless hrefs (`#`, `javascript:void(0)`, `data:`, `vbscript:`) and replace with plain text |
133
+ | `blankLines` | `boolean` | `false` | Collapse 3+ consecutive blank lines to 2 |
134
+ | `redundantLinks` | `boolean` | `false` | Strip links where text equals URL: `[https://x.com](https://x.com)` becomes `https://x.com` |
135
+ | `selfLinkHeadings` | `boolean` | `false` | Strip self-referencing heading anchors: `## [Title](#title)` becomes `## Title` |
136
+ | `emptyImages` | `boolean` | `false` | Strip images with no alt text (decorative images, tracking pixels) |
137
+ | `emptyLinkText` | `boolean` | `false` | Drop links that produce no visible text: `[](url)` is removed entirely |
138
+
139
+ When `clean: true` is passed, all options except `urls` and `blankLines` are enabled.
140
+
141
+ ---
142
+
143
+ ## Plugins
144
+
145
+ ### `createPlugin(plugin)`
146
+
147
+ Factory function for creating type-safe transform plugins. All hooks are optional.
57
148
 
58
149
  ```typescript
59
- import { htmlToMarkdown } from '@mdream/js'
60
150
  import { createPlugin } from '@mdream/js/plugins'
61
151
 
62
- const md = htmlToMarkdown(html, {
63
- hooks: [
64
- createPlugin({
65
- onNodeEnter(element) {
66
- if (element.name === 'aside')
67
- return '' // skip asides
68
- },
69
- }),
70
- ],
152
+ const myPlugin = createPlugin({
153
+ beforeNodeProcess(event, state) {
154
+ // Return { skip: true } to skip this node entirely
155
+ },
156
+ onNodeEnter(element, state) {
157
+ // Return a string to inject markdown at this position
158
+ },
159
+ onNodeExit(element, state) {
160
+ // Return a string to inject markdown at this position
161
+ },
162
+ processAttributes(element, state) {
163
+ // Inspect or modify element attributes
164
+ },
165
+ processTextNode(textNode, state) {
166
+ // Return { content: string, skip: boolean } to transform text
167
+ // Return undefined for no transformation
168
+ },
71
169
  })
72
170
  ```
73
171
 
74
- ### Minimal Preset
172
+ #### Plugin Hook Reference
173
+
174
+ | Hook | Parameters | Return Type | Description |
175
+ |---|---|---|---|
176
+ | `beforeNodeProcess` | `(event: NodeEvent, state)` | `{ skip: boolean } \| void` | Called before any node processing. Return `{ skip: true }` to skip the node. |
177
+ | `onNodeEnter` | `(element: ElementNode, state)` | `string \| void` | Called when entering an element. Return a string to inject markdown. |
178
+ | `onNodeExit` | `(element: ElementNode, state)` | `string \| void` | Called when exiting an element. Return a string to inject markdown. |
179
+ | `processAttributes` | `(element: ElementNode, state)` | `void` | Called to inspect or modify element attributes. |
180
+ | `processTextNode` | `(textNode: TextNode, state)` | `{ content: string, skip: boolean } \| void` | Called for each text node. Return an object to transform text or skip it. |
181
+
182
+ ### `filterPlugin(options)`
183
+
184
+ Filters elements by CSS selectors, tag names, or TAG_* constants.
185
+
186
+ ```typescript
187
+ import { TAG_FOOTER, TAG_NAV } from '@mdream/js'
188
+ import { filterPlugin } from '@mdream/js/plugins'
189
+
190
+ // Exclude navigation and footer
191
+ const plugin = filterPlugin({
192
+ exclude: [TAG_NAV, TAG_FOOTER, '.sidebar', '#ads'],
193
+ })
194
+
195
+ // Include only specific elements
196
+ const plugin2 = filterPlugin({
197
+ include: ['article', 'main'],
198
+ processChildren: true, // default: true
199
+ })
200
+ ```
201
+
202
+ | Option | Type | Default | Description |
203
+ |---|---|---|---|
204
+ | `include` | `(string \| number)[]` | `[]` | CSS selectors, tag names, or TAG_* constants for elements to include (all others excluded) |
205
+ | `exclude` | `(string \| number)[]` | `[]` | CSS selectors, tag names, or TAG_* constants for elements to exclude |
206
+ | `processChildren` | `boolean` | `true` | Whether to also process children of matching elements |
207
+
208
+ ### `frontmatterPlugin(options?)`
209
+
210
+ Extracts metadata from HTML `<head>` into YAML frontmatter. Collects `<title>` and `<meta>` tags.
211
+
212
+ ```typescript
213
+ import { frontmatterPlugin } from '@mdream/js/plugins'
214
+
215
+ const plugin = frontmatterPlugin({
216
+ additionalFields: { source: 'crawler' },
217
+ metaFields: ['robots', 'viewport'],
218
+ })
219
+ ```
220
+
221
+ Default meta fields extracted: `description`, `keywords`, `author`, `date`, `og:title`, `og:description`, `twitter:title`, `twitter:description`.
222
+
223
+ ### `isolateMainPlugin()`
224
+
225
+ Isolates the main content area of a page using a priority-based strategy:
226
+
227
+ 1. If an explicit `<main>` element exists (within 5 depth levels), use its content exclusively.
228
+ 2. Otherwise, find content between the first heading (`h1`-`h6`) that is not inside a `<header>` tag and the first `<footer>`.
229
+ 3. The `<head>` section is always passed through for other plugins (e.g., frontmatter extraction).
230
+
231
+ ```typescript
232
+ import { isolateMainPlugin } from '@mdream/js/plugins'
233
+
234
+ const plugin = isolateMainPlugin()
235
+ ```
236
+
237
+ ### `tailwindPlugin()`
238
+
239
+ Converts Tailwind utility classes to Markdown formatting. Supports mobile-first responsive breakpoints (`sm:`, `md:`, `lg:`, `xl:`, `2xl:`).
240
+
241
+ | Tailwind Class | Markdown Output |
242
+ |---|---|
243
+ | `font-bold`, `font-semibold`, `font-black`, `font-extrabold`, `font-medium` | `**text**` |
244
+ | `italic`, `font-italic` | `*text*` |
245
+ | `line-through` | `~~text~~` |
246
+ | `hidden`, `invisible` | Element skipped |
247
+ | `absolute`, `fixed`, `sticky` | Element skipped |
248
+
249
+ ```typescript
250
+ import { tailwindPlugin } from '@mdream/js/plugins'
251
+
252
+ const plugin = tailwindPlugin()
253
+ ```
254
+
255
+ ### `extractionPlugin(selectors)` (Deprecated)
256
+
257
+ > Deprecated. Use `plugins.extraction` config for declarative extraction that works with both JS and Rust engines.
258
+
259
+ Extracts elements matching CSS selectors during conversion. Callbacks receive matching elements with their accumulated text content.
260
+
261
+ ```typescript
262
+ import { extractionPlugin } from '@mdream/js/plugins'
263
+
264
+ const plugin = extractionPlugin({
265
+ 'h2': (element, state) => {
266
+ console.log('Heading:', element.textContent)
267
+ },
268
+ 'img[alt]': (element, state) => {
269
+ console.log('Image:', element.attributes.src)
270
+ },
271
+ })
272
+ ```
273
+
274
+ ### `extractionCollectorPlugin(selectors)`
275
+
276
+ Internal extraction collector for the `plugins.extraction` config. Collects results during processing and calls callbacks post-conversion to match Rust engine behavior.
277
+
278
+ ---
279
+
280
+ ## Presets
281
+
282
+ ### `withMinimalPreset(options?)`
283
+
284
+ Returns a declarative config combining frontmatter, isolateMain, tailwind, and filter plugins. Also enables `clean: true` by default. You can override any option.
75
285
 
76
286
  ```typescript
77
287
  import { htmlToMarkdown } from '@mdream/js'
@@ -82,18 +292,138 @@ const md = htmlToMarkdown(html, withMinimalPreset({
82
292
  }))
83
293
  ```
84
294
 
85
- ### Content Negotiation
295
+ The minimal preset excludes these elements by default: `<form>`, `<fieldset>`, `<object>`, `<embed>`, `<footer>`, `<aside>`, `<iframe>`, `<input>`, `<textarea>`, `<select>`, `<button>`, `<nav>`.
296
+
297
+ You can override or extend the plugin config:
298
+
299
+ ```typescript
300
+ const md = htmlToMarkdown(html, withMinimalPreset({
301
+ origin: 'https://example.com',
302
+ clean: { urls: true, fragments: true },
303
+ plugins: {
304
+ frontmatter: {
305
+ additionalFields: { source: 'my-crawler' },
306
+ },
307
+ },
308
+ }))
309
+ ```
310
+
311
+ ---
312
+
313
+ ## Content Negotiation
314
+
315
+ ### `shouldServeMarkdown(acceptHeader?, secFetchDest?)`
316
+
317
+ Determines if a client prefers Markdown over HTML using HTTP content negotiation.
318
+
319
+ - Returns `true` when `text/markdown` or `text/plain` has higher quality than `text/html` in the Accept header.
320
+ - If qualities are equal, earlier position wins.
321
+ - Bare wildcards (`*/*`) do not trigger Markdown (prevents breaking OG crawlers).
322
+ - `sec-fetch-dest: document` always returns `false` (browser navigation).
86
323
 
87
324
  ```typescript
88
325
  import { shouldServeMarkdown } from '@mdream/js/negotiate'
89
326
 
90
- // Returns true when Accept header prefers text/markdown over text/html
91
327
  if (shouldServeMarkdown(request.headers.accept, request.headers['sec-fetch-dest'])) {
92
- return new Response(markdown, { headers: { 'content-type': 'text/markdown' } })
328
+ return new Response(markdown, {
329
+ headers: { 'content-type': 'text/markdown' },
330
+ })
331
+ }
332
+ ```
333
+
334
+ **Parameters:**
335
+
336
+ | Parameter | Type | Description |
337
+ |---|---|---|
338
+ | `acceptHeader` | `string \| undefined` | The HTTP `Accept` header value |
339
+ | `secFetchDest` | `string \| undefined` | The `Sec-Fetch-Dest` header value |
340
+
341
+ **Returns:** `boolean`
342
+
343
+ ### `parseAcceptHeader(accept)`
344
+
345
+ Parses an HTTP Accept header into an ordered list of media types with quality values.
346
+
347
+ ```typescript
348
+ import { parseAcceptHeader } from '@mdream/js/negotiate'
349
+
350
+ const entries = parseAcceptHeader('text/markdown;q=0.9, text/html')
351
+ // [{ type: 'text/markdown', q: 0.9, position: 0 }, { type: 'text/html', q: 1, position: 1 }]
352
+ ```
353
+
354
+ **Returns:** `Array<{ type: string, q: number, position: number }>`
355
+
356
+ ---
357
+
358
+ ## Markdown Splitter
359
+
360
+ ### `htmlToMarkdownSplitChunks(html, options?)`
361
+
362
+ Converts HTML to Markdown and returns an array of chunks. Compatible with LangChain's `Document` structure.
363
+
364
+ ```typescript
365
+ import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
366
+
367
+ const chunks = htmlToMarkdownSplitChunks(html, {
368
+ chunkSize: 1000,
369
+ chunkOverlap: 200,
370
+ origin: 'https://example.com',
371
+ })
372
+
373
+ for (const chunk of chunks) {
374
+ console.log(chunk.content)
375
+ console.log(chunk.metadata.headers) // e.g., { h2: 'Section Title' }
376
+ console.log(chunk.metadata.code) // e.g., 'typescript'
377
+ console.log(chunk.metadata.loc) // e.g., { lines: { from: 1, to: 10 } }
378
+ }
379
+ ```
380
+
381
+ **Returns:** `MarkdownChunk[]`
382
+
383
+ ### `htmlToMarkdownSplitChunksStream(html, options?)`
384
+
385
+ Generator version that yields chunks during processing for better memory efficiency.
386
+
387
+ ```typescript
388
+ import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'
389
+
390
+ for (const chunk of htmlToMarkdownSplitChunksStream(html, options)) {
391
+ process.stdout.write(chunk.content)
93
392
  }
94
393
  ```
95
394
 
96
- ### llms.txt Generation
395
+ **Returns:** `Generator<MarkdownChunk, void, undefined>`
396
+
397
+ ### `SplitterOptions`
398
+
399
+ Extends `EngineOptions` with chunking-specific settings.
400
+
401
+ | Option | Type | Default | Description |
402
+ |---|---|---|---|
403
+ | `headersToSplitOn` | `number[]` | `[TAG_H2, TAG_H3, TAG_H4, TAG_H5, TAG_H6]` | TAG_* header constants to split on |
404
+ | `returnEachLine` | `boolean` | `false` | Return each non-empty line as an individual chunk |
405
+ | `stripHeaders` | `boolean` | `true` | Strip header lines from chunk content |
406
+ | `chunkSize` | `number` | `1000` | Maximum chunk size (measured by `lengthFunction`) |
407
+ | `chunkOverlap` | `number` | `200` | Overlap between chunks for context preservation. Must be less than `chunkSize`. |
408
+ | `lengthFunction` | `(text: string) => number` | `(text) => text.length` | Function to measure chunk length. Replace with a token counter for LLM applications. |
409
+ | `keepSeparator` | `boolean` | `false` | Keep separators in the split chunks |
410
+
411
+ ### `MarkdownChunk`
412
+
413
+ | Field | Type | Description |
414
+ |---|---|---|
415
+ | `content` | `string` | The markdown content of the chunk |
416
+ | `metadata.headers` | `Record<string, string> \| undefined` | Header hierarchy at this chunk position (e.g., `{ h2: 'API', h3: 'Methods' }`) |
417
+ | `metadata.code` | `string \| undefined` | Code block language if chunk contains code |
418
+ | `metadata.loc` | `{ lines: { from: number, to: number } } \| undefined` | Line number range in original document |
419
+
420
+ ---
421
+
422
+ ## llms.txt Generation
423
+
424
+ ### `generateLlmsTxtArtifacts(options)`
425
+
426
+ Generates llms.txt content, optionally including llms-full.txt and individual markdown files.
97
427
 
98
428
  ```typescript
99
429
  import { generateLlmsTxtArtifacts } from '@mdream/js/llms-txt'
@@ -101,35 +431,318 @@ import { generateLlmsTxtArtifacts } from '@mdream/js/llms-txt'
101
431
  const result = await generateLlmsTxtArtifacts({
102
432
  files: processedPages,
103
433
  siteName: 'My Site',
434
+ description: 'A description of my site',
104
435
  origin: 'https://example.com',
105
436
  generateFull: true,
437
+ generateMarkdown: true,
438
+ sections: [
439
+ {
440
+ title: 'Documentation',
441
+ description: 'API reference and guides',
442
+ links: [
443
+ { title: 'Getting Started', href: '/docs/start', description: 'Quick start guide' },
444
+ ],
445
+ },
446
+ ],
447
+ notes: 'Generated by mdream',
106
448
  })
107
449
 
108
- // result.llmsTxt -- index file with links
109
- // result.llmsFullTxt -- single file with all page content
450
+ // result.llmsTxt -- index file with links to pages
451
+ // result.llmsFullTxt -- single file with all page content (if generateFull: true)
452
+ // result.markdownFiles -- array of { path, content } (if generateMarkdown: true)
453
+ // result.processedFiles -- the input files passed through
110
454
  ```
111
455
 
112
- ### Markdown Splitter
456
+ #### `LlmsTxtArtifactsOptions`
457
+
458
+ | Option | Type | Default | Description |
459
+ |---|---|---|---|
460
+ | `files` | `ProcessedFile[]` | (required) | Array of processed page files |
461
+ | `siteName` | `string` | `'Site'` | Site name for the header |
462
+ | `description` | `string` | `undefined` | Site description (rendered as blockquote) |
463
+ | `origin` | `string` | `''` | Origin URL to prepend to relative URLs |
464
+ | `generateFull` | `boolean` | `false` | Generate llms-full.txt with complete page content |
465
+ | `generateMarkdown` | `boolean` | `false` | Generate individual markdown files under `md/` directory |
466
+ | `outputDir` | `string` | `undefined` | Output directory (used to compute relative file paths) |
467
+ | `sections` | `LlmsTxtSection[]` | `undefined` | Custom sections to write before the Pages section |
468
+ | `notes` | `string \| string[]` | `undefined` | Notes to append at the end |
469
+
470
+ #### `ProcessedFile`
471
+
472
+ | Field | Type | Description |
473
+ |---|---|---|
474
+ | `title` | `string` | Page title |
475
+ | `content` | `string` | Markdown content of the page |
476
+ | `url` | `string` | URL path of the page |
477
+ | `filePath` | `string \| undefined` | Optional file path on disk |
478
+ | `metadata` | `{ title?, description?, keywords?, author? }` | Optional metadata extracted from the page |
479
+
480
+ #### `LlmsTxtSection`
481
+
482
+ | Field | Type | Description |
483
+ |---|---|---|
484
+ | `title` | `string` | Section heading |
485
+ | `description` | `string \| string[]` | Section description (can be multiple paragraphs) |
486
+ | `links` | `LlmsTxtLink[]` | Links in the section |
487
+
488
+ #### `LlmsTxtLink`
489
+
490
+ | Field | Type | Description |
491
+ |---|---|---|
492
+ | `title` | `string` | Link title |
493
+ | `href` | `string` | Link URL |
494
+ | `description` | `string \| undefined` | Optional description shown after the link |
495
+
496
+ ### `createLlmsTxtStream(options)`
497
+
498
+ Creates a `WritableStream<ProcessedFile>` that generates llms.txt artifacts incrementally. Writes files to disk as pages are streamed in, never keeping full content in memory. Pages in llms.txt are sorted by URL path hierarchy on close.
499
+
500
+ ```typescript
501
+ import { createLlmsTxtStream } from '@mdream/js/llms-txt'
502
+
503
+ const stream = createLlmsTxtStream({
504
+ outputDir: './output',
505
+ siteName: 'My Site',
506
+ origin: 'https://example.com',
507
+ generateFull: true,
508
+ })
509
+
510
+ const writer = stream.getWriter()
511
+ await writer.write({ title: 'Home', content: '# Home\nWelcome', url: '/' })
512
+ await writer.write({ title: 'About', content: '# About\nInfo', url: '/about' })
513
+ await writer.close()
514
+ // Writes llms.txt and llms-full.txt to ./output/
515
+ ```
516
+
517
+ **Returns:** `WritableStream<ProcessedFile>`
518
+
519
+ ---
520
+
521
+ ## Low-Level Parser
522
+
523
+ ### `parseHtml(html, options?)`
524
+
525
+ Parses HTML into a list of DOM events. Returns the events and any remaining unparsed HTML.
526
+
527
+ ```typescript
528
+ import { parseHtml } from '@mdream/js/parse'
529
+
530
+ const { events, remainingHtml } = parseHtml('<p>Hello</p>')
531
+ ```
532
+
533
+ **Returns:** `{ events: NodeEvent[], remainingHtml: string }`
534
+
535
+ ### `parseHtmlStream(html, state, onEvent)`
536
+
537
+ Streaming parser that calls `onEvent` for each DOM event. Returns any remaining unparsed HTML (useful for processing partial chunks).
538
+
539
+ ```typescript
540
+ import { parseHtmlStream } from '@mdream/js/parse'
541
+
542
+ const state = { depthMap: new Uint8Array(1024), depth: 0 }
543
+ const remaining = parseHtmlStream(htmlChunk, state, (event) => {
544
+ // Process each event
545
+ })
546
+ ```
547
+
548
+ **Returns:** `string` (remaining unparsed HTML)
549
+
550
+ ---
551
+
552
+ ## CLI
553
+
554
+ Reads HTML from stdin and outputs Markdown to stdout.
555
+
556
+ ```bash
557
+ # Basic conversion
558
+ curl -s https://example.com | npx @mdream/js
559
+
560
+ # With origin URL for resolving relative paths
561
+ curl -s https://example.com | npx @mdream/js --origin https://example.com
562
+
563
+ # With minimal preset
564
+ curl -s https://example.com | npx @mdream/js --origin https://example.com --preset minimal
565
+ ```
566
+
567
+ ### CLI Options
568
+
569
+ | Flag | Description |
570
+ |---|---|
571
+ | `--origin <url>` | Origin URL for resolving relative image paths and links |
572
+ | `--preset <preset>` | Conversion preset. Currently supports: `minimal` |
573
+ | `-v, --version` | Show version number |
574
+ | `-h, --help` | Show help |
575
+
576
+ ---
577
+
578
+ ## Usage Examples
579
+
580
+ ### Custom Tag Overrides
581
+
582
+ Map custom HTML elements to standard Markdown behavior:
583
+
584
+ ```typescript
585
+ import { htmlToMarkdown } from '@mdream/js'
586
+
587
+ const md = htmlToMarkdown('<x-heading>Title</x-heading>', {
588
+ plugins: {
589
+ tagOverrides: {
590
+ // String alias: make <x-heading> behave like <h2>
591
+ 'x-heading': 'h2',
592
+
593
+ // Object override: custom enter/exit strings
594
+ 'callout': {
595
+ enter: '> **Note:** ',
596
+ exit: '\n',
597
+ spacing: [2, 2],
598
+ },
599
+ },
600
+ },
601
+ })
602
+ ```
603
+
604
+ ### Declarative Extraction
605
+
606
+ Extract data from elements during conversion:
607
+
608
+ ```typescript
609
+ import { htmlToMarkdown } from '@mdream/js'
610
+
611
+ const images: { src: string, alt: string }[] = []
612
+
613
+ const md = htmlToMarkdown(html, {
614
+ plugins: {
615
+ extraction: {
616
+ 'img[alt]': (element) => {
617
+ images.push({
618
+ src: element.attributes.src,
619
+ alt: element.attributes.alt,
620
+ })
621
+ },
622
+ },
623
+ },
624
+ })
625
+
626
+ console.log('Found images:', images)
627
+ ```
628
+
629
+ ### Frontmatter with Callback
630
+
631
+ Receive structured frontmatter data for further processing:
632
+
633
+ ```typescript
634
+ import { htmlToMarkdown } from '@mdream/js'
635
+
636
+ let metadata: Record<string, string> = {}
637
+
638
+ const md = htmlToMarkdown(html, {
639
+ plugins: {
640
+ frontmatter: {
641
+ additionalFields: { source: 'crawler' },
642
+ metaFields: ['robots'],
643
+ onExtract: (fm) => {
644
+ metadata = fm
645
+ },
646
+ },
647
+ },
648
+ })
649
+
650
+ console.log('Title:', metadata.title)
651
+ console.log('Description:', metadata.description)
652
+ ```
653
+
654
+ ### Custom Hooks for Content Filtering
655
+
656
+ ```typescript
657
+ import { htmlToMarkdown } from '@mdream/js'
658
+ import { createPlugin } from '@mdream/js/plugins'
659
+
660
+ const md = htmlToMarkdown(html, {
661
+ hooks: [
662
+ createPlugin({
663
+ beforeNodeProcess(event) {
664
+ const { node } = event
665
+ if (node.type === 1 /* ELEMENT_NODE */) {
666
+ const el = node as any
667
+ // Skip ads and promotional content
668
+ if (el.attributes?.class?.includes('ad')
669
+ || el.attributes?.id?.includes('promo')) {
670
+ return { skip: true }
671
+ }
672
+ }
673
+ },
674
+ processTextNode(textNode) {
675
+ // Uppercase all TODO markers
676
+ if (textNode.value.includes('TODO')) {
677
+ return { content: textNode.value.toUpperCase(), skip: false }
678
+ }
679
+ },
680
+ }),
681
+ ],
682
+ })
683
+ ```
684
+
685
+ ### Token-Based Chunking for LLMs
113
686
 
114
687
  ```typescript
115
688
  import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
689
+ import { encoding_for_model } from 'tiktoken'
690
+
691
+ const enc = encoding_for_model('gpt-4')
116
692
 
117
693
  const chunks = htmlToMarkdownSplitChunks(html, {
118
- chunkSize: 1000,
694
+ chunkSize: 512,
695
+ chunkOverlap: 64,
696
+ lengthFunction: text => enc.encode(text).length,
119
697
  origin: 'https://example.com',
120
698
  })
121
699
  ```
122
700
 
123
- ## License
701
+ ---
702
+
703
+ ## Exported Types
704
+
705
+ ```typescript
706
+ import type {
707
+ BuiltinPlugins,
708
+ CleanOptions,
709
+ ElementNode,
710
+ EngineOptions,
711
+ ExtractedElement,
712
+ FrontmatterConfig,
713
+ MarkdownChunk,
714
+ MdreamOptions,
715
+ Node,
716
+ NodeEvent,
717
+ PluginContext,
718
+ SplitterOptions,
719
+ TagOverride,
720
+ TextNode,
721
+ TransformPlugin,
722
+ } from '@mdream/js'
723
+ ```
724
+
725
+ ## Exported Constants
726
+
727
+ ```typescript
728
+ import {
729
+ ELEMENT_NODE, // Node type for HTML elements (1)
730
+ NodeEventEnter, // Event type for entering a node (0)
731
+ NodeEventExit, // Event type for exiting a node (1)
732
+ TAG_H1, // Tag ID constant for <h1>
733
+ TAG_H2, // Tag ID constant for <h2>
734
+ TAG_H3, // Tag ID constant for <h3>
735
+ TAG_H4, // Tag ID constant for <h4>
736
+ TAG_H5, // Tag ID constant for <h5>
737
+ TAG_H6, // Tag ID constant for <h6>
738
+ TEXT_NODE, // Node type for text content (3)
739
+ } from '@mdream/js'
740
+ ```
124
741
 
125
- [MIT License](./LICENSE)
126
742
 
127
- <!-- Badges -->
128
743
  [npm-version-src]: https://img.shields.io/npm/v/@mdream/js/latest.svg?style=flat&colorA=18181B&colorB=4C9BE0
129
744
  [npm-version-href]: https://npmjs.com/package/@mdream/js
130
-
131
745
  [npm-downloads-src]: https://img.shields.io/npm/dm/@mdream/js.svg?style=flat&colorA=18181B&colorB=4C9BE0
132
746
  [npm-downloads-href]: https://npm.chart.dev/@mdream/js
133
-
134
747
  [license-src]: https://img.shields.io/npm/l/@mdream/js.svg?style=flat&colorA=18181B&colorB=4C9BE0
135
748
  [license-href]: https://npmjs.com/package/@mdream/js
@@ -4,15 +4,6 @@ const TAG_H3 = 9;
4
4
  const TAG_H4 = 10;
5
5
  const TAG_H5 = 11;
6
6
  const TAG_H6 = 12;
7
- const HTML_ENTITIES = {
8
- "&amp;": "&",
9
- "&lt;": "<",
10
- "&gt;": ">",
11
- "&quot;": "\"",
12
- "&#39;": "'",
13
- "&apos;": "'",
14
- "&nbsp;": " "
15
- };
16
7
  const ELEMENT_NODE = 1;
17
8
  const TEXT_NODE = 2;
18
9
  const NodeEventEnter = 0;
@@ -134,4 +125,4 @@ const BLOCKQUOTE_SPACING = [1, 1];
134
125
  const LIST_ITEM_SPACING = [1, 0];
135
126
  const TABLE_ROW_SPACING = [0, 1];
136
127
  //#endregion
137
- export { TagIdMap as _, LIST_ITEM_SPACING as a, NodeEventExit as c, TAG_H2 as d, TAG_H3 as f, TEXT_NODE as g, TAG_H6 as h, HTML_ENTITIES as i, TABLE_ROW_SPACING as l, TAG_H5 as m, DEFAULT_BLOCK_SPACING as n, NO_SPACING as o, TAG_H4 as p, ELEMENT_NODE as r, NodeEventEnter as s, BLOCKQUOTE_SPACING as t, TAG_H1 as u };
128
+ export { NO_SPACING as a, TABLE_ROW_SPACING as c, TAG_H3 as d, TAG_H4 as f, TagIdMap as g, TEXT_NODE as h, LIST_ITEM_SPACING as i, TAG_H1 as l, TAG_H6 as m, DEFAULT_BLOCK_SPACING as n, NodeEventEnter as o, TAG_H5 as p, ELEMENT_NODE as r, NodeEventExit as s, BLOCKQUOTE_SPACING as t, TAG_H2 as u };
@@ -1,4 +1,4 @@
1
- import { _ as TagIdMap, a as LIST_ITEM_SPACING, i as HTML_ENTITIES, l as TABLE_ROW_SPACING, o as NO_SPACING, t as BLOCKQUOTE_SPACING } from "./const.mjs";
1
+ import { a as NO_SPACING, c as TABLE_ROW_SPACING, g as TagIdMap, i as LIST_ITEM_SPACING, t as BLOCKQUOTE_SPACING } from "./const.mjs";
2
2
  //#region src/tags.ts
3
3
  function resolveUrl(url, origin) {
4
4
  if (!url) return url;
@@ -598,43 +598,292 @@ function buildTagOverrideHandlers(overrides) {
598
598
  return result;
599
599
  }
600
600
  //#endregion
601
+ //#region src/entities.ts
602
+ const HTML_ENTITIES = {
603
+ "&aacute;": "á",
604
+ "&Aacute;": "Á",
605
+ "&acirc;": "â",
606
+ "&Acirc;": "Â",
607
+ "&acute;": "´",
608
+ "&aelig;": "æ",
609
+ "&AElig;": "Æ",
610
+ "&agrave;": "à",
611
+ "&Agrave;": "À",
612
+ "&alefsym;": "ℵ",
613
+ "&alpha;": "α",
614
+ "&Alpha;": "Α",
615
+ "&amp;": "&",
616
+ "&and;": "∧",
617
+ "&ang;": "∠",
618
+ "&apos;": "'",
619
+ "&aring;": "å",
620
+ "&Aring;": "Å",
621
+ "&asymp;": "≈",
622
+ "&atilde;": "ã",
623
+ "&Atilde;": "Ã",
624
+ "&auml;": "ä",
625
+ "&Auml;": "Ä",
626
+ "&bdquo;": "„",
627
+ "&beta;": "β",
628
+ "&Beta;": "Β",
629
+ "&brvbar;": "¦",
630
+ "&bull;": "•",
631
+ "&cap;": "∩",
632
+ "&ccedil;": "ç",
633
+ "&Ccedil;": "Ç",
634
+ "&cedil;": "¸",
635
+ "&cent;": "¢",
636
+ "&chi;": "χ",
637
+ "&Chi;": "Χ",
638
+ "&circ;": "ˆ",
639
+ "&clubs;": "♣",
640
+ "&cong;": "≅",
641
+ "&copy;": "©",
642
+ "&crarr;": "↵",
643
+ "&cup;": "∪",
644
+ "&curren;": "¤",
645
+ "&dagger;": "†",
646
+ "&Dagger;": "‡",
647
+ "&darr;": "↓",
648
+ "&dArr;": "⇓",
649
+ "&deg;": "°",
650
+ "&delta;": "δ",
651
+ "&Delta;": "Δ",
652
+ "&diams;": "♦",
653
+ "&divide;": "÷",
654
+ "&eacute;": "é",
655
+ "&Eacute;": "É",
656
+ "&ecirc;": "ê",
657
+ "&Ecirc;": "Ê",
658
+ "&egrave;": "è",
659
+ "&Egrave;": "È",
660
+ "&empty;": "∅",
661
+ "&epsilon;": "ε",
662
+ "&Epsilon;": "Ε",
663
+ "&equiv;": "≡",
664
+ "&eta;": "η",
665
+ "&Eta;": "Η",
666
+ "&eth;": "ð",
667
+ "&ETH;": "Ð",
668
+ "&euml;": "ë",
669
+ "&Euml;": "Ë",
670
+ "&euro;": "€",
671
+ "&exist;": "∃",
672
+ "&fnof;": "ƒ",
673
+ "&forall;": "∀",
674
+ "&frac12;": "½",
675
+ "&frac14;": "¼",
676
+ "&frac34;": "¾",
677
+ "&frasl;": "⁄",
678
+ "&gamma;": "γ",
679
+ "&Gamma;": "Γ",
680
+ "&ge;": "≥",
681
+ "&gt;": ">",
682
+ "&harr;": "↔",
683
+ "&hArr;": "⇔",
684
+ "&hearts;": "♥",
685
+ "&hellip;": "…",
686
+ "&iacute;": "í",
687
+ "&Iacute;": "Í",
688
+ "&icirc;": "î",
689
+ "&Icirc;": "Î",
690
+ "&iexcl;": "¡",
691
+ "&igrave;": "ì",
692
+ "&Igrave;": "Ì",
693
+ "&image;": "ℑ",
694
+ "&infin;": "∞",
695
+ "&int;": "∫",
696
+ "&iota;": "ι",
697
+ "&Iota;": "Ι",
698
+ "&iquest;": "¿",
699
+ "&isin;": "∈",
700
+ "&iuml;": "ï",
701
+ "&Iuml;": "Ï",
702
+ "&kappa;": "κ",
703
+ "&Kappa;": "Κ",
704
+ "&lambda;": "λ",
705
+ "&Lambda;": "Λ",
706
+ "&lang;": "⟨",
707
+ "&laquo;": "«",
708
+ "&larr;": "←",
709
+ "&lArr;": "⇐",
710
+ "&lceil;": "⌈",
711
+ "&ldquo;": "“",
712
+ "&le;": "≤",
713
+ "&lfloor;": "⌊",
714
+ "&lowast;": "∗",
715
+ "&loz;": "◊",
716
+ "&lsaquo;": "‹",
717
+ "&lsquo;": "‘",
718
+ "&lt;": "<",
719
+ "&macr;": "¯",
720
+ "&mdash;": "—",
721
+ "&micro;": "µ",
722
+ "&middot;": "·",
723
+ "&minus;": "−",
724
+ "&mu;": "μ",
725
+ "&Mu;": "Μ",
726
+ "&nabla;": "∇",
727
+ "&nbsp;": " ",
728
+ "&ndash;": "–",
729
+ "&ne;": "≠",
730
+ "&ni;": "∋",
731
+ "&not;": "¬",
732
+ "&notin;": "∉",
733
+ "&nsub;": "⊄",
734
+ "&ntilde;": "ñ",
735
+ "&Ntilde;": "Ñ",
736
+ "&nu;": "ν",
737
+ "&Nu;": "Ν",
738
+ "&oacute;": "ó",
739
+ "&Oacute;": "Ó",
740
+ "&ocirc;": "ô",
741
+ "&Ocirc;": "Ô",
742
+ "&oelig;": "œ",
743
+ "&OElig;": "Œ",
744
+ "&ograve;": "ò",
745
+ "&Ograve;": "Ò",
746
+ "&oline;": "‾",
747
+ "&omega;": "ω",
748
+ "&Omega;": "Ω",
749
+ "&omicron;": "ο",
750
+ "&Omicron;": "Ο",
751
+ "&oplus;": "⊕",
752
+ "&or;": "∨",
753
+ "&ordf;": "ª",
754
+ "&ordm;": "º",
755
+ "&oslash;": "ø",
756
+ "&Oslash;": "Ø",
757
+ "&otilde;": "õ",
758
+ "&Otilde;": "Õ",
759
+ "&otimes;": "⊗",
760
+ "&ouml;": "ö",
761
+ "&Ouml;": "Ö",
762
+ "&para;": "¶",
763
+ "&part;": "∂",
764
+ "&permil;": "‰",
765
+ "&perp;": "⊥",
766
+ "&phi;": "φ",
767
+ "&Phi;": "Φ",
768
+ "&pi;": "π",
769
+ "&Pi;": "Π",
770
+ "&piv;": "ϖ",
771
+ "&plusmn;": "±",
772
+ "&pound;": "£",
773
+ "&prime;": "′",
774
+ "&Prime;": "″",
775
+ "&prod;": "∏",
776
+ "&prop;": "∝",
777
+ "&psi;": "ψ",
778
+ "&Psi;": "Ψ",
779
+ "&quot;": "\"",
780
+ "&radic;": "√",
781
+ "&rang;": "⟩",
782
+ "&raquo;": "»",
783
+ "&rarr;": "→",
784
+ "&rArr;": "⇒",
785
+ "&rceil;": "⌉",
786
+ "&rdquo;": "”",
787
+ "&real;": "ℜ",
788
+ "&reg;": "®",
789
+ "&rfloor;": "⌋",
790
+ "&rho;": "ρ",
791
+ "&Rho;": "Ρ",
792
+ "&rsaquo;": "›",
793
+ "&rsquo;": "’",
794
+ "&sbquo;": "‚",
795
+ "&scaron;": "š",
796
+ "&Scaron;": "Š",
797
+ "&sdot;": "⋅",
798
+ "&sect;": "§",
799
+ "&sigma;": "σ",
800
+ "&Sigma;": "Σ",
801
+ "&sigmaf;": "ς",
802
+ "&sim;": "∼",
803
+ "&spades;": "♠",
804
+ "&sub;": "⊂",
805
+ "&sube;": "⊆",
806
+ "&sum;": "∑",
807
+ "&sup;": "⊃",
808
+ "&sup1;": "¹",
809
+ "&sup2;": "²",
810
+ "&sup3;": "³",
811
+ "&supe;": "⊇",
812
+ "&szlig;": "ß",
813
+ "&tau;": "τ",
814
+ "&Tau;": "Τ",
815
+ "&there4;": "∴",
816
+ "&theta;": "θ",
817
+ "&Theta;": "Θ",
818
+ "&thetasym;": "ϑ",
819
+ "&thorn;": "þ",
820
+ "&THORN;": "Þ",
821
+ "&tilde;": "˜",
822
+ "&times;": "×",
823
+ "&trade;": "™",
824
+ "&uacute;": "ú",
825
+ "&Uacute;": "Ú",
826
+ "&uarr;": "↑",
827
+ "&uArr;": "⇑",
828
+ "&ucirc;": "û",
829
+ "&Ucirc;": "Û",
830
+ "&ugrave;": "ù",
831
+ "&Ugrave;": "Ù",
832
+ "&uml;": "¨",
833
+ "&upsih;": "ϒ",
834
+ "&upsilon;": "υ",
835
+ "&Upsilon;": "Υ",
836
+ "&uuml;": "ü",
837
+ "&Uuml;": "Ü",
838
+ "&weierp;": "℘",
839
+ "&xi;": "ξ",
840
+ "&Xi;": "Ξ",
841
+ "&yacute;": "ý",
842
+ "&Yacute;": "Ý",
843
+ "&yen;": "¥",
844
+ "&yuml;": "ÿ",
845
+ "&Yuml;": "Ÿ",
846
+ "&zeta;": "ζ",
847
+ "&Zeta;": "Ζ"
848
+ };
849
+ //#endregion
601
850
  //#region src/utils.ts
602
851
  /**
603
- * Decode HTML entities - optimized version with single pass
852
+ * Decode HTML entities - single pass with O(1) named entity lookup
604
853
  */
605
854
  function decodeHTMLEntities(text) {
606
855
  let result = "";
607
856
  let i = 0;
608
- while (i < text.length) {
609
- if (text[i] === "&") {
610
- let match = false;
611
- for (const [entity, replacement] of Object.entries(HTML_ENTITIES)) if (text.startsWith(entity, i)) {
612
- result += replacement;
613
- i += entity.length;
614
- match = true;
615
- break;
857
+ const len = text.length;
858
+ while (i < len) {
859
+ if (text.charCodeAt(i) === 38) if (i + 2 < len && text.charCodeAt(i + 1) === 35) {
860
+ const start = i;
861
+ i += 2;
862
+ const isHex = text.charCodeAt(i) === 120 || text.charCodeAt(i) === 88;
863
+ if (isHex) i++;
864
+ const numStart = i;
865
+ const maxDigits = i + (isHex ? 7 : 8);
866
+ while (i < len && i < maxDigits && text.charCodeAt(i) !== 59) i++;
867
+ if (i < len && text.charCodeAt(i) === 59) {
868
+ const codePoint = Number.parseInt(text.substring(numStart, i), isHex ? 16 : 10);
869
+ if (codePoint >= 0 && codePoint <= 1114111 && !Number.isNaN(codePoint)) {
870
+ result += String.fromCodePoint(codePoint);
871
+ i++;
872
+ continue;
873
+ }
616
874
  }
617
- if (match) continue;
618
- if (i + 2 < text.length && text[i + 1] === "#") {
619
- const start = i;
620
- i += 2;
621
- const isHex = text[i] === "x" || text[i] === "X";
622
- if (isHex) i++;
623
- const numStart = i;
624
- while (i < text.length && text[i] !== ";") i++;
625
- if (i < text.length && text[i] === ";") {
626
- const numStr = text.substring(numStart, i);
627
- const base = isHex ? 16 : 10;
628
- try {
629
- const codePoint = Number.parseInt(numStr, base);
630
- if (!Number.isNaN(codePoint)) {
631
- result += String.fromCodePoint(codePoint);
632
- i++;
633
- continue;
634
- }
635
- } catch {}
875
+ i = start;
876
+ } else {
877
+ const maxEnd = i + 33 < len ? i + 33 : len;
878
+ let j = i + 1;
879
+ while (j < maxEnd && text.charCodeAt(j) !== 59) j++;
880
+ if (j < maxEnd && text.charCodeAt(j) === 59) {
881
+ const replacement = HTML_ENTITIES[text.substring(i, j + 1)];
882
+ if (replacement !== void 0) {
883
+ result += replacement;
884
+ i = j + 1;
885
+ continue;
636
886
  }
637
- i = start;
638
887
  }
639
888
  }
640
889
  result += text[i];
@@ -1,4 +1,4 @@
1
- import { _ as TagIdMap } from "./const.mjs";
1
+ import { g as TagIdMap } from "./const.mjs";
2
2
  //#region src/pluggable/plugin.ts
3
3
  /**
4
4
  * Create a plugin with type-safe hook definitions.
@@ -1,4 +1,4 @@
1
- import { n as DEFAULT_BLOCK_SPACING, o as NO_SPACING } from "./const.mjs";
1
+ import { a as NO_SPACING, n as DEFAULT_BLOCK_SPACING } from "./const.mjs";
2
2
  import { r as parseHtmlStream } from "./parse.mjs";
3
3
  import { a as extractionCollectorPlugin, i as filterPlugin, n as isolateMainPlugin, r as frontmatterPlugin, t as tailwindPlugin } from "./plugins.mjs";
4
4
  //#region src/plugin-processor.ts
@@ -240,6 +240,10 @@ function createMarkdownProcessor(options = {}, resolvedPlugins = [], tagOverride
240
240
  const currentContent = state.buffer.join("").trimStart();
241
241
  const newContent = currentContent.slice(lastYieldedLength);
242
242
  lastYieldedLength = currentContent.length;
243
+ if (state.buffer.length > 1) {
244
+ state.buffer.length = 0;
245
+ state.buffer.push(currentContent);
246
+ }
243
247
  return newContent;
244
248
  }
245
249
  return {
package/dist/index.mjs CHANGED
@@ -1,5 +1,5 @@
1
1
  import { n as streamHtmlToMarkdown, t as htmlToMarkdown } from "./_chunks/src.mjs";
2
- import { c as NodeEventExit, d as TAG_H2, f as TAG_H3, g as TEXT_NODE, h as TAG_H6, m as TAG_H5, p as TAG_H4, r as ELEMENT_NODE, s as NodeEventEnter, u as TAG_H1 } from "./_chunks/const.mjs";
2
+ import { d as TAG_H3, f as TAG_H4, h as TEXT_NODE, l as TAG_H1, m as TAG_H6, o as NodeEventEnter, p as TAG_H5, r as ELEMENT_NODE, s as NodeEventExit, u as TAG_H2 } from "./_chunks/const.mjs";
3
3
  import "./_chunks/parse.mjs";
4
4
  import "./_chunks/resolve-plugins.mjs";
5
5
  import { s as createPlugin } from "./_chunks/plugins.mjs";
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@mdream/js",
3
3
  "type": "module",
4
- "version": "1.0.0-beta.10",
4
+ "version": "1.0.0-beta.12",
5
5
  "description": "JavaScript HTML-to-Markdown engine for mdream. Escape hatch for hooks and edge runtimes.",
6
6
  "author": {
7
7
  "name": "Harlan Wilton",