flux-md 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +72 -0
- package/LICENSE +21 -0
- package/README.md +398 -0
- package/package.json +50 -0
- package/src/client.ts +315 -0
- package/src/hi.ts +244 -0
- package/src/html-to-react.ts +282 -0
- package/src/index.ts +33 -0
- package/src/react.tsx +223 -0
- package/src/renderers/CodeBlock.tsx +62 -0
- package/src/renderers/Math.tsx +26 -0
- package/src/renderers/Mermaid.tsx +26 -0
- package/src/types.ts +130 -0
- package/src/wasm/flux_md_core.d.ts +94 -0
- package/src/wasm/flux_md_core.js +399 -0
- package/src/wasm/flux_md_core_bg.wasm +0 -0
- package/src/wasm/flux_md_core_bg.wasm.d.ts +18 -0
- package/src/wasm/package.json +17 -0
- package/src/worker.ts +151 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
Notable changes to flux-md. Format based on
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/); this project aims to follow
|
|
5
|
+
[Semantic Versioning](https://semver.org/).
|
|
6
|
+
|
|
7
|
+
## 0.3.1 — 2026-05-27
|
|
8
|
+
|
|
9
|
+
### Performance
|
|
10
|
+
|
|
11
|
+
- Streaming a long unbroken paragraph is now O(n) instead of O(n²) — including
|
|
12
|
+
paragraphs **dense with inline constructs** (emphasis, code spans, links,
|
|
13
|
+
inline math), not just plain text. The open paragraph commits its settled
|
|
14
|
+
prefix and re-renders only the short active tail. Because inline output isn't
|
|
15
|
+
prefix-stable (a late `*` re-emphasizes earlier text, a late backtick opens a
|
|
16
|
+
code span), the stable boundary is computed inside the inline renderer itself:
|
|
17
|
+
it tracks unmatched openers, unpaired forward-pairable emphasis, and resolved
|
|
18
|
+
emphasis spans, and commits only up to the largest provably-final cut. Output
|
|
19
|
+
is byte-identical. Measured on 200 KB single paragraphs at 16-byte chunks:
|
|
20
|
+
plain **34,167 ms → ~130 ms** (~260×); emphasis-rich **60,569 ms → ~157 ms**
|
|
21
|
+
(~386×).
|
|
22
|
+
- The open-code-fence fast path no longer clones the accumulated escaped body on
|
|
23
|
+
every append; it assembles the block HTML directly from the cached pieces,
|
|
24
|
+
dropping one full O(body) copy per append. A 200 KB fence streams in **~82 ms**
|
|
25
|
+
at 16-byte chunks (was ~154 ms, ~1.9×). Output is byte-identical.
|
|
26
|
+
|
|
27
|
+
## 0.3.0
|
|
28
|
+
|
|
29
|
+
### Added
|
|
30
|
+
|
|
31
|
+
- **`gfmMath`** — opt-in math. Inline `$…$` and `\(…\)`; display `$$…$$` and
|
|
32
|
+
`\[…\]`. Inline `$` uses the pandoc rule, so currency like `$5 and $10` stays
|
|
33
|
+
literal. Emits KaTeX-ready markup (`<span class="math math-inline">` /
|
|
34
|
+
`<div class="math math-display">`) carrying the LaTeX as text content — bring
|
|
35
|
+
your own KaTeX (flux-md stays zero-dep) or override `components.MathBlock`
|
|
36
|
+
(which receives the LaTeX as `text`). Display fences are blank-line tolerant
|
|
37
|
+
and stream incrementally. Addresses [Streamdown #522]. Off by default.
|
|
38
|
+
- **`dirAuto`** — opt-in per-block `dir="auto"` on block-level text elements
|
|
39
|
+
(`p`, `h1`–`h6`, `blockquote`, `ul`/`ol`/`li`, `table`, alerts, footnotes), so
|
|
40
|
+
the browser detects each block's direction (RTL/LTR) independently in
|
|
41
|
+
mixed-language documents. Code blocks stay LTR. Addresses [Streamdown #509].
|
|
42
|
+
Off by default.
|
|
43
|
+
|
|
44
|
+
### Performance
|
|
45
|
+
|
|
46
|
+
- Streaming a long fenced code block is now **O(n) instead of O(n²)**: an open
|
|
47
|
+
code fence caches its escaped body and extends it by only the newly arrived
|
|
48
|
+
lines. Measured on a 200 KB fence — **14,278 ms → 230 ms** at 16-byte chunks,
|
|
49
|
+
**898 ms → 22 ms** at 256-byte chunks. Output is byte-identical.
|
|
50
|
+
- Dropped a redundant per-append clone of the link-reference table.
|
|
51
|
+
|
|
52
|
+
### Known limitations
|
|
53
|
+
|
|
54
|
+
- Streaming a very long **unbroken** paragraph (no blank lines) is still O(n²):
|
|
55
|
+
inline rendering re-runs over the whole paragraph each chunk, and unlike code
|
|
56
|
+
it can't be prefix-cached (a late `*` can emphasize earlier text). Tracked for
|
|
57
|
+
a future release; breaking the text into paragraphs avoids it.
|
|
58
|
+
|
|
59
|
+
### Internal
|
|
60
|
+
|
|
61
|
+
- Added a Rust streaming-throughput benchmark (`cargo run --release --example
|
|
62
|
+
bench`) plus char-by-char streaming-parity tests for the code-fence cache,
|
|
63
|
+
math, and bidi paths.
|
|
64
|
+
|
|
65
|
+
## 0.2.0
|
|
66
|
+
|
|
67
|
+
- Initial public release: zero-dep streaming markdown, Rust→WASM core, one Web
|
|
68
|
+
Worker per stream, CommonMark 0.31 (652/652) + GFM (tables, strikethrough,
|
|
69
|
+
task lists, extended autolinks, GitHub alerts, footnotes).
|
|
70
|
+
|
|
71
|
+
[Streamdown #522]: https://github.com/vercel/streamdown/issues/522
|
|
72
|
+
[Streamdown #509]: https://github.com/vercel/streamdown/issues/509
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 siinghd
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,398 @@
|
|
|
1
|
+
# flux-md
|
|
2
|
+
|
|
3
|
+
Zero-dep streaming markdown for the browser. Rust→WASM core, one Web Worker per stream, incremental parse with speculative closure for mid-stream constructs.
|
|
4
|
+
|
|
5
|
+
Built because [Streamdown](https://streamdown.ai) crashes the main thread when you run 5 concurrent LLM calls. Same input, this library uses **~8× less peak heap, ~6× less retained memory, and ~2× less main-thread blocking** — measured in-browser with `performance.memory`. See [the live demo](https://md.hsingh.app/) for an A/B comparison.
|
|
6
|
+
|
|
7
|
+
## Install
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
bun add flux-md # or: npm i flux-md / pnpm add flux-md
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
flux-md ships as **source** (TypeScript + the compiled WASM). The worker and
|
|
14
|
+
WASM asset are referenced with the **web-standard `new URL(asset,
|
|
15
|
+
import.meta.url)`** pattern, so any bundler with asset-module support resolves
|
|
16
|
+
them: **Vite** (the reference setup), **webpack 5**, **Rollup** (with asset
|
|
17
|
+
modules), and **Parcel**. Next.js (webpack/turbopack) should work but is
|
|
18
|
+
untested — file an issue if it doesn't. It is **browser-only** (it constructs
|
|
19
|
+
Web Workers); it does not run under SSR/RSC. `react` is an optional peer
|
|
20
|
+
dependency — only needed if you import `flux-md/react`.
|
|
21
|
+
|
|
22
|
+
## Quick start
|
|
23
|
+
|
|
24
|
+
```ts
|
|
25
|
+
import { FluxClient, FluxMarkdown } from "flux-md";
|
|
26
|
+
|
|
27
|
+
// One client per stream. Spawns a Web Worker that owns a Rust parser.
|
|
28
|
+
const client = new FluxClient();
|
|
29
|
+
|
|
30
|
+
// Feed chunks as they arrive from your SSE / fetch reader.
|
|
31
|
+
for await (const delta of streamFromAi()) {
|
|
32
|
+
client.append(delta);
|
|
33
|
+
}
|
|
34
|
+
client.finalize();
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
In React:
|
|
38
|
+
|
|
39
|
+
```tsx
|
|
40
|
+
import { useMemo } from "react";
|
|
41
|
+
import { FluxClient, FluxMarkdown } from "flux-md";
|
|
42
|
+
|
|
43
|
+
export function ChatMessage({ stream }: { stream: AsyncIterable<string> }) {
|
|
44
|
+
const client = useMemo(() => new FluxClient(), []);
|
|
45
|
+
useEffect(() => {
|
|
46
|
+
(async () => {
|
|
47
|
+
for await (const chunk of stream) client.append(chunk);
|
|
48
|
+
client.finalize();
|
|
49
|
+
})();
|
|
50
|
+
return () => client.destroy();
|
|
51
|
+
}, [stream]);
|
|
52
|
+
return <FluxMarkdown client={client} />;
|
|
53
|
+
}
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Multiple concurrent streams just need multiple clients — each runs in its own worker, so they don't share main-thread budget.
|
|
57
|
+
|
|
58
|
+
## What it does
|
|
59
|
+
|
|
60
|
+
| Concern | flux-md | typical react-markdown / Streamdown |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| Re-parse on each token | No — only the active tail | Yes, full string |
|
|
63
|
+
| Where parse runs | Web Worker (off main thread) | Main thread |
|
|
64
|
+
| Block identity across chunks | Stable monotonic IDs | New keys on every render |
|
|
65
|
+
| Mid-stream unclosed `` ``` `` / `*` / `**` | Speculatively closed in render, replaced cleanly | Often renders raw or breaks |
|
|
66
|
+
| Heavy renderers (syntax, math, mermaid) | Deferred until block close | Re-run per chunk |
|
|
67
|
+
| XSS sanitization | Allowlist in Rust + URL scheme check | rehype-sanitize on JS thread |
|
|
68
|
+
|
|
69
|
+
## Public API
|
|
70
|
+
|
|
71
|
+
### `FluxClient`
|
|
72
|
+
|
|
73
|
+
```ts
|
|
74
|
+
class FluxClient {
|
|
75
|
+
constructor(options?: { pool?: FluxPool; config?: ParserConfig });
|
|
76
|
+
append(chunk: string): void; // queue text for parsing
|
|
77
|
+
finalize(): void; // mark stream complete
|
|
78
|
+
reset(): void; // wipe and reuse
|
|
79
|
+
destroy(): void; // free this stream's parser
|
|
80
|
+
whenReady(): Promise<void>; // resolves once WASM loaded
|
|
81
|
+
subscribe(listener: () => void): () => void; // React-friendly store
|
|
82
|
+
getSnapshot(): Block[]; // ordered current blocks
|
|
83
|
+
getMetrics(): { bytes, patches, totalParseMs, throughputKBs,
|
|
84
|
+
retainedBytes, wasmMemoryBytes, ... };
|
|
85
|
+
}
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
#### Per-stream config
|
|
89
|
+
|
|
90
|
+
```ts
|
|
91
|
+
const client = new FluxClient({
|
|
92
|
+
config: {
|
|
93
|
+
gfmAutolinks: true, // bare www./http(s):// URLs + emails → links (default true)
|
|
94
|
+
gfmAlerts: true, // > [!NOTE] → callouts (default true)
|
|
95
|
+
gfmFootnotes: true, // [^1] + [^1]: → footnote section (default false)
|
|
96
|
+
gfmMath: true, // $…$ / \(…\) inline + $$…$$ / \[…\] display math (default false)
|
|
97
|
+
dirAuto: true, // per-block dir="auto" for RTL/bidi text (default false)
|
|
98
|
+
unsafeHtml: false, // pass raw HTML through (default false — keep it false for untrusted input)
|
|
99
|
+
},
|
|
100
|
+
});
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Omitted fields use the defaults above, so `new FluxClient()` is unchanged.
|
|
104
|
+
Config is applied when the stream's parser is created and is **immutable** for
|
|
105
|
+
that stream (`reset()` keeps it; use a new client for different flags).
|
|
106
|
+
|
|
107
|
+
**Footnotes** (`gfmFootnotes`) work in streaming with one honest caveat: a
|
|
108
|
+
`[^1]` reference renders speculatively the moment it's seen (committed blocks
|
|
109
|
+
can't re-render), and the footnote **section is emitted at finalize**. So a
|
|
110
|
+
reference whose definition never arrives leaves a dangling link — the same
|
|
111
|
+
forward-reference cost as link reference definitions. Multiple references to
|
|
112
|
+
the same footnote each get a **unique id** (`fnref-N`, `fnref-N-2`, …) and the
|
|
113
|
+
definition lists **one backref per reference**. Remaining v1 limits:
|
|
114
|
+
single-block definitions (no continuation-indent / multi-paragraph) and no
|
|
115
|
+
nested footnotes. The section uses GitHub-style markup
|
|
116
|
+
(`<section class="footnotes">`, `<sup class="footnote-ref">`).
|
|
117
|
+
|
|
118
|
+
**Math** (`gfmMath`) recognizes both delimiter families LLMs emit — `$…$` /
|
|
119
|
+
`$$…$$` and LaTeX `\(…\)` / `\[…\]`. Inline math renders to
|
|
120
|
+
`<span class="math math-inline">…</span>`, display math to
|
|
121
|
+
`<div class="math math-display">…</div>` (and inline display to a `math-display`
|
|
122
|
+
span), each carrying the **HTML-escaped LaTeX as its text content** — exactly
|
|
123
|
+
what [KaTeX](https://katex.org)'s auto-render / `rehype-katex` consume. flux-md
|
|
124
|
+
stays **zero-dep**: it produces the KaTeX-ready markup and never processes the
|
|
125
|
+
body as markdown; you bring the KaTeX pass (or override `components.MathBlock`,
|
|
126
|
+
which receives the raw LaTeX as `text`). Single `$` uses the **pandoc rule** so
|
|
127
|
+
prose and currency stay literal — the opener needs a non-space to its right, the
|
|
128
|
+
closer a non-space to its left and no digit after it, so `$5 and $10` is **not**
|
|
129
|
+
math. A `$$`/`\[` block is **blank-line tolerant** (multi-line `\begin{aligned}…`
|
|
130
|
+
stays one block) and renders incrementally while streaming, like a code fence.
|
|
131
|
+
Off by default (so `$` in plain prose is untouched) — enable it per stream when
|
|
132
|
+
your model emits LaTeX.
|
|
133
|
+
|
|
134
|
+
**Bidirectional text** (`dirAuto`) emits `dir="auto"` on each block-level text
|
|
135
|
+
element (`p`, `h1`–`h6`, `blockquote`, `ul`/`ol`/`li`, `table`), so the browser
|
|
136
|
+
runs the Unicode bidi algorithm **per block** — an Arabic/Hebrew paragraph
|
|
137
|
+
renders RTL while the English one beside it stays LTR, with no JS direction
|
|
138
|
+
detection. Code blocks never get it (code is always LTR). This is the per-block
|
|
139
|
+
model GitHub uses; it's the right fix for the common failure mode of detecting
|
|
140
|
+
one direction for a whole mixed-language document. Off by default (strict
|
|
141
|
+
CommonMark output is unchanged); turn it on for RTL or mixed-direction content.
|
|
142
|
+
|
|
143
|
+
### `FluxMarkdown` (React)
|
|
144
|
+
|
|
145
|
+
Subscribes to a `FluxClient`, renders each block keyed by its stable parser-assigned ID. Memoized so unchanged blocks never re-reconcile.
|
|
146
|
+
|
|
147
|
+
```tsx
|
|
148
|
+
<FluxMarkdown client={client} />
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
#### Custom components / overrides
|
|
152
|
+
|
|
153
|
+
Pass a `components` map to replace how elements render — the same idea as
|
|
154
|
+
react-markdown's `components` prop, but the keys come in **two namespaces**:
|
|
155
|
+
|
|
156
|
+
```tsx
|
|
157
|
+
import { useMemo } from "react";
|
|
158
|
+
import { FluxClient, FluxMarkdown, type Components } from "flux-md";
|
|
159
|
+
|
|
160
|
+
function Message({ client }: { client: FluxClient }) {
|
|
161
|
+
// ⚠️ Memoize (or hoist to module scope). A fresh object every render busts
|
|
162
|
+
// FluxMarkdown's block memo, so every block re-parses on every patch.
|
|
163
|
+
const components: Components = useMemo(
|
|
164
|
+
() => ({
|
|
165
|
+
// tag-level (lowercase HTML names) — applied inside a block's HTML
|
|
166
|
+
table: (props) => <table className="rounded border" {...props} />,
|
|
167
|
+
a: (props) => <a target="_blank" rel="noreferrer" {...props} />,
|
|
168
|
+
h1: "h2", // a string value just swaps the tag
|
|
169
|
+
|
|
170
|
+
// block-kind (capitalized BlockKindTag) — replaces the whole block
|
|
171
|
+
CodeBlock: ({ text, language, open }) => (
|
|
172
|
+
<MyCodeBlockWithCopyButton code={text} lang={language} streaming={open} />
|
|
173
|
+
),
|
|
174
|
+
}),
|
|
175
|
+
[],
|
|
176
|
+
);
|
|
177
|
+
return <FluxMarkdown client={client} components={components} />;
|
|
178
|
+
}
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
**Tag-level** keys (`table`, `thead`, `tr`, `td`, `a`, `code`, `pre`, `h1`–`h6`,
|
|
182
|
+
`ul`, `ol`, `li`, `blockquote`, `p`, `img`, `del`, `input`, `hr`, …) replace that
|
|
183
|
+
element wherever it appears. The component receives the element's parsed
|
|
184
|
+
attributes (with `class`→`className` and `style` as an object) plus `children`.
|
|
185
|
+
|
|
186
|
+
**Block-kind** keys (`CodeBlock`, `Mermaid`, `MathBlock`, `Alert`, `Paragraph`,
|
|
187
|
+
`Heading`, `List`, `Blockquote`, `Table`, `Rule`, `Html`) replace the entire
|
|
188
|
+
block. The component receives [`BlockComponentProps`](#types): `{ block, html,
|
|
189
|
+
open, speculative }`, plus `text`/`language` for code/math blocks (the alert
|
|
190
|
+
type is at `block.kind.data.kind`).
|
|
191
|
+
|
|
192
|
+
Rules worth knowing:
|
|
193
|
+
|
|
194
|
+
- **There is no `node` prop.** flux-md has no hast tree; introspect via
|
|
195
|
+
`className` / `data-*` instead.
|
|
196
|
+
- **Open (streaming) blocks render via `innerHTML`** — their HTML is still
|
|
197
|
+
partial, so a tag-level override takes effect the moment the block commits.
|
|
198
|
+
- **No `components` prop ⇒ the original fast path** (`innerHTML`, byte-identical
|
|
199
|
+
output). The HTML→React conversion only runs for closed blocks when you
|
|
200
|
+
actually supply overrides, and is memoized per `(block id, html)`.
|
|
201
|
+
- For **code blocks** the built-in highlighter is the default; it is bypassed
|
|
202
|
+
(so your override wins) when you pass `components.CodeBlock`, `components.pre`,
|
|
203
|
+
or `components.code`.
|
|
204
|
+
|
|
205
|
+
### Types
|
|
206
|
+
|
|
207
|
+
```ts
|
|
208
|
+
interface Block {
|
|
209
|
+
id: number;
|
|
210
|
+
kind: { type: "Paragraph" | "Heading" | "CodeBlock" | "List" | ...; data?: unknown };
|
|
211
|
+
html: string; // safe to inject via dangerouslySetInnerHTML
|
|
212
|
+
open: boolean; // still being built (last block in active tail)
|
|
213
|
+
speculative: boolean; // closed by inference, may be revised
|
|
214
|
+
start: number;
|
|
215
|
+
end: number;
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
// Override map for <FluxMarkdown components={...} />
|
|
219
|
+
type Components = Record<string, React.ComponentType<any> | string>;
|
|
220
|
+
|
|
221
|
+
// Props a block-kind override receives (e.g. components.CodeBlock)
|
|
222
|
+
interface BlockComponentProps {
|
|
223
|
+
block: Block;
|
|
224
|
+
html: string;
|
|
225
|
+
open: boolean;
|
|
226
|
+
speculative: boolean;
|
|
227
|
+
text?: string; // decoded source — CodeBlock / MathBlock
|
|
228
|
+
language?: string; // info string — CodeBlock
|
|
229
|
+
}
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
`htmlToReact(html, components)` and `parseTrustedHtml(html)` are also exported
|
|
233
|
+
for advanced use (e.g. rendering a single block's HTML to a React tree yourself).
|
|
234
|
+
|
|
235
|
+
### `highlight(code, lang)`
|
|
236
|
+
|
|
237
|
+
Optional. Tiny native-RegExp tokenizer covering js/ts/tsx/jsx, rust, python, go, bash, sql, json, html, css. Unknown languages fall through to plain escaped text.
|
|
238
|
+
|
|
239
|
+
```ts
|
|
240
|
+
import { highlight } from "flux-md/highlight";
|
|
241
|
+
const html = highlight("const x = 1;", "ts");
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
## Coverage
|
|
245
|
+
|
|
246
|
+
**CommonMark 0.31: 100% (652/652 spec examples)** — every section, including
|
|
247
|
+
the hard ones (nested/loose lists, link reference definitions, link precedence,
|
|
248
|
+
lazy blockquote continuation). Plus GFM extensions: tables, strikethrough, task
|
|
249
|
+
lists, extended autolinks, GitHub alerts (`> [!NOTE]` → styled callouts),
|
|
250
|
+
footnotes (`[^1]` + `[^1]:`), and math (`$…$`, `$$…$$`, `\(…\)`, `\[…\]`).
|
|
251
|
+
Autolinks and alerts are on by default; footnotes and math are opt-in per stream
|
|
252
|
+
(see [Per-stream config](#per-stream-config)). See
|
|
253
|
+
`crates/flux-md-core/tests/{cmark_spec,gfm_spec,footnotes,math}.rs` for runners and floors.
|
|
254
|
+
|
|
255
|
+
GitHub alerts render to GitHub-compatible markup
|
|
256
|
+
(`<div class="markdown-alert markdown-alert-note">…`), so existing markdown CSS
|
|
257
|
+
styles them, and they're overridable as a block kind via `components.Alert`.
|
|
258
|
+
|
|
259
|
+
## What it doesn't do
|
|
260
|
+
|
|
261
|
+
By design, not yet, or only partially:
|
|
262
|
+
|
|
263
|
+
- **Raw HTML in markdown** — escaped by default, not passed through. (Security
|
|
264
|
+
default. A `setUnsafeHtml(true)` opt-in exists but must never be enabled for
|
|
265
|
+
untrusted input.)
|
|
266
|
+
- **Forward link references when streaming** — a `[ref]` used *before* its later
|
|
267
|
+
`[ref]: url` definition can't resolve until the definition arrives; one-shot
|
|
268
|
+
parsing handles it fully, streaming converges once the definition streams in.
|
|
269
|
+
- **Definition lists** — out of scope for v1.
|
|
270
|
+
- **KaTeX / Mermaid rendering** — flux-md emits KaTeX-ready math markup
|
|
271
|
+
(`<span>`/`<div class="math …">` with `gfmMath` on) and a `Mermaid` slot, but
|
|
272
|
+
stays zero-dep: bring your own KaTeX / mermaid pass (or a `components.MathBlock`
|
|
273
|
+
/ `components.Mermaid` override) for the actual SVG/MathML output.
|
|
274
|
+
- **Syntax highlighting on open code blocks** — deferred until close. This is a
|
|
275
|
+
deliberate perf choice.
|
|
276
|
+
|
|
277
|
+
## Security
|
|
278
|
+
|
|
279
|
+
flux-md is XSS-safe by default — its HTML output is meant to be injected via
|
|
280
|
+
`innerHTML` without a downstream sanitizer:
|
|
281
|
+
|
|
282
|
+
- **Raw HTML is escaped** (the `unsafe_html` / `setUnsafeHtml(true)` opt-in
|
|
283
|
+
disables this; **never enable it for untrusted input**).
|
|
284
|
+
- **Dangerous URL schemes are neutralized** in `<a href>` and `<img src>` —
|
|
285
|
+
`javascript:`, `vbscript:`, `data:text/html`, `data:text/javascript` become
|
|
286
|
+
`#`. The check runs on the *decoded* URL and strips characters browsers
|
|
287
|
+
ignore in the scheme, so obfuscations like `javascript:…`,
|
|
288
|
+
`javascript\:…`, `javascript:…`, and embedded tabs/newlines are caught,
|
|
289
|
+
not just the literal form. (See `crates/flux-md-core/tests/security.rs`.)
|
|
290
|
+
- **`htmlToReact` defends in depth**: it drops inline `on*` event-handler
|
|
291
|
+
attributes and runs URL attributes through the same scheme filter. It's
|
|
292
|
+
intended for flux-md's own (already-sanitized) HTML; if you hand it arbitrary
|
|
293
|
+
third-party HTML, these guards are your only line of defense — prefer a
|
|
294
|
+
dedicated HTML sanitizer for genuinely hostile input.
|
|
295
|
+
|
|
296
|
+
## Scaling
|
|
297
|
+
|
|
298
|
+
`FluxClient`s share a **worker pool** (`getDefaultPool()`), so concurrency
|
|
299
|
+
doesn't oversubscribe OS threads. Worker creation is lazy and load-aware:
|
|
300
|
+
|
|
301
|
+
- **1 stream → 1 worker**, and each new stream gets its own worker until the cap
|
|
302
|
+
(`Math.min(navigator.hardwareConcurrency || 4, 8)`) — identical to the
|
|
303
|
+
per-worker behavior for small stream counts.
|
|
304
|
+
- **Past the cap**, new streams attach to the least-loaded worker, which
|
|
305
|
+
multiplexes them (a `FluxParser` per stream id). So **50 concurrent streams
|
|
306
|
+
run on ≤8 workers (~6 each)**, not 50 threads.
|
|
307
|
+
|
|
308
|
+
`destroy()` frees a stream's parser and keeps the worker warm for its siblings;
|
|
309
|
+
the workers persist for the life of the page. Need isolation or manual
|
|
310
|
+
teardown? Construct your own `new FluxPool(factory, cap)` and pass it to
|
|
311
|
+
`new FluxClient(pool)`, or call `pool.disposeAll()`.
|
|
312
|
+
|
|
313
|
+
`getDefaultPool()` is **browser-only** (it constructs `Worker`s) and is a
|
|
314
|
+
**per-page singleton** — don't rely on it in SSR/RSC. For isolation between
|
|
315
|
+
independent feature areas, give each its own `new FluxPool()`.
|
|
316
|
+
|
|
317
|
+
### Long documents — `virtualize`
|
|
318
|
+
|
|
319
|
+
For very long documents (hundreds+ of blocks), pass `virtualize` to apply CSS
|
|
320
|
+
`content-visibility: auto` (+ a per-kind `contain-intrinsic-size`) to **closed**
|
|
321
|
+
blocks, so the browser skips style/layout/paint for off-screen content:
|
|
322
|
+
|
|
323
|
+
```tsx
|
|
324
|
+
<FluxMarkdown client={client} virtualize />
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
It's opt-in (off by default — short docs gain nothing) and never defers the
|
|
328
|
+
streaming tail (open/speculative blocks always render fully, so no flicker
|
|
329
|
+
where you're looking). It cuts **rendering cost, not DOM node count** — nodes
|
|
330
|
+
stay in the document (search, anchors, and a11y all keep working), they just
|
|
331
|
+
don't lay out while off-screen. Measured on a ~1800-block demo, an off-screen
|
|
332
|
+
**layout pass is ~7× cheaper** (≈1980ms → ≈284ms over 30 forced relayouts),
|
|
333
|
+
identical node count — i.e. whenever the browser would otherwise lay out
|
|
334
|
+
off-screen blocks (initial paint, resize, font load, scroll), that work is
|
|
335
|
+
skipped. No JS windowing, no scroll math, no dep — the browser does it natively.
|
|
336
|
+
|
|
337
|
+
Works best when `<FluxMarkdown>`'s parent uses normal block flow; a `flex`/`grid`
|
|
338
|
+
parent can interact with `contain-intrinsic-size` in surprising ways.
|
|
339
|
+
|
|
340
|
+
### Stick to bottom while streaming — `stickToBottom`
|
|
341
|
+
|
|
342
|
+
Pass `stickToBottom` and the view **follows the streaming tail, releasing when
|
|
343
|
+
the user scrolls up** (and re-locking when they scroll back near the bottom) —
|
|
344
|
+
the behavior every chat UI wants. It's **CSS-only** (CSS Scroll Snap, no JS, no
|
|
345
|
+
scroll listeners): flux-md emits a bottom snap target; you add one line to your
|
|
346
|
+
scroll container:
|
|
347
|
+
|
|
348
|
+
```tsx
|
|
349
|
+
<div className="chat-scroller"> {/* your existing scroll container */}
|
|
350
|
+
<FluxMarkdown client={client} stickToBottom />
|
|
351
|
+
</div>
|
|
352
|
+
```
|
|
353
|
+
```css
|
|
354
|
+
.chat-scroller { overflow-y: auto; scroll-snap-type: y proximity; }
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
That's the whole feature. `proximity` (not `mandatory`) is what lets the user
|
|
358
|
+
scroll up freely. Note it **follows** the bottom — during very fast streaming
|
|
359
|
+
the lock can lag by a few px between snaps; it doesn't *hard-pin*. Re-snap on
|
|
360
|
+
content growth is solid in Chromium/Firefox; **Safari is weaker** at
|
|
361
|
+
re-snapping during streaming, so treat smooth following there as best-effort.
|
|
362
|
+
|
|
363
|
+
> **Metrics note:** because workers are shared, `getMetrics().wasmMemoryBytes`
|
|
364
|
+
> is the *shared* worker's heap — clients on the same worker report the same
|
|
365
|
+
> value. Aggregate with `Math.max`, not a sum.
|
|
366
|
+
|
|
367
|
+
## Architecture
|
|
368
|
+
|
|
369
|
+
```
|
|
370
|
+
┌── main thread ────────────────────────┐
|
|
371
|
+
│ FluxMarkdown — React, useSyncStore │
|
|
372
|
+
│ FluxClient — message routing │
|
|
373
|
+
└──┬──── postMessage(chunk) ────────────┘
|
|
374
|
+
▼
|
|
375
|
+
┌── Web Worker ─────────────────────────┐
|
|
376
|
+
│ worker.ts — coalesces chunks per │
|
|
377
|
+
│ microtask, calls WASM │
|
|
378
|
+
└──┬──── ffi ───────────────────────────┘
|
|
379
|
+
▼
|
|
380
|
+
┌── Rust → WASM (~150 KB after opt) ────┐
|
|
381
|
+
│ StreamParser: │
|
|
382
|
+
│ buffer: append-only │
|
|
383
|
+
│ committed_offset │
|
|
384
|
+
│ [committed_blocks] │
|
|
385
|
+
│ [active_blocks] (re-parsed tail) │
|
|
386
|
+
│ │
|
|
387
|
+
│ scanner.rs → raw blocks │
|
|
388
|
+
│ inline.rs → emphasis stack + safe │
|
|
389
|
+
│ link/code rendering │
|
|
390
|
+
│ render.rs → HTML with URL sanitize │
|
|
391
|
+
└───────────────────────────────────────┘
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
Active tail re-parses on each chunk; committed blocks are frozen forever. Each block's ID is monotonic and is *preserved* across re-parses when its start offset and kind match a previously-seen active block — so React's keyed reconciliation reuses the DOM instead of remounting.
|
|
395
|
+
|
|
396
|
+
## License
|
|
397
|
+
|
|
398
|
+
MIT.
|
package/package.json
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "flux-md",
|
|
3
|
+
"version": "0.3.1",
|
|
4
|
+
"description": "Zero-dep streaming markdown for the browser. Rust→WASM core, Web Worker per stream, incremental parse with speculative closure.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "./src/index.ts",
|
|
7
|
+
"types": "./src/index.ts",
|
|
8
|
+
"exports": {
|
|
9
|
+
".": "./src/index.ts",
|
|
10
|
+
"./client": "./src/client.ts",
|
|
11
|
+
"./react": "./src/react.tsx",
|
|
12
|
+
"./highlight": "./src/hi.ts",
|
|
13
|
+
"./types": "./src/types.ts"
|
|
14
|
+
},
|
|
15
|
+
"files": [
|
|
16
|
+
"src",
|
|
17
|
+
"README.md",
|
|
18
|
+
"CHANGELOG.md"
|
|
19
|
+
],
|
|
20
|
+
"peerDependencies": {
|
|
21
|
+
"react": ">=18"
|
|
22
|
+
},
|
|
23
|
+
"peerDependenciesMeta": {
|
|
24
|
+
"react": { "optional": true }
|
|
25
|
+
},
|
|
26
|
+
"devDependencies": {
|
|
27
|
+
"@types/react": "^18.3.12",
|
|
28
|
+
"@types/react-dom": "^18.3.1",
|
|
29
|
+
"react": "^18.3.1",
|
|
30
|
+
"react-dom": "^18.3.1",
|
|
31
|
+
"typescript": "^5.6.3"
|
|
32
|
+
},
|
|
33
|
+
"scripts": {
|
|
34
|
+
"test": "bun test",
|
|
35
|
+
"prepublishOnly": "cd ../.. && bun run build:wasm"
|
|
36
|
+
},
|
|
37
|
+
"keywords": ["markdown", "streaming", "wasm", "rust", "react", "incremental", "llm", "ai", "math", "katex", "latex", "bidi", "rtl"],
|
|
38
|
+
"license": "MIT",
|
|
39
|
+
"publishConfig": {
|
|
40
|
+
"access": "public"
|
|
41
|
+
},
|
|
42
|
+
"repository": {
|
|
43
|
+
"type": "git",
|
|
44
|
+
"url": "git+https://github.com/siinghd/flux-md.git"
|
|
45
|
+
},
|
|
46
|
+
"homepage": "https://md.hsingh.app",
|
|
47
|
+
"bugs": {
|
|
48
|
+
"url": "https://github.com/siinghd/flux-md/issues"
|
|
49
|
+
}
|
|
50
|
+
}
|