@ai-react-markdown/engine 2.4.3 → 2.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +95 -0
- package/dist/index.cjs +440 -83
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +109 -58
- package/dist/index.d.ts +109 -58
- package/dist/index.dev.cjs +440 -83
- package/dist/index.dev.cjs.map +1 -1
- package/dist/index.dev.js +440 -81
- package/dist/index.dev.js.map +1 -1
- package/dist/index.js +440 -81
- package/dist/index.js.map +1 -1
- package/package.json +12 -2
package/README.md
CHANGED
|
@@ -1,5 +1,18 @@
|
|
|
1
1
|
# @ai-react-markdown/engine
|
|
2
2
|
|
|
3
|
+
[](https://www.npmjs.com/package/@ai-react-markdown/engine)
|
|
4
|
+
[](https://www.npmjs.com/package/@ai-react-markdown/engine)
|
|
5
|
+
[](https://bundlephobia.com/package/@ai-react-markdown/engine)
|
|
6
|
+
[](https://www.typescriptlang.org/)
|
|
7
|
+
|
|
8
|
+
[](https://nodejs.org/)
|
|
9
|
+
[](#install)
|
|
10
|
+
[](https://github.com/AIEPhoenix/ai-react-markdown/blob/main/LICENSE)
|
|
11
|
+
|
|
12
|
+
[](https://github.com/AIEPhoenix/ai-react-markdown/actions/workflows/ci.yml)
|
|
13
|
+
[](https://github.com/AIEPhoenix/ai-react-markdown/actions/workflows/release.yml)
|
|
14
|
+
[](https://github.com/AIEPhoenix/ai-react-markdown)
|
|
15
|
+
|
|
3
16
|
Framework-agnostic Markdown engine for [ai-react-markdown](https://github.com/AIEPhoenix/ai-react-markdown) — incremental parsing, LaTeX preprocessing, definition/footnote machinery, and the unified plugin pipeline. Takes Markdown text in, produces a [hast](https://github.com/syntax-tree/hast) tree plus incremental-parse state out; rendering that tree is the job of a framework adapter such as [`@ai-react-markdown/core`](https://www.npmjs.com/package/@ai-react-markdown/core) (React).
|
|
4
17
|
|
|
5
18
|
> **Status: internal supplier.** This package exists to serve
|
|
@@ -10,12 +23,94 @@ Framework-agnostic Markdown engine for [ai-react-markdown](https://github.com/AI
|
|
|
10
23
|
> instead; this package is interesting to you only if you are building a
|
|
11
24
|
> framework adapter of your own.
|
|
12
25
|
|
|
26
|
+
## What's inside
|
|
27
|
+
|
|
28
|
+
Everything is exported from the package root (`import { … } from '@ai-react-markdown/engine'`); the barrel is grouped by layer:
|
|
29
|
+
|
|
30
|
+
| Layer | Modules | Highlights |
|
|
31
|
+
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
32
|
+
| Preprocessors | `preprocessors/latex`, `preprocessors/remend`, `preprocessAIMDContent` | `preprocessLaTeX(text)` (currency `$`, `\[…\]` / `\(…\)` normalization, code-fence and inline-code protection), `createIncrementalLatexPreprocessor()` for append-only streams, `remend` for unterminated-markup mending |
|
|
33
|
+
| Incremental parsing | `incrementalParse/*` | `advanceIncrementalParse(state, content, options)` — the prefix-freeze engine: a line scanner decides a verified-safe freeze boundary, only the tail re-parses, and the two trees are spliced; every frame is deep-equal to a full parse (enforced by the arbiter suites) or falls back to one |
|
|
34
|
+
| Pipeline assembly | `markdown/*`, `pluginChain`, `plugins/catalog`, `customMdastHandlers`, `remarkInjectPhantomDefs`, `rehypeRebaseHashLinks`, `rehypeFooterAdorn` | `buildCoreRemarkPlugins` / `buildCoreRehypePlugins` / `buildCoreRemarkRehypeOptions` — the exact chains the React renderer uses; the sealed engine-plugin catalog (`highlight`, `definitionList`, `removeComments`, `smartypants`, `pangu`, `defaultEnginePlugins`) |
|
|
35
|
+
| Cross-chunk coordination | `documentRegistry`, `collectDefLabels`, `extractContributions`, `extractDefBodiesFromHast`, `crossChunkUrlSanitize` | `createRegistry()` — the per-document store that numbers footnotes and resolves link definitions across chunks; `sanitizeCrossChunkUrl()` mirrors the standalone two-gate URL policy |
|
|
36
|
+
| Sanitization | `sanitizeSchema`, `extendSanitizeSchema`, `markdown/urlTransform` | The library default `rehype-sanitize` schema (read-only singleton — clone with `extendSanitizeSchema`), `defaultUrlTransform` |
|
|
37
|
+
| Streaming | `smoothStream/controller` | `createSmoothStreamController()` — the framework-agnostic typewriter pacing state machine behind `<AIMarkdownSmoothStream>`, with `SMOOTH_STREAM_PACING_PRESETS` |
|
|
38
|
+
| Leaves | `hastPredicates`, `normalizeId`, `shortenDocumentId`, `devStageTimings`, `fixtures/scenarios` | Small pure helpers and the shared test corpus |
|
|
39
|
+
|
|
40
|
+
## Install
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
npm install @ai-react-markdown/engine
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Dual ESM/CJS build with types for both. No React dependency. The only peer is `katex` (`^0.16 || ^0.17`, **optional** — needed only if you render math). It ships transitively via `rehype-katex`, so hoisted installers resolve it automatically; strict-isolation installers (yarn PnP, `pnpm --node-linker=isolated`) must install it explicitly in your app.
|
|
47
|
+
|
|
48
|
+
## Example: the LaTeX preprocessor on its own
|
|
49
|
+
|
|
50
|
+
```ts
|
|
51
|
+
import { preprocessLaTeX } from '@ai-react-markdown/engine';
|
|
52
|
+
|
|
53
|
+
preprocessLaTeX('Price is $100, and \\(x^2\\) is inline math.');
|
|
54
|
+
// → 'Price is \\$100, and $$x^2$$ is inline math.'
|
|
55
|
+
// (currency `$` escaped; `\\(…\\)` normalized to the `$$…$$` form remark-math's inline rule accepts)
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The same function runs inside `@ai-react-markdown/core` before every parse; the incremental variant (`createIncrementalLatexPreprocessor`) reuses work across append-only frames.
|
|
59
|
+
|
|
60
|
+
## Example: driving the incremental parser
|
|
61
|
+
|
|
62
|
+
```ts
|
|
63
|
+
import {
|
|
64
|
+
advanceIncrementalParse,
|
|
65
|
+
buildCoreRemarkPlugins,
|
|
66
|
+
buildCoreRehypePlugins,
|
|
67
|
+
buildCoreRemarkRehypeOptions,
|
|
68
|
+
defaultEnginePlugins,
|
|
69
|
+
sanitizeSchema,
|
|
70
|
+
} from '@ai-react-markdown/engine';
|
|
71
|
+
|
|
72
|
+
const options = {
|
|
73
|
+
remarkPlugins: buildCoreRemarkPlugins(defaultEnginePlugins),
|
|
74
|
+
rehypePlugins: buildCoreRehypePlugins(sanitizeSchema, ''),
|
|
75
|
+
remarkRehypeOptions: buildCoreRemarkRehypeOptions(false),
|
|
76
|
+
depsKey: [],
|
|
77
|
+
defListEnabled: false,
|
|
78
|
+
};
|
|
79
|
+
|
|
80
|
+
let state = null;
|
|
81
|
+
for (const frame of ['# Hello', '# Hello\n\nworld', '# Hello\n\nworld and more']) {
|
|
82
|
+
const result = advanceIncrementalParse(state, frame, options);
|
|
83
|
+
state = result.nextState;
|
|
84
|
+
// result.hast — the full-document hast for this frame
|
|
85
|
+
// result.usedIncremental / result.boundary — whether the frame spliced, and where
|
|
86
|
+
}
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
`AdvanceOptions` is documented in `incrementalParse/advanceIncrementalParse.ts`; the React renderer's `MarkdownContent` is the reference consumer.
|
|
90
|
+
|
|
91
|
+
## Verification
|
|
92
|
+
|
|
93
|
+
The incremental engine ships with a five-layer equivalence stack (fixture pins, fuzz arbiter, direction battery, exhaustive census, arbiter-sensitivity meta-suite) plus a release-gate soak (`scripts/run-soak.sh`); the full record lives in `src/experiments/prefixFreeze/README.md`. Every reachable divergence found so far is pinned as a deterministic test.
|
|
94
|
+
|
|
13
95
|
## Runtime support
|
|
14
96
|
|
|
15
97
|
Pure computation over strings and syntax trees: no DOM access, no
|
|
16
98
|
Node-only APIs, and no unguarded environment reads. Runs in browsers,
|
|
17
99
|
Node, workers, and embedded JS runtimes (e.g. Hermes/JavaScriptCore).
|
|
18
100
|
|
|
101
|
+
## Versioning
|
|
102
|
+
|
|
103
|
+
Lockstep with `@ai-react-markdown/core`, which pins this package **exactly** — the export surface follows what core consumes and may change in any release before 3.0.0 (see the status note above). Release notes: [release highlights](https://github.com/AIEPhoenix/ai-react-markdown/blob/main/docs/release-highlights.md).
|
|
104
|
+
|
|
105
|
+
## Package family
|
|
106
|
+
|
|
107
|
+
| Package | Role | Version policy |
|
|
108
|
+
| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
|
109
|
+
| [`@ai-react-markdown/core`](https://www.npmjs.com/package/@ai-react-markdown/core) | The React renderer — `<AIMarkdown>`, `<AIMarkdownSmoothStream>`, `<AIMarkdownDocuments>`, hooks, providers | Release train |
|
|
110
|
+
| [`@ai-react-markdown/mantine`](https://www.npmjs.com/package/@ai-react-markdown/mantine) | Mantine UI bindings — themed typography, code-highlight tabs, Mermaid, color-scheme wiring | Release train (lockstep with core) |
|
|
111
|
+
| [`@ai-react-markdown/engine`](https://www.npmjs.com/package/@ai-react-markdown/engine) | Framework-agnostic engine — incremental parsing, LaTeX preprocessing, plugin pipeline, cross-chunk registry | Release train (lockstep, pinned exactly by core; internal supplier) |
|
|
112
|
+
| [`@ai-react-markdown/remark-mark-highlight`](https://www.npmjs.com/package/@ai-react-markdown/remark-mark-highlight) | remark plugin for `==mark==` highlight syntax | Independent semver |
|
|
113
|
+
|
|
19
114
|
## License
|
|
20
115
|
|
|
21
116
|
MIT
|