@stream-mdx/unified 0.0.0-stage → 1.0.0

package/CHANGELOG.md

@@ -0,0 +1,102 @@
+# @stream-mdx/unified
+
+## 1.0.0
+
+### Major Changes
+
+- 28f8a1e: Remove `blockRenderer` and the `BlockComponents` registry. Use `components: NodePresentations` for native block wrappers and views, including unified rendering.
+- 59622ba: Add `components: NodePresentations` to static/server and unified rendering, using the same native-node wrapper/view dispatch as streaming. Unified's HAST element mapping moves to `hastComponents`.
+- 159a237: Use source-owned records for root code fences, tables, and paragraphs, with bounded line, row, and span updates and on-demand full-source reads. Patch consumers must accept wire protocol 2.0, replace `appendLines` with `appendCode`, adopt `appendParagraph` and `updateParagraphSpans`, and handle native table operations; the retired operation has no compatibility alias. Plugins must use the source snapshot and on-demand `readBlocks` document context instead of a retained `blocks` array.
+
+  Live consumers now initialize one canonical root input document from the
+  `INITIALIZED` snapshot baseline and retain later admitted bytes with ordered
+  semantic `appendInputSource` root patches. Missing live baselines are
+  incompatible; cold `Block[]` and standalone snapshots keep only their honest
+  per-block source slices. Native code, table, and paragraph windows share the
+  consumer input document while retaining independent owner generations and
+  exact newly owned `sourceAppend` suffixes. Empty baselines are valid, and static
+  artifact version 1 is unchanged.
+
+  Preserve owner identity and retirement guards when replacing active snapshots. Reject regressing revisions and source frontiers, while allowing valid source-stable table and paragraph dependency meaning updates after finalization. Same-generation finalized source cannot grow or reopen. Keep final code highlighting and fallback presentation intact without retransmitting source in presentation-only patches.
+
+  For paragraph owners after raw Markdown EOF, reopen only the actual trailing paragraph under a fresh owner generation. Do not fuse intervening definition or root source into an earlier paragraph. Keep genuinely unowned provisional following roots eligible for continuation.
+
+  Do not re-finalize completed paragraph owners when later roots extend the shared input. Preserve their original parser frontier and captured source while still publishing valid late dependency changes.
+
+  Queue imperative append and finalize calls behind their matching initialization acknowledgement. Ignore errors from detached or superseded initializations without changing public worker attachment readiness, and keep detached snapshot callbacks callable.
+
+  Render managed paragraph children through the native span view while preserving registered source props and finalized block overrides. Default appends avoid global block/MDX scans; legacy full-source custom props use an explicit cold projection. Cold Block exports also retain admitted open native tables.
+
+  Blocks whose rendering depends on HTML stay on the base HTML grouping path. Genuine inline-HTML paragraphs and HTML-bearing containers retire native owners instead of adopting or rehoming them; tail-local auto-close and native ownership for non-HTML blocks remain.
+
+- 072e0f1: Expose generation-bound source reading and resolved link-reference metadata to node presentations, and report unified failures with block ID, type, and finalized state.
+
+- Release the coordinated 1.0.0 package line. Upgrade all installed StreamMDX packages together and recopy the hosted worker from the matching version. Internal runtime dependencies remain exact versions; the optional unified adapter's StreamMDX peers require the 1.x line.
+
+  Migration from 0.5.0:
+
+  - Remove imports of `@stream-mdx/plugins/registry`, `/base`, and `/math`, their `stream-mdx/plugins/*` convenience re-exports, and `@stream-mdx/worker/streaming/custom-matcher`. `PluginRegistry`, `globalPluginRegistry`, legacy match handlers, and legacy math plugin factories are no longer public. Use native Lezer extensions, serializable `SyntaxRuleSpec` rules, and per-engine `DocumentPluginRegistry` document plugins. Native extension functions belong in the engine or custom worker bundle, not in structured-clone `INIT` messages.
+  - `MarkdownEngine` owns its parser, queue, document state, plugins, and caches. Construct it with `{ post, extensions? }`, send configuration and document resets through `handle({ type: "INIT", ... })`, and call terminal `dispose()` when finished. Use public `serveMarkdownWorker` and `createInlineEngineWorker` factories instead of old global runtime plumbing. Worker URLs and actual Worker factories create renderer-owned workers that terminate on detach; an existing Worker is caller-owned. Caller-created inline adapters also need caller-owned termination. See `docs/PUBLIC_API.md` and `docs/PLUGIN_ABI.md` for supported wiring.
+    `managedWorker={true}` requires a supplied Worker instance or caller-accessible factory and caller-owned initialization/document commands. URL/default workers and `engine="main-thread"` are rejected, even with a supplied worker source. Managed auto cannot silently fall back inline. Borrowed instances remain caller-owned; factory-created workers remain component-owned and terminate on detach.
+  - Provisional lookahead inputs over 512 characters now return raw text before completeness scanning, including closed strings. Oversized finalized/nonprovisional complete content is preserved; within-budget finalized projection retains its behavior. Internal defaults are `maxScanChars: 512`, `maxNewlines: 2`, `maxSyntheticOps: 6`, `maxNestingDepth: 8`, `maxValidationFailures: 1`, and `maxProviderMs: 5`. Provider timers start before scanning; these caps are not React props or latency guarantees.
+  - Core root exports no longer expose Lezer converter, math extension factory, link-label normalization, entity decoder, or default/regex plugin adapter internals. Use retained parser/preparation APIs and public worker/compiler entries. Owning parser settings remain authoritative, including disabled math in nested snapshot fallbacks.
+  - `ComponentRegistry` treats a block component as custom only when it differs from the built-in default. Custom paragraph and quote wrappers must render the supplied `children` projection to retain native inline, HTML, and nested-block ownership. Do not reconstruct those children from raw source.
+  - `CodeFenceRendererProps.renderCode` is a React node, not a callback. Return it directly as a fallback. `language`, fence metadata, and emitted `language-*` classes preserve the authored info word, such as `js`, rather than the canonical Shiki grammar name. Renderer lookups still fall back to canonical names and accept aliases. Unlabeled and indented code has no language class or `data-language` attribute.
+  - `restart()` cancels the current async iterator generation; it does not replay a consumed iterator. Supply a fresh iterator for a new stream. Renderer-owned text replacements reset the engine document, while caller-initialized managed workers retain their document ownership.
+  - URL transformation and HTML/style sanitization now apply across native client rendering, SSR, and the optional unified adapter. Rejected URLs are omitted; custom components must honor the supplied sanitized attributes and children. `allowedElements` and `disallowedElements` are mutually exclusive.
+
+  The remaining consumed changesets describe the new opt-in renderers, controls, animation, table virtualization, themes, native extension APIs, and correctness/security fixes. These features do not enable themselves when upgrading. See the root README migration section and the generated package changelogs before upgrading custom integrations.
+
+### Minor Changes
+
+- 7879ed5: Add isolated MarkdownEngine instances, the public worker/engine entry, custom bundled worker serving, and a main-thread Worker adapter. React supports explicit worker, main-thread, and auto execution modes with a dynamically imported creation-time fallback. Updating onError no longer restarts streamed content.
+
+  Add serializable inline syntax rules with capture attributes, recursively parsed content, lookahead, streaming anticipation, and invalid-INIT recovery. Native worker entries accept Lezer markdown/inline extensions and instance document plugins. HTML supports per-tag attribute lists, mapped components, and opt-in markdown children while retaining literal raw-block defaults and respecting disabled math. Preserve whitespace beyond the required four columns on internal indented-code blank lines, including nested containers. Include invisible reference definitions when determining loose lists. Recognize bare FTP links with native GFM URL punctuation and parenthesis handling.
+
+  Add code fence renderer registration, React-node code fallbacks, math renderer overrides, structured diff views, and sandboxed HTML artifacts with CSP and host link routing. Mermaid accepts the code fence renderer contract and confines invalid-diagram output to its block.
+
+  Add the optional unified adapter for finalized-block remark/rehype transforms, explicit block routing, container replacement, synchronous SSR, asynchronous native fallback, error callbacks, and bounded per-renderer caches. Restore finalized heading levels from engine metadata before running remark and rehype plugins, preserving multiline Setext content and soft breaks. Cover compiler output and real inline-transport updates through the store-backed DOM renderer, including provisional rendering and terminal finalization. Document public exports and replace obsolete plugin factory examples.
+
+  React's minor version change covers public props (engine, codeRenderers, mathRenderer, blockRenderer) and renderer extension modes. Table virtualization is recorded separately in measured-table-rows, and nested block snapshot fixes are recorded in prepared-nested-blocks.
+
+### Patch Changes
+
+- e02e7cf: Security hardening: render-time URL transformation across client and SSR, raw HTML attribute and style sanitization, unified HAST URL sanitization, and ArtifactFrame height validation.
+
+  - `@stream-mdx/core`: Replaced dead URL sanitization code with `defaultUrlTransform(url, attribute, context)` which sanitizes URLs for `href` and `src` attributes. Rejects `javascript:`, `vbscript:`, `file:`, and unapproved `data:` URLs (allowing only `data:image/(png|gif|jpeg|webp|avif)` on `src`), while permitting relative paths, fragments, and custom schemes so CommonMark conformance stays at 675/675. In HTML schemas, `on*` event handlers and `srcdoc` attributes are strictly forbidden regardless of user `allowedTags` configuration, and `script` tags are never permitted. Exported `sanitizeStyle` for DOM-free CSS style sanitization.
+  - `@stream-mdx/react`: Added `urlTransform` prop to `<StreamingMarkdown />`, `ComponentRegistry`, and `MarkdownBlocksRenderer`. Applied at render time across link and image node views, raw HTML attributes (`href`, `src`, `action`, `formaction`, `xlink:href`), SSR rendering, and `openLink`/`linkSafety` navigation (preventing unsafe scheme execution). Raw HTML `style` attributes now enforce the DOM-free CSS property allowlist (dropping `position`, `url()`, backslash escapes, and expressions while retaining colors and fonts). `ArtifactFrame` strictly verifies `Number.isFinite(data.height)` and clamps between 50 and 10000 px, ignoring `NaN` and `Infinity`.
+  - `@stream-mdx/unified`: Added `urlTransform` option to `createUnifiedRenderer` that applies URL sanitization via a final HAST transformation pass after user rehype plugins without stripping user classes.
+
+- Updated dependencies [1669abb]
+- Updated dependencies [0e96940]
+- Updated dependencies [5833985]
+- Updated dependencies [4ed0ed3]
+- Updated dependencies [9c482bd]
+- Updated dependencies [150a0ac]
+- Updated dependencies [7879ed5]
+- Updated dependencies [73bdb36]
+- Updated dependencies [38273db]
+- Updated dependencies [fdaa309]
+- Updated dependencies [032136c]
+- Updated dependencies [e2c13e5]
+- Updated dependencies [95be0bf]
+- Updated dependencies
+- Updated dependencies [e02e7cf]
+- Updated dependencies
+  - @stream-mdx/core@1.0.0
+  - @stream-mdx/react@1.0.0
+
+## 0.5.0
+
+### Minor Changes
+
+- Align the unified adapter with the StreamMDX `0.5.0` release and require core and React `>=0.5.0`.
+- Support finalized-block routing, container replacement, synchronous SSR, asynchronous native fallback, error callbacks, and bounded renderer caches.
+- Restore heading levels and link references before plugin transforms and apply URL sanitization after user rehype plugins.
+- Bundle the unified pipeline dependencies so both the ESM and CJS entries can execute plugins in packed installs.
+
+## 0.4.0
+
+### Minor Changes
+
+- Initial release of `@stream-mdx/unified`: remark and rehype unified plugin adapter for StreamMDX.

package/README.md

@@ -1,3 +1,75 @@
-# Temporary Holding Version
+# @stream-mdx/unified
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Remark and Rehype unified plugin adapter for StreamMDX (`StreamingMarkdown`).
+
+Enables `react-markdown` and `Streamdown` users to bring their existing `remarkPlugins` and `rehypePlugins` to StreamMDX without sacrificing streaming performance.
+
+## How It Works
+
+- **Finalized-only execution**: Unified plugins run only on finalized blocks. Provisional blocks continue to use the native StreamMDX renderer.
+- **Sync and async pipelines**: Synchronous plugins execute once per cache key and support `renderToStaticMarkup` and `renderToString`. Asynchronous plugins retain the default rendered element while pending and commit their result on the client.
+- **Async SSR behavior**: Server rendering with async plugins returns the default StreamMDX element. It does not wait for plugin completion.
+- **Deduplication and stale results**: Concurrent jobs for the same cache key share one transformation. Older jobs may finish after content or metadata changes, but their results cannot replace the newer block.
+- **Bounded cache**: Finalized transformations use a bounded LRU cache, with 500 entries by default. Keys include `block.id`, block type, reconstructed markdown source, and resolved link references.
+- **Link reference resolution**: Definitions from `block.payload.meta.linkReferences` are prepended before parsing, so routed blocks retain resolved reference links and titles.
+- **Container replacement**: Routed lists and blockquotes run their canonical raw markdown through unified and replace native child rendering, including prepared child snapshots. Set `blockTypes` to exclude those containers when you need native code, math, or custom syntax renderers inside them. Unrouted containers retain their native element.
+- **Error handling**: Unified processing errors reach `onError` with the block's ID, type, and finalized state, while retaining the default rendered element. Promise rejections and callback-style transformer errors use the same fallback.
+- **Zero bundle weight in `@stream-mdx/react`**: Heavy unified and AST parser dependencies remain isolated within `@stream-mdx/unified`.
+
+## Installation
+
+```bash
+npm install @stream-mdx/unified
+```
+
+## Usage
+
+```tsx
+import { StreamingMarkdown } from "@stream-mdx/react";
+import { createUnifiedRenderer } from "@stream-mdx/unified";
+import rehypeSlug from "rehype-slug";
+import rehypeAutolinkHeadings from "rehype-autolink-headings";
+
+const unifiedRenderer = createUnifiedRenderer({
+  rehypePlugins: [rehypeSlug, rehypeAutolinkHeadings],
+  components: {
+    heading: {
+      wrapper: ({ children, metadata }) => (
+        <section data-finalized={metadata.isFinalized}>{children}</section>
+      ),
+    },
+  },
+  hastComponents: {
+    a: (props) => <a {...props} className="text-blue-500 hover:underline" />,
+  },
+  onError: (error, context) => {
+    console.error(`Unified plugin error in block ${context.id}:`, error);
+  },
+
+export function Doc({ text }: { text: string }) {
+  return (
+    <StreamingMarkdown
+      text={text}
+      components={unifiedRenderer}
+    />
+  );
+}
+```
+
+## Options
+
+`createUnifiedRenderer(options)` accepts:
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `remarkPlugins` | `PluggableList` | `[]` | List of remark plugins or plugin tuples `[plugin, options]` |
+| `rehypePlugins` | `PluggableList` | `[]` | List of rehype plugins or plugin tuples `[plugin, options]` |
+| `components` | `NodePresentations` | `{}` | Node wrapper/view registry shared with the streaming and static React renderer |
+| `hastComponents` | `Record<string, Component>` | `{}` | Custom React component mapping passed to `hast-util-to-jsx-runtime` |
+| `blockTypes` | `string[]` | `['paragraph', 'heading', 'blockquote', 'list', 'table', 'code', 'hr']` | Block types to route to unified pipeline |
+| `cacheSize` | `number` | `500` | Maximum number of rendered blocks to retain in LRU cache |
+| `math` | `boolean` | `false` | Enable `remark-math` support |
+| `gfm` | `boolean` | `true` | Enable `remark-gfm` support |
+| `onError` | `(error: Error, context: { id: string; type: string; finalized: boolean }) => void` | `undefined` | Callback invoked when unified processing fails for a block |
+
+> **Note on Block Types**: Standard CommonMark/GFM blocks are routed by default. Specialized block types like `html` and `mdx` are opt-in via `blockTypes` (e.g. `blockTypes: ['paragraph', 'html']`). When routing `html` blocks, ensure your unified pipeline includes appropriate plugins (e.g. `rehype-raw`) if you wish to parse embedded raw HTML.