@toaq-oss/omni-mdx 0.1.11 → 0.1.13

package/README.md

@@ -1,221 +1,140 @@
 # @toaq-oss/omni-mdx
 
-The React/Next.js visual rendering engine of the TOAQ-oss MDX ecosystem.
+**The high-performance MDX engine.** A unified React & Next.js rendering layer powered by a dual Rust backend (Native Node.js + WebAssembly).
 
 [![GitHub](https://img.shields.io/badge/GitHub-TOAQ--oss-181717?logo=github)](https://github.com/toaq-oss)
-
-
-This package consumes the ultra-fast Abstract Syntax Tree (AST) generated by our Rust core in WebAssembly (@toaq-oss/core-parser) and transforms it into interactive, secure, and highly customizable React components.
+[![Documentation](https://img.shields.io/badge/Docs-omni--core.org-blue)](https://omni-core.org/mdx)
 
 ---
 
-## Why this exists
-
-Most MDX pipelines (next-mdx-remote, @next/mdx, contentlayer) run at the JS level and require client-side hydration for math, syntax highlighting, and custom components. This means:
+## ⚡ Why Omni-MDX?
 
-- A large JS bundle sent to every visitor
-- Math rendered on the client (flash of unstyled content)
-- Custom components that can't be pure Server Components
+Traditional MDX pipelines are often slow or require complex client-side hydration. `omni-mdx` solves this by moving the heavy lifting to Rust, providing a seamless bridge between raw content and React components.
 
-`@toaq-oss/omni-mdx` solves this by moving the parse step into Rust and the render step into React Server Components. The result is **zero JS for content** — everything is HTML by the time it reaches the browser.
-
-The WASM build is available as a fallback for Edge runtimes or environments where native addons are not supported.
+- **Dual-Engine Architecture:** Uses native `.node` binaries on the server for maximum speed and **WASM** in the browser for instant live previews.
+- **Zero-Hydration Math:** LaTeX is pre-parsed by Rust and rendered via KaTeX with zero layout shift.
+- **RSC Optimized:** Built from the ground up for Next.js App Router and Server Components.
+- **Non-Blocking:** Offloads parsing from the Node.js main thread to Rust, keeping your server responsive even under heavy load.
 
 ---
 
-## Performance & Architecture
-
-`@toaq-oss/omni-mdx` is designed for scale. While traditional parsers block the Node.js main thread, our parser offloads the heavy lifting to a native Rust core via WebAssembly or N-API.
+## 📖 Documentation
 
-**1. Non-Blocking by Design**
-Parsing complex documents or extracting data from thousands of files for ML datasets won't freeze your server. Node.js remains completely free to handle other HTTP requests while Rust does the work in the background.
+For full guides, API references, and advanced configuration, visit:
+👉 **[omni-core.org/mdx](https://omni-core.org/mdx)**
 
-**2. Faster than the Standard**
-In a strict end-to-end benchmark (parsing raw text to an exploitable JSX AST) over 1,000 iterations of a complex document:
-- `@toaq-oss/omni-mdx` is consistently **~30% faster** than the official `@mdx-js/mdx` compiler.
-
-**3. Zero Client JS**
-Because the AST is generated on the server and maps directly to React Server Components, the client receives 0 bytes of parsing logic.
+---
 
-## Installation
+## 📦 Installation
 
 ```bash
 npm install @toaq-oss/omni-mdx
-# KaTeX is optional — only needed if your content has math
+# KaTeX is required for math rendering
 npm install katex
 ```
 
-Add the KaTeX stylesheet to your `layout.tsx`:
-
-```tsx
-import "katex/dist/katex.min.css";
+## Next.js Configuration
+To enable the WebAssembly engine for client-side rendering, update your `next.config.js`:
+```typescript
+/** @type {import('next').NextConfig} */
+const nextConfig = {
+  webpack: (config) => {
+    config.experiments = { ...config.experiments, asyncWebAssembly: true };
+    config.module.rules.push({
+      test: /\.wasm$/,
+      type: "asset/resource",
+    });
+    return config;
+  },
+};
+
+export default nextConfig;
 ```
 
 ---
 
-## Usage
+## 🚀 Usage
+### 1. Server-Side Rendering (RSC)
+Recommended for documentation, blogs, and research papers.
 
-### Server Component (recommended)
+> Full example available here :
+> * Basic setup: [TOAQ-oss/omni-core-sandox](https://github.com/TOAQ-oss/omni-mdx-sandbox/tree/main/next/basic-setup)
+> * Advanced rendering : [TOAQ-oss/omni-core-sandox](https://github.com/TOAQ-oss/omni-mdx-sandbox/tree/main/next/advanced-rendering)
 
 ```tsx
 import { parseMdx, MDXServerRenderer } from "@toaq-oss/omni-mdx/server";
-import { Note, Details }               from "@/components/mdx";
-
-const COMPONENTS = { Note, Details };
-
-export default async function ArticlePage({ params }) {
-  const content = await getArticleContent(params.slug);
-  const ast     = await parseMdx(content);
-
-  return <MDXServerRenderer ast={ast} components={COMPONENTS} />;
-}
-```
-
-That's it. No `getStaticProps`, no serialisation workarounds, no client bundle for the content.
-
-### Static generation
+import { MyComponent } from "@/components/mdx";
 
-Because `parseMdx` is async and runs on the server, it works naturally with `generateStaticParams`:
-
-```tsx
-export async function generateStaticParams() {
-  const slugs = await getAllSlugs();
-  return slugs.map(slug => ({ slug }));
-}
+export default async function Page({ content }) {
+  // Parsed via Native Rust Addon (.node)
+  const ast = await parseMdx(content);
 
-export default async function ArticlePage({ params }) {
-  const content = await getArticleContent(params.slug);
-  const ast     = await parseMdx(content);
-  return <MDXServerRenderer ast={ast} components={COMPONENTS} />;
+  return (
+    <MDXServerRenderer 
+      ast={ast} 
+      components={{ MyComponent }} 
+    />
+  );
 }
 ```
 
-At `next build`, Next.js calls `parseMdx` once per article and pre-renders everything to static HTML. The Rust parser never runs in the browser.
+###  2. Live Client Editor (WASM)
+Perfect for real-time previews or CMS interfaces.
 
-### Live editor (client-side)
-
-For live MDX previews where the content changes in the browser:
+> Full example available here : [TOAQ-oss/omni-core-sandox](https://github.com/TOAQ-oss/omni-mdx-sandbox/tree/main/next/client-rendering)
 
 ```tsx
 "use client";
-import { MDXClientRenderer } from "@toaq-oss/omni-mdx/client";
-
-export function LivePreview({ ast, components }) {
-  return <MDXClientRenderer ast={ast} components={components} katex />;
+import { useState } from "react";
+import { parseMdx, MDXClientRenderer } from "@toaq-oss/omni-mdx/client";
+
+export default function Editor() {
+  const [ast, setAst] = useState(null);
+
+  const handleChange = async (text) => {
+    // Runs 100% locally in the browser via WebAssembly
+    const result = await parseMdx(text);
+    setAst(result);
+  };
+
+  return (
+    <div>
+      <textarea onChange={(e) => handleChange(e.target.value)} />
+      <MDXClientRenderer ast={ast} />
+    </div>
+  );
 }
 ```
 
-The `ast` prop should be computed server-side and passed down, or computed client-side by calling a Route Handler that runs `parseMdx`.
-
----
-
-## Import map
-
-|Import path|What you get|Where to use|
-|----|---|----|
-|`@toaq-oss/omni-mdx`|Types + `MDX_COMPONENTS` registry|Anywhere|
-|`@toaq-oss/omni-mdx/server`|`parseMdx`, `MDXServerRenderer`, `MDXParseError` |Server Components only|
-|`@toaq-oss/omni-mdx/client`|`MDXClientRenderer`, `MDXErrorBoundary`|Client Components only|
-
 ---
-
-## Custom components
-
-Register components by name — the key matches the JSX tag in the MDX source:
-
-```tsx
-// MDX source:
-// <Note type="warning" title="Heads up">
-//   Be careful with **this**.
-// </Note>
-
-import { Note }    from "@/components/Note";    // can be a Server Component
-import { Details } from "@/components/Details"; // can be a Server Component
-import { Chart }   from "@/components/Chart";   // "use client" inside
-
-const COMPONENTS = { Note, Details, Chart };
-
-const ast = await parseMdx(content);
-return <MDXServerRenderer ast={ast} components={COMPONENTS} />;
-```
-
-Server Components in the registry are rendered on the server with zero client JS. Client Components are hydrated normally.
+|Environment|Backend|Entry Point|
+|:---|:---|:---|
+|**Server** (Node.js)|Native Addon (`.node`)|`@toaq-oss/omni-mdx/server`|
+|**Client** (Browser)|WebAssembly (`.wasm`)|`@toaq-oss/omni-mdx/client`|
+|**Edge** (Vercel)|Native Addon (`.wasm`)|`@toaq-oss/omni-mdx/client`|
 
 ---
 
-## Error handling
-
-### Parse errors
-
+## 🧩 Features
+### 📐 Professional Math
+Math is handled via KaTeX. Simply include the CSS in your `layout.tsx`:
 ```tsx
-import { parseMdx, MDXParseError } from "@toaq-oss/omni-mdx/server";
-
-try {
-  const ast = await parseMdx(content);
-} catch (e) {
-  if (e instanceof MDXParseError) {
-    console.error("MDX syntax error:", e.message);
-    console.error("Source:", e.source);
-  }
-}
+import "katex/dist/katex.min.css";
 ```
+* **Inline:** $E=mc^2$
+* **Block:** $$\int f(x)dx$$
 
-### Component render errors
-
-In `MDXClientRenderer`, every custom component is automatically wrapped in `MDXErrorBoundary`. If a component throws, the error is isolated — the rest of the document continues to render.
-
-You can also use `MDXErrorBoundary` directly:
-
+### 🎨 Custom Components
+Register any React component (Server or Client) to handle custom tags:
 ```tsx
-import { MDXErrorBoundary } from "@toaq-oss/omni-mdx/client";
-
-<MDXErrorBoundary componentName="Chart">
-  <Chart data={maybeNull} />
-</MDXErrorBoundary>
+const components = {
+  Callout: ({ children }) => <div className="p-4 bg-blue-50">{children}</div>,
+  VocalDataset: dynamic(() => import('./VocalDataset'), { ssr: false })
+};
 ```
 
 ---
-
-## Math
-
-Math is handled entirely by the Rust parser — no remark-math or rehype-katex needed.
-
-| Syntax | Output |
-|---|---|
-| `$E = mc^2$`  | `<span class="math math-inline" data-math="…">` |
-| `$$…$$`       | `<div class="math math-display" data-math="…">` |
-
-KaTeX hydrates the `data-math` attributes on the client via `MDXClientRenderer`, or you can use any KaTeX auto-render script in your layout.
-
----
-
-## Runtime support
-
-| Runtime | Native `.node` | WASM fallback |
-|-----------------|:--------------:|:-------------:|
-| Node.js 18+ | ✅ | ✅ |
-| Next.js (RSC) | ✅ | ✅ |
-| Edge runtime | ❌ | ✅ |
-| Browser | ❌ | ✅ |
-
-The package auto-detects the environment and loads the appropriate backend. No configuration required.
-
----
-
-## Building the native addon
-
-The `.node` files are pre-built for common platforms. To build for your platform:
-
-```bash
-cd core-parser
-napi build --platform --release --features node --no-js
-cp toaq-parser-core.*.node ../packages/mdx-next/native/
-```
-
-To build the WASM fallback:
-
-```bash
-cd core-parser
-wasm-pack build --target bundler --features wasm
-mv pkg/* ../packages/mdx-next/wasm/
-```
+## 🤝 Contributing
+This package is part of the TOAQ open-source ecosystem.
+* **Core Parser (Rust):** [TOAQ-oss/omni-mdx-core](https://github.com/TOAQ-oss/omni-mdx-core)
+* **Reporting Issues:** Please use the GitHub issue tracker for bugs or feature requests.

package/dist/client.cjs

@@ -38,7 +38,8 @@ __export(omni_mdx_core_exports, {
   initSync: () => initSync,
   parse_mdx_to_json: () => parse_mdx_to_json,
   parse_mdx_to_json_pretty: () => parse_mdx_to_json_pretty,
-  parse_mdx_version: () => parse_mdx_version
+  parse_mdx_version: () => parse_mdx_version,
+  parse_to_binary: () => parse_to_binary
 });
 function parse_mdx_to_json(input) {
   let deferred3_0;
@@ -94,6 +95,17 @@ function parse_mdx_version() {
     wasm.__wbindgen_free(deferred1_0, deferred1_1, 1);
   }
 }
+function parse_to_binary(mdx_input) {
+  const ptr0 = passArray8ToWasm0(mdx_input, wasm.__wbindgen_malloc);
+  const len0 = WASM_VECTOR_LEN;
+  const ret = wasm.parse_to_binary(ptr0, len0);
+  if (ret[3]) {
+    throw takeFromExternrefTable0(ret[2]);
+  }
+  var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+  wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+  return v2;
+}
 function __wbg_get_imports() {
   const import0 = {
     __proto__: null,
@@ -101,6 +113,10 @@ function __wbg_get_imports() {
       const ret = Error(getStringFromWasm0(arg0, arg1));
       return ret;
     },
+    __wbindgen_cast_0000000000000001: function(arg0, arg1) {
+      const ret = getStringFromWasm0(arg0, arg1);
+      return ret;
+    },
     __wbindgen_init_externref_table: function() {
       const table = wasm.__wbindgen_externrefs;
       const offset = table.grow(4);
@@ -116,6 +132,10 @@ function __wbg_get_imports() {
     "./omni_mdx_core_bg.js": import0
   };
 }
+function getArrayU8FromWasm0(ptr, len) {
+  ptr = ptr >>> 0;
+  return getUint8ArrayMemory0().subarray(ptr / 1, ptr / 1 + len);
+}
 function getStringFromWasm0(ptr, len) {
   ptr = ptr >>> 0;
   return decodeText(ptr, len);
@@ -126,6 +146,12 @@ function getUint8ArrayMemory0() {
   }
   return cachedUint8ArrayMemory0;
 }
+function passArray8ToWasm0(arg, malloc) {
+  const ptr = malloc(arg.length * 1, 1) >>> 0;
+  getUint8ArrayMemory0().set(arg, ptr / 1);
+  WASM_VECTOR_LEN = arg.length;
+  return ptr;
+}
 function passStringToWasm0(arg, malloc, realloc) {
   if (realloc === void 0) {
     const buf = cachedTextEncoder.encode(arg);
@@ -276,6 +302,7 @@ var client_exports = {};
 __export(client_exports, {
   MDXClientRenderer: () => MDXClientRenderer,
   MDXErrorBoundary: () => MDXErrorBoundary,
+  MdxBinaryDecoder: () => MdxBinaryDecoder,
   parseMdx: () => parseMdxClient
 });
 module.exports = __toCommonJS(client_exports);
@@ -472,6 +499,111 @@ function MDXClientRenderer({
   return /* @__PURE__ */ (0, import_jsx_runtime2.jsx)("div", { className: "omni-mdx-root", children: /* @__PURE__ */ (0, import_jsx_runtime2.jsx)(MDXClientContent, { ast, components }) });
 }
 
+// src/utils/binaryDecoder.ts
+var NODE_TEXT = 1;
+var NODE_ELEMENT = 2;
+var ATTR_TEXT = 16;
+var ATTR_EXPRESSION = 17;
+var ATTR_BOOLEAN = 18;
+var ATTR_AST = 19;
+var MdxBinaryDecoder = class {
+  constructor(buffer) {
+    this.offset = 0;
+    this.decoder = new TextDecoder("utf-8");
+    this.stringCache = /* @__PURE__ */ new Map();
+    this.buffer = buffer;
+    this.view = new DataView(buffer.buffer, buffer.byteOffset, buffer.byteLength);
+  }
+  decode() {
+    const rootCount = this.readU32();
+    const nodes = [];
+    for (let i = 0; i < rootCount; i++) {
+      nodes.push(this.decodeNode());
+    }
+    return nodes;
+  }
+  decodeNode() {
+    const type = this.readU8();
+    if (type === NODE_TEXT) {
+      return {
+        node_type: "text",
+        content: this.readStringU32()
+      };
+    }
+    if (type === NODE_ELEMENT) {
+      const node_type = this.readStringU16();
+      const self_closing = this.readU8() === 1;
+      const attrCount = this.readU16();
+      let attributes = void 0;
+      if (attrCount > 0) {
+        attributes = {};
+        for (let i = 0; i < attrCount; i++) {
+          const key = this.readStringU16();
+          const attrKind = this.readU8();
+          if (attrKind === ATTR_TEXT) {
+            attributes[key] = { kind: "text", value: this.readStringU32() };
+          } else if (attrKind === ATTR_EXPRESSION) {
+            attributes[key] = { kind: "expression", value: this.readStringU32() };
+          } else if (attrKind === ATTR_BOOLEAN) {
+            attributes[key] = { kind: "boolean" };
+          } else if (attrKind === ATTR_AST) {
+            const subNodeCount = this.readU32();
+            const subNodes = [];
+            for (let j = 0; j < subNodeCount; j++) {
+              subNodes.push(this.decodeNode());
+            }
+            attributes[key] = { kind: "ast", value: subNodes };
+          }
+        }
+      }
+      const childCount = this.readU32();
+      const children = [];
+      if (childCount > 0) {
+        for (let i = 0; i < childCount; i++) {
+          children.push(this.decodeNode());
+        }
+      }
+      return {
+        node_type,
+        self_closing,
+        attributes,
+        children
+      };
+    }
+    throw new Error(`[Omni-Core] Unknown binary opcode: ${type} at offset ${this.offset}`);
+  }
+  readU8() {
+    const val = this.view.getUint8(this.offset);
+    this.offset += 1;
+    return val;
+  }
+  readU16() {
+    const val = this.view.getUint16(this.offset, true);
+    this.offset += 2;
+    return val;
+  }
+  readU32() {
+    const val = this.view.getUint32(this.offset, true);
+    this.offset += 4;
+    return val;
+  }
+  readStringU16() {
+    const len = this.readU16();
+    const str = this.decoder.decode(this.buffer.subarray(this.offset, this.offset + len));
+    this.offset += len;
+    let cached = this.stringCache.get(str);
+    if (cached) return cached;
+    this.stringCache.set(str, str);
+    return str;
+  }
+  readStringU32() {
+    const len = this.readU32();
+    const str = this.decoder.decode(this.buffer.subarray(this.offset, this.offset + len));
+    this.offset += len;
+    return str;
+  }
+};
+
 // src/parse.client.ts
 var import_meta2 = {};
 var _parseClient = null;
@@ -482,15 +614,17 @@ async function getClientParser() {
     const wasmUrl = new URL("./omni_mdx_core_bg.wasm", import_meta2.url);
     await wasm2.default(wasmUrl);
   }
-  _parseClient = (mdx) => wasm2.parse_mdx_to_json(mdx);
+  _parseClient = (mdx) => wasm2.parse_to_binary(mdx);
   return _parseClient;
 }
 async function parseMdxClient(mdx) {
   if (typeof window === "undefined") return [];
   try {
     const parse = await getClientParser();
-    const json = parse(mdx);
-    return JSON.parse(json);
+    const inputBuffer = typeof mdx === "string" ? new TextEncoder().encode(mdx) : mdx;
+    const binaryAst = parse(inputBuffer);
+    const decoder = new MdxBinaryDecoder(binaryAst);
+    return decoder.decode();
   } catch (err) {
     console.error("[omni-mdx] WASM client parse error:", err);
     return [];