@kreuzberg/html-to-markdown-wasm 2.19.0-rc.1

package/dist/README.md

@@ -0,0 +1,202 @@
+# html-to-markdown
+
+<img width="1128" height="191" alt="Linkedin- Banner (1)" src="https://github.com/user-attachments/assets/f8e91036-20a5-40f9-9fcc-9e6c6e15f1f5" />
+
+
+
+High-performance HTML → Markdown conversion powered by Rust. Shipping as a Rust crate, Python package, PHP extension, Ruby gem, Elixir Rustler NIF, Node.js bindings, WebAssembly, and standalone CLI with identical rendering behaviour.
+
+Part of the Kreuzberg.dev document intelligence ecosystem. Kreuzberg is a polyglot document intelligence framework with a fast Rust core. We build tools that help developers extract, process, and understand documents at scale, from PDFs to Office files, images, archives, emails, in 50+ formats. We've set out to make high-performance document intelligence faster and more ecological.
+
+[![Crates.io](https://img.shields.io/crates/v/html-to-markdown-rs.svg?logo=rust&label=crates.io)](https://crates.io/crates/html-to-markdown-rs)
+[![npm (node)](https://img.shields.io/npm/v/%40kreuzberg%2Fhtml-to-markdown-node.svg?logo=npm)](https://www.npmjs.com/package/@kreuzberg/html-to-markdown-node)
+[![npm (wasm)](https://img.shields.io/npm/v/%40kreuzberg%2Fhtml-to-markdown-wasm.svg?logo=npm)](https://www.npmjs.com/package/@kreuzberg/html-to-markdown-wasm)
+[![PyPI](https://img.shields.io/pypi/v/html-to-markdown.svg?logo=pypi)](https://pypi.org/project/html-to-markdown/)
+[![Packagist](https://img.shields.io/packagist/v/goldziher/html-to-markdown.svg)](https://packagist.org/packages/goldziher/html-to-markdown)
+[![RubyGems](https://badge.fury.io/rb/html-to-markdown.svg)](https://rubygems.org/gems/html-to-markdown)
+[![Hex.pm](https://img.shields.io/hexpm/v/html_to_markdown.svg)](https://hex.pm/packages/html_to_markdown)
+[![NuGet](https://img.shields.io/nuget/v/KreuzbergDev.HtmlToMarkdown.svg)](https://www.nuget.org/packages/KreuzbergDev.HtmlToMarkdown/)
+[![Maven Central](https://img.shields.io/maven-central/v/dev.kreuzberg/html-to-markdown.svg)](https://central.sonatype.com/artifact/dev.kreuzberg/html-to-markdown)
+[![Go Reference](https://pkg.go.dev/badge/github.com/kreuzberg-dev/html-to-markdown/packages/go/v2/htmltomarkdown.svg)](https://pkg.go.dev/github.com/kreuzberg-dev/html-to-markdown/packages/go/v2/htmltomarkdown)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/kreuzberg-dev/html-to-markdown/blob/main/LICENSE)
+[![Discord](https://img.shields.io/badge/Discord-Join%20our%20community-7289da)](https://discord.gg/pXxagNK2zN)
+
+
+---
+
+## 🎮 **[Try the Live Demo →](https://kreuzberg-dev.github.io/html-to-markdown/)**
+
+Experience WebAssembly-powered HTML to Markdown conversion instantly in your browser. No installation needed!
+
+---
+
+## Why html-to-markdown?
+
+- **Blazing Fast**: Rust-powered core delivers 10-80× faster conversion than pure Python alternatives
+- **Universal**: Works everywhere - Node.js, Bun, Deno, browsers, Python, Rust, and standalone CLI
+- **Smart Conversion**: Handles complex documents including nested tables, code blocks, task lists, and hOCR OCR output
+- **Metadata Extraction**: Extract document metadata (title, description, headers, links, images) alongside conversion
+- **Highly Configurable**: Control heading styles, code block fences, list formatting, whitespace handling, and HTML sanitization
+- **Tag Preservation**: Keep specific HTML tags unconverted when markdown isn't expressive enough
+- **Secure by Default**: Built-in HTML sanitization prevents malicious content
+- **Consistent Output**: Identical markdown rendering across all language bindings
+
+## Quick Start
+
+**Node.js / Bun (Native - Fastest):**
+
+```typescript
+import { convert } from '@kreuzberg/html-to-markdown-node';
+
+const html = '<h1>Hello</h1><p>Rust ❤️ Markdown</p>';
+const markdown = convert(html, {
+  headingStyle: 'Atx',
+  codeBlockStyle: 'Backticks',
+  wrap: true,
+  preserveTags: ['table'],
+});
+```
+
+**Python:**
+
+```python
+from html_to_markdown import convert
+
+html = '<h1>Hello</h1><p>Rust ❤️ Markdown</p>'
+markdown = convert(html, heading_style='Atx', wrap=True)
+```
+
+**Ruby:**
+
+```ruby
+require 'html_to_markdown'
+
+html = '<h1>Hello</h1><p>Rust ❤️ Markdown</p>'
+markdown = HtmlToMarkdown.convert(html, heading_style: :atx, wrap: true)
+```
+
+Full language guides: See [Language Guides](#language-guides) below.
+
+## Installation
+
+| Target                      | Command(s)                                                                                                       |
+| --------------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| **Node.js/Bun** (native)    | `npm install @kreuzberg/html-to-markdown-node`                                                                   |
+| **WebAssembly** (universal) | `npm install @kreuzberg/html-to-markdown-wasm`                                                                   |
+| **Deno**                    | `import { convert } from "npm:@kreuzberg/html-to-markdown-wasm"`                                                 |
+| **Python** (bindings + CLI) | `pip install html-to-markdown`                                                                                   |
+| **PHP** (extension + helpers) | `PHP_EXTENSION_DIR=$(php-config --extension-dir) pie install goldziher/html-to-markdown`<br>`composer require goldziher/html-to-markdown` |
+| **Ruby** gem                | `bundle add html-to-markdown` or `gem install html-to-markdown`                                                  |
+| **Elixir** (Rustler NIF)    | `{:html_to_markdown, "~> 2.8"}`                                                                                  |
+| **Rust** crate              | `cargo add html-to-markdown-rs`                                                                                  |
+| **Java** (Maven)            | `<groupId>dev.kreuzberg</groupId><artifactId>html-to-markdown</artifactId>`                                     |
+| **C#/.NET** (NuGet)         | `dotnet add package KreuzbergDev.HtmlToMarkdown`                                                                 |
+| Rust CLI (crates.io)        | `cargo install html-to-markdown-cli`                                                                             |
+| Homebrew CLI                | `brew install html-to-markdown` (core)                                                                          |
+| Releases                    | [GitHub Releases](https://github.com/kreuzberg-dev/html-to-markdown/releases)                                    |
+
+## Performance
+
+Benchmarked on Apple M4 using the shared fixture harness in `tools/benchmark-harness`.
+
+### Comparative Throughput (Median Across Fixtures)
+
+| Runtime | Median ops/sec | Median throughput (MB/s) | Peak memory (MB) | Successes |
+| ------- | -------------- | ------------------------ | ---------------- | --------- |
+| Rust | 1,060.3 | 116.4 | 171.3 | 56/56 |
+| Go | 1,496.3 | 131.1 | 22.9 | 16/16 |
+| Ruby | 2,155.5 | 300.4 | 280.3 | 48/48 |
+| PHP | 2,357.7 | 308.0 | 223.5 | 48/48 |
+| Elixir | 1,564.1 | 269.1 | 384.7 | 48/48 |
+| C# | 1,234.2 | 272.4 | 187.8 | 16/16 |
+| Java | 1,298.7 | 167.1 | 527.2 | 16/16 |
+| WASM | 1,485.8 | 157.6 | 95.3 | 48/48 |
+| Node.js (NAPI) | 2,054.2 | 306.5 | 95.4 | 48/48 |
+| Python (PyO3) | 3,120.3 | 307.5 | 83.5 | 48/48 |
+
+Use `task bench:harness` to regenerate throughput numbers. See [Performance Guide](./examples/performance/) for benchmarking strategies and optimization tips.
+
+## Language Guides
+
+Complete documentation with examples for each language:
+
+- **Python** – [README](./packages/python/README.md) | PyO3 bindings, metadata extraction, inline images
+- **JavaScript/TypeScript** – [Node.js](./crates/html-to-markdown-node/README.md) | [TypeScript](./packages/typescript/README.md) | [WASM](./crates/html-to-markdown-wasm/README.md)
+- **Ruby** – [README](./packages/ruby/README.md) | Magnus bindings, RBS type definitions, Steep checking
+- **PHP** – [Package](./packages/php/README.md) | [Extension (PIE)](./packages/php-ext/README.md) | ext-php-rs extension
+- **Go** – [README](./packages/go/README.md) | FFI bindings with cgo
+- **Java** – [README](./packages/java/README.md) | Panama FFI, Maven/Gradle setup
+- **C#/.NET** – [README](./packages/csharp/README.md) | P/Invoke FFI, NuGet distribution
+- **Elixir** – [README](./packages/elixir/README.md) | Rustler NIF bindings
+- **Rust** – [README](./crates/html-to-markdown/README.md) | Core library, error handling, advanced features
+
+## Feature Guides
+
+### Visitor Pattern
+Customize HTML→Markdown conversion with callbacks for specific elements. Use cases: domain-specific dialects, content filtering, URL rewriting, accessibility validation.
+
+**→ [Full Guide with Examples](./examples/visitor-pattern/)** (Python, TypeScript, Ruby)
+
+### Metadata Extraction
+Extract comprehensive metadata during conversion: title, description, headers, links, images, structured data. Use cases: SEO extraction, TOC generation, link validation, accessibility auditing, content migration.
+
+**→ [Full Guide with Examples](./examples/metadata-extraction/)** (Python, TypeScript, Ruby)
+
+### Performance & Benchmarking
+Understand performance characteristics, run benchmarks, optimize for your use case. Includes benchmarking tools, memory profiling, streaming strategies, and optimization tips.
+
+**→ [Full Guide](./examples/performance/)**
+
+## Examples
+
+Explore working code examples in multiple languages:
+
+| Example | Path | Languages |
+| ------- | ---- | --------- |
+| **Visitor Pattern** | [examples/visitor-pattern/](./examples/visitor-pattern/) | Python, TypeScript, Ruby |
+| **Metadata Extraction** | [examples/metadata-extraction/](./examples/metadata-extraction/) | Python, TypeScript, Ruby |
+| **Performance** | [examples/performance/](./examples/performance/) | Benchmarks, profiling, optimization |
+
+## Testing
+
+Run the test suite locally:
+
+```bash
+# All core test suites (Rust, Python, Ruby, Node, PHP, Go, C#, Elixir, Java)
+task test
+
+# Run the Wasmtime-backed WASM integration tests
+task wasm:test:wasmtime
+```
+
+## Compatibility & Migrations
+
+### v2.19.0 Breaking Changes (Package Namespace Updates)
+
+Several language bindings were updated to use new namespaces and package owners:
+
+- **npm packages**: Scoped under `@kreuzberg` organization
+  - Old: `html-to-markdown-node` → New: `@kreuzberg/html-to-markdown-node`
+  - Old: `html-to-markdown-wasm` → New: `@kreuzberg/html-to-markdown-wasm`
+- **Java**: Package namespace changed from `io.github.goldziher` to `dev.kreuzberg`
+- **C#/.NET**: Package changed from `Goldziher.HtmlToMarkdown` to `KreuzbergDev.HtmlToMarkdown`
+
+See [MIGRATION.md](./MIGRATION.md) for step-by-step upgrade instructions for each language.
+
+### v1 → v2 Compatibility
+
+- V2's Rust core sustains **150–210 MB/s** throughput; V1 averaged **≈ 2.5 MB/s** (60–80× faster).
+- Python compatibility shim available in `html_to_markdown.v1_compat` (deprecated; emits warnings; plan migrations now). See [Python README](./packages/python/README.md#v1-compatibility) for keyword mappings.
+- CLI flag changes and other breaking updates in [CHANGELOG](./CHANGELOG.md#breaking-changes).
+
+## Community
+
+- **Discord** – [Join our community](https://discord.gg/pXxagNK2zN)
+- **Ecosystem** – Explore [Kreuzberg](https://kreuzberg.dev) document-processing tools
+- **Contribute** – [CONTRIBUTING.md](./CONTRIBUTING.md)
+- **Sponsor** – [GitHub Sponsors](https://github.com/sponsors/kreuzberg-dev)
+- **Changelog** – [Version history](./CHANGELOG.md)
+
+## License
+
+MIT License – see [LICENSE](./LICENSE) for details.

package/dist/html_to_markdown_wasm.d.ts

@@ -0,0 +1,200 @@
+/* tslint:disable */
+/* eslint-disable */
+
+export class WasmConversionOptionsHandle {
+  free(): void;
+  [Symbol.dispose](): void;
+  constructor(options?: WasmConversionOptions | null);
+}
+
+export class WasmHtmlExtraction {
+  private constructor();
+  free(): void;
+  [Symbol.dispose](): void;
+  readonly inlineImages: WasmInlineImage[];
+  readonly markdown: string;
+  readonly warnings: WasmInlineImageWarning[];
+}
+
+export class WasmInlineImage {
+  private constructor();
+  free(): void;
+  [Symbol.dispose](): void;
+  readonly attributes: Record<string, string>;
+  readonly dimensions: Uint32Array | undefined;
+  readonly description: string | undefined;
+  readonly data: Uint8Array;
+  readonly format: string;
+  readonly source: string;
+  readonly filename: string | undefined;
+}
+
+export class WasmInlineImageConfig {
+  free(): void;
+  [Symbol.dispose](): void;
+  constructor(max_decoded_size_bytes?: number | null);
+  set captureSvg(value: boolean);
+  set filenamePrefix(value: string | null | undefined);
+  set inferDimensions(value: boolean);
+}
+
+export class WasmInlineImageWarning {
+  private constructor();
+  free(): void;
+  [Symbol.dispose](): void;
+  readonly index: number;
+  readonly message: string;
+}
+
+export class WasmMetadataConfig {
+  free(): void;
+  [Symbol.dispose](): void;
+  /**
+   * Create a new metadata configuration with defaults
+   *
+   * All extraction types enabled by default with 1MB structured data limit
+   */
+  constructor();
+  extract_links: boolean;
+  extract_images: boolean;
+  extract_headers: boolean;
+  extract_document: boolean;
+  extract_structured_data: boolean;
+  max_structured_data_size: number;
+}
+
+/**
+ * Convert HTML to Markdown
+ *
+ * # Arguments
+ *
+ * * `html` - The HTML string to convert
+ * * `options` - Optional conversion options (as a JavaScript object)
+ *
+ * # Example
+ *
+ * ```javascript
+ * import { convert } from 'html-to-markdown-wasm';
+ *
+ * const html = '<h1>Hello World</h1>';
+ * const markdown = convert(html);
+ * console.log(markdown); // # Hello World
+ * ```
+ */
+export function convert(html: string, options?: WasmConversionOptions | null): string;
+
+export function convertBytes(html: Uint8Array, options?: WasmConversionOptions | null): string;
+
+export function convertBytesWithInlineImages(html: Uint8Array, options?: WasmConversionOptions | null, image_config?: WasmInlineImageConfig | null): WasmHtmlExtraction;
+
+/**
+ * Convert HTML bytes to Markdown with metadata extraction
+ *
+ * # Arguments
+ *
+ * * `html` - The HTML bytes to convert
+ * * `options` - Optional conversion options (as a JavaScript object)
+ * * `metadata_config` - Metadata extraction configuration
+ *
+ * # Returns
+ *
+ * JavaScript object with `markdown` (string) and `metadata` (object) fields
+ */
+export function convertBytesWithMetadata(html: Uint8Array, options?: WasmConversionOptions | null, metadata_config?: WasmMetadataConfig | null): any;
+
+export function convertBytesWithOptionsHandle(html: Uint8Array, handle: WasmConversionOptionsHandle): string;
+
+export function convertWithInlineImages(html: string, options?: WasmConversionOptions | null, image_config?: WasmInlineImageConfig | null): WasmHtmlExtraction;
+
+/**
+ * Convert HTML to Markdown with metadata extraction
+ *
+ * # Arguments
+ *
+ * * `html` - The HTML string to convert
+ * * `options` - Optional conversion options (as a JavaScript object)
+ * * `metadata_config` - Metadata extraction configuration
+ *
+ * # Returns
+ *
+ * JavaScript object with `markdown` (string) and `metadata` (object) fields
+ *
+ * # Example
+ *
+ * ```javascript
+ * import { convertWithMetadata, WasmMetadataConfig } from 'html-to-markdown-wasm';
+ *
+ * const html = '<h1>Hello World</h1><a href="https://example.com">Link</a>';
+ * const config = new WasmMetadataConfig();
+ * config.extractHeaders = true;
+ * config.extractLinks = true;
+ *
+ * const result = convertWithMetadata(html, null, config);
+ * console.log(result.markdown); // # Hello World\n\n[Link](https://example.com)
+ * console.log(result.metadata.headers); // [{ level: 1, text: "Hello World", ... }]
+ * console.log(result.metadata.links); // [{ href: "https://example.com", text: "Link", ... }]
+ * ```
+ */
+export function convertWithMetadata(html: string, options?: WasmConversionOptions | null, metadata_config?: WasmMetadataConfig | null): any;
+
+export function convertWithOptionsHandle(html: string, handle: WasmConversionOptionsHandle): string;
+
+export function createConversionOptionsHandle(options?: WasmConversionOptions | null): WasmConversionOptionsHandle;
+
+/**
+ * Initialize panic hook for better error messages in the browser
+ */
+export function init(): void;
+
+export declare function initWasm(): Promise<void>;
+export declare const wasmReady: Promise<void>;
+
+
+export type WasmHeadingStyle = "underlined" | "atx" | "atxClosed";
+export type WasmListIndentType = "spaces" | "tabs";
+export type WasmWhitespaceMode = "normalized" | "strict";
+export type WasmNewlineStyle = "spaces" | "backslash";
+export type WasmCodeBlockStyle = "indented" | "backticks" | "tildes";
+export type WasmHighlightStyle = "doubleEqual" | "html" | "bold" | "none";
+export type WasmPreprocessingPreset = "minimal" | "standard" | "aggressive";
+
+export interface WasmPreprocessingOptions {
+  enabled?: boolean;
+  preset?: WasmPreprocessingPreset;
+  removeNavigation?: boolean;
+  removeForms?: boolean;
+}
+
+export interface WasmConversionOptions {
+  headingStyle?: WasmHeadingStyle;
+  listIndentType?: WasmListIndentType;
+  listIndentWidth?: number;
+  bullets?: string;
+  strongEmSymbol?: string;
+  escapeAsterisks?: boolean;
+  escapeUnderscores?: boolean;
+  escapeMisc?: boolean;
+  escapeAscii?: boolean;
+  codeLanguage?: string;
+  autolinks?: boolean;
+  defaultTitle?: boolean;
+  brInTables?: boolean;
+  hocrSpatialTables?: boolean;
+  highlightStyle?: WasmHighlightStyle;
+  extractMetadata?: boolean;
+  whitespaceMode?: WasmWhitespaceMode;
+  stripNewlines?: boolean;
+  wrap?: boolean;
+  wrapWidth?: number;
+  convertAsInline?: boolean;
+  subSymbol?: string;
+  supSymbol?: string;
+  newlineStyle?: WasmNewlineStyle;
+  codeBlockStyle?: WasmCodeBlockStyle;
+  keepInlineImagesIn?: string[];
+  preprocessing?: WasmPreprocessingOptions | null;
+  encoding?: string;
+  debug?: boolean;
+  stripTags?: string[];
+  preserveTags?: string[];
+}

package/dist/html_to_markdown_wasm.js

@@ -0,0 +1,116 @@
+import * as wasmModule from "./html_to_markdown_wasm_bg.wasm";
+export * from "./html_to_markdown_wasm_bg.js";
+import * as imports_mod from "./html_to_markdown_wasm_bg.js";
+
+const notReadyError = () =>
+  new Error("html-to-markdown-wasm: WebAssembly bundle is still initializing. Await initWasm() before calling convert() in runtimes that load WASM asynchronously (e.g., Cloudflare Workers).");
+
+const notReadyProxy = new Proxy({}, {
+  get(_target, prop) {
+    if (prop === "__esModule") {
+      return true;
+    }
+    throw notReadyError();
+  }
+});
+
+let wasmExports;
+let initialized = false;
+let initPromise;
+
+imports_mod.__wbg_set_wasm(notReadyProxy);
+
+function asExports(value) {
+  if (!value) {
+    return null;
+  }
+  if (typeof value.__wbindgen_start === "function") {
+    return value;
+  }
+  if (value instanceof WebAssembly.Instance) {
+    return value.exports;
+  }
+  if (typeof value === "object") {
+    if (value.instance instanceof WebAssembly.Instance) {
+      return value.instance.exports;
+    }
+    if (value.default instanceof WebAssembly.Instance) {
+      return value.default.exports;
+    }
+    if (value.default && value.default.instance instanceof WebAssembly.Instance) {
+      return value.default.instance.exports;
+    }
+  }
+  return null;
+}
+
+function finalize(exports) {
+  wasmExports = exports;
+  imports_mod.__wbg_set_wasm(exports);
+  if (typeof exports.__wbindgen_start === "function") {
+    exports.__wbindgen_start();
+  }
+  initialized = true;
+  return exports;
+}
+
+function trySyncInit() {
+  try {
+    const exports = asExports(wasmModule);
+    if (exports) {
+      finalize(exports);
+    }
+  } catch {
+    // ignore and fall back to async init
+  }
+}
+
+trySyncInit();
+
+async function ensureInitPromise() {
+  if (initialized) {
+    return Promise.resolve(wasmExports);
+  }
+  if (!initPromise) {
+    initPromise = (async () => {
+      let module = wasmModule;
+
+      // Handle promise-wrapped modules
+      if (module && typeof module.then === "function") {
+        module = await module;
+      }
+
+      // Handle function loaders (like @rollup/plugin-wasm)
+      if (module && typeof module.default === "function") {
+        module = await module.default(module);
+      }
+
+      // Handle WebAssembly.Module (Wrangler/esbuild)
+      if (module && module.default instanceof WebAssembly.Module) {
+        const imports = {};
+        imports["./html_to_markdown_wasm_bg.js"] = {};
+        for (const key in imports_mod) {
+          if ((key.startsWith('__wbg_') || key.startsWith('__wbindgen_')) && key !== '__wbg_set_wasm' && typeof imports_mod[key] === 'function') {
+            imports["./html_to_markdown_wasm_bg.js"][key] = imports_mod[key];
+          }
+        }
+        const instance = await WebAssembly.instantiate(module.default, imports);
+        return finalize(instance.exports);
+      }
+
+      // Try standard export detection
+      const exports = asExports(module);
+      if (!exports) {
+        throw new Error("html-to-markdown-wasm: failed to initialize WebAssembly bundle. Call initWasm() with a supported bundler configuration.");
+      }
+      return finalize(exports);
+    })();
+  }
+  return initPromise;
+}
+
+export const wasmReady = ensureInitPromise();
+
+export async function initWasm() {
+  return ensureInitPromise();
+}