RubyGems - html-to-markdown - Versions diffs - 3.4.0 → 3.6.0.pre.rc.23 - Mend

html-to-markdown 3.4.0 → 3.6.0.pre.rc.23

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (188) hide show

data/vendor/html-to-markdown-rs/README.md DELETED Viewed

@@ -1,278 +0,0 @@
-# html-to-markdown-rs
-High-performance HTML to Markdown converter built with Rust.
-This crate is the core engine compiled into the Python wheels, Ruby gem, Node.js NAPI bindings, WebAssembly package, and CLI, ensuring identical Markdown output across every language.
-[![Crates.io](https://img.shields.io/crates/v/html-to-markdown-rs.svg)](https://crates.io/crates/html-to-markdown-rs)
-[![npm version](https://img.shields.io/npm/v/@kreuzberg/html-to-markdown.svg?logo=npm)](https://www.npmjs.com/package/@kreuzberg/html-to-markdown)
-[![PyPI version](https://img.shields.io/pypi/v/html-to-markdown.svg?logo=pypi)](https://pypi.org/project/html-to-markdown/)
-[![Gem Version](https://badge.fury.io/rb/html-to-markdown.svg)](https://rubygems.org/gems/html-to-markdown)
-[![Packagist](https://img.shields.io/packagist/v/kreuzberg-dev/html-to-markdown.svg)](https://packagist.org/packages/kreuzberg-dev/html-to-markdown)
-[![docs.rs](https://docs.rs/html-to-markdown-rs/badge.svg)](https://docs.rs/html-to-markdown-rs)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/kreuzberg-dev/html-to-markdown/blob/main/LICENSE)
-Fast, reliable HTML to Markdown conversion with full CommonMark compliance. Built with `html5ever` for correctness and a DOM-based filter for safe preprocessing.
-## Installation
-```toml
-[dependencies]
-html-to-markdown-rs = "3.0"
-```
-## Basic Usage
-`convert()` returns a structured `ConversionResult` with the converted text, metadata, tables, and more:
-```rust
-use html_to_markdown_rs::convert;
-fn main() -> Result<(), Box<dyn std::error::Error>> {
-    let html = r#"
-        <html lang="en">
-          <head><title>Welcome</title></head>
-          <body>
-            <h1>Welcome</h1>
-            <p>This is <strong>fast</strong> conversion!</p>
-            <ul>
-                <li>Built with Rust</li>
-                <li>CommonMark compliant</li>
-            </ul>
-          </body>
-        </html>
-    "#;
-    let result = convert(html, None)?;
-    println!("{}", result.content.unwrap_or_default());
-    if let Some(metadata) = &result.metadata {
-        println!("Title: {:?}", metadata.document.title);
-        println!("Headers: {:?}", metadata.headers);
-    }
-    for table in &result.tables {
-        println!("Table with {} rows", table.cells.len());
-    }
-    Ok(())
-}
-```
-## Error Handling
-Conversion returns a `Result<ConversionResult, ConversionError>`. Inputs that look like binary data are rejected with
-`ConversionError::InvalidInput` to prevent runaway allocations. Table `colspan`/`rowspan` values are also clamped
-internally to keep output sizes bounded.
-## Configuration
-### Builder Pattern
-```rust
-use html_to_markdown_rs::{
-    convert, ConversionOptions, HeadingStyle, CodeBlockStyle,
-};
-let options = ConversionOptions::builder()
-    .heading_style(HeadingStyle::Atx)
-    .list_indent_width(2)
-    .bullets("-")
-    .autolinks(true)
-    .wrap(true)
-    .wrap_width(80)
-    .build();
-let result = convert(html, Some(options))?;
-println!("{}", result.content.unwrap_or_default());
-```
-### Struct Literal
-```rust
-use html_to_markdown_rs::{
-    convert, ConversionOptions, HeadingStyle, ListIndentType,
-};
-let options = ConversionOptions {
-    heading_style: HeadingStyle::Atx,
-    list_indent_width: 2,
-    list_indent_type: ListIndentType::Spaces,
-    bullets: "-".to_string(),
-    strong_em_symbol: '*',
-    escape_asterisks: false,
-    escape_underscores: false,
-    newline_style: html_to_markdown_rs::NewlineStyle::Backslash,
-    code_block_style: html_to_markdown_rs::CodeBlockStyle::Backticks,
-    ..Default::default()
-};
-let result = convert(html, Some(options))?;
-println!("{}", result.content.unwrap_or_default());
-```
-### Preserving HTML Tags
-The `preserve_tags` option allows you to keep specific HTML tags in their original form instead of converting them to Markdown:
-```rust
-use html_to_markdown_rs::{convert, ConversionOptions};
-let html = r#"
-<p>Before table</p>
-<table class="data">
-    <tr><th>Name</th><th>Value</th></tr>
-    <tr><td>Item 1</td><td>100</td></tr>
-</table>
-<p>After table</p>
-"#;
-let options = ConversionOptions {
-    preserve_tags: vec!["table".to_string()],
-    ..Default::default()
-};
-let result = convert(html, Some(options))?;
-// result.content => "Before table\n\n<table class=\"data\">...</table>\n\nAfter table\n"
-```
-## Web Scraping with Preprocessing
-```rust
-use html_to_markdown_rs::{convert, ConversionOptions, PreprocessingOptions};
-let mut options = ConversionOptions::default();
-options.preprocessing.enabled = true;
-options.preprocessing.preset = html_to_markdown_rs::PreprocessingPreset::Aggressive;
-options.preprocessing.remove_navigation = true;
-options.preprocessing.remove_forms = true;
-let result = convert(scraped_html, Some(options))?;
-println!("{}", result.content.unwrap_or_default());
-```
-## Metadata Extraction
-Metadata is automatically included in the result. Configure which fields to extract via `MetadataConfig`:
-```rust
-use html_to_markdown_rs::{convert, ConversionOptions, MetadataConfig};
-let options = ConversionOptions::builder()
-    .metadata_config(MetadataConfig {
-        extract_headers: true,
-        extract_links: true,
-        extract_images: false,
-        ..Default::default()
-    })
-    .build();
-let result = convert(html, Some(options))?;
-if let Some(metadata) = &result.metadata {
-    println!("Title: {:?}", metadata.document.title);
-    for header in &metadata.headers {
-        println!("H{}: {}", header.level, header.text);
-    }
-    for link in &metadata.links {
-        println!("Link: {} -> {}", link.text, link.href);
-    }
-}
-```
-## Image Extraction
-```rust
-use html_to_markdown_rs::{convert, ConversionOptions};
-let options = ConversionOptions::builder()
-    .extract_images(true)
-    .max_image_size(5 * 1024 * 1024) // 5 MB max
-    .infer_dimensions(true)
-    .build();
-let result = convert(html, Some(options))?;
-println!("{}", result.content.unwrap_or_default());
-for img in &result.images {
-    println!("Image: {} ({} bytes)", img.src, img.data.as_ref().map_or(0, |d| d.len()));
-}
-```
-## Table Extraction
-Structured table data is always included in `ConversionResult.tables`:
-```rust
-use html_to_markdown_rs::convert;
-let html = r#"
-<table>
-    <tr><th>Name</th><th>Age</th></tr>
-    <tr><td>Alice</td><td>30</td></tr>
-    <tr><td>Bob</td><td>25</td></tr>
-</table>
-"#;
-let result = convert(html, None)?;
-println!("{}", result.content.unwrap_or_default());
-for table in &result.tables {
-    println!("Table with {} rows:", table.cells.len());
-    for (i, row) in table.cells.iter().enumerate() {
-        let prefix = if table.is_header_row[i] { "Header" } else { "Row" };
-        println!("  {}: {:?}", prefix, row);
-    }
-}
-```
-## Custom Visitors
-```rust
-use html_to_markdown_rs::{convert, ConversionOptions};
-use html_to_markdown_rs::visitor::{HtmlVisitor, NodeContext, VisitResult};
-struct NoImagesVisitor;
-impl HtmlVisitor for NoImagesVisitor {
-    fn visit_image(
-        &mut self,
-        _ctx: &NodeContext,
-        _src: &str,
-        _alt: &str,
-        _title: Option<&str>,
-    ) -> VisitResult {
-        VisitResult::Skip
-    }
-}
-let options = ConversionOptions::builder()
-    .visitor(Box::new(NoImagesVisitor))
-    .build();
-let result = convert(html, Some(options))?;
-println!("{}", result.content.unwrap_or_default());
-```
-## Other Language Bindings
-This is the core Rust library. For other languages:
-- **JavaScript/TypeScript**: [html-to-markdown-node](https://github.com/kreuzberg-dev/html-to-markdown/tree/main/crates/html-to-markdown-node) (NAPI-RS) or [html-to-markdown-wasm](https://github.com/kreuzberg-dev/html-to-markdown/tree/main/crates/html-to-markdown-wasm) (WebAssembly)
-- **Python**: [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown/tree/main/crates/html-to-markdown-py) (PyO3)
-- **PHP**: [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown/tree/main/packages/php) (PIE + Composer helpers)
-- **Ruby**: [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown/tree/main/packages/ruby) (Magnus + rb-sys)
-- **CLI**: [html-to-markdown-cli](https://github.com/kreuzberg-dev/html-to-markdown/tree/main/crates/html-to-markdown-cli)
-## Documentation
-- [Full Documentation](https://docs.html-to-markdown.kreuzberg.dev)
-- [API Reference](https://docs.rs/html-to-markdown-rs)
-- [Migration Guide (v2 -> v3)](https://docs.html-to-markdown.kreuzberg.dev/migration/v3/)
-- [Contributing Guide](https://github.com/kreuzberg-dev/html-to-markdown/blob/main/CONTRIBUTING.md)
-## Performance
-10-30x faster than pure Python/JavaScript implementations, delivering 150-280 MB/s throughput.
-## License
-MIT

data/vendor/html-to-markdown-rs/examples/basic.rs DELETED Viewed

@@ -1,24 +0,0 @@
-//! Example: Basic HTML to Markdown conversion
-fn convert(
-    html: &str,
-    opts: Option<html_to_markdown_rs::ConversionOptions>,
-) -> html_to_markdown_rs::error::Result<String> {
-    html_to_markdown_rs::convert(html, opts).map(|r| r.content.unwrap_or_default())
-}
-fn main() {
-    let html = "<h1>Hello World</h1><p>This is a <strong>test</strong>.</p>";
-    match convert(html, None) {
-        Ok(markdown) => {
-            println!("HTML:");
-            println!("{html}");
-            println!("\nMarkdown:");
-            println!("{markdown}");
-        }
-        Err(e) => {
-            eprintln!("Error: {e}");
-        }
-    }
-}

data/vendor/html-to-markdown-rs/examples/table.rs DELETED Viewed

@@ -1,25 +0,0 @@
-//! Example: Converting HTML tables to Markdown
-fn convert(
-    html: &str,
-    opts: Option<html_to_markdown_rs::ConversionOptions>,
-) -> html_to_markdown_rs::error::Result<String> {
-    html_to_markdown_rs::convert(html, opts).map(|r| r.content.unwrap_or_default())
-}
-fn main() {
-    let html = r"<table>
-        <tr><th>Name</th><th>Age</th></tr>
-        <tr><td>Alice</td><td>30</td></tr>
-        <tr><td>Bob</td><td>25</td></tr>
-    </table>";
-    match convert(html, None) {
-        Ok(markdown) => {
-            println!("Markdown:\n{markdown}");
-        }
-        Err(e) => {
-            eprintln!("Error: {e}");
-        }
-    }
-}

data/vendor/html-to-markdown-rs/examples/test_deser.rs DELETED Viewed

@@ -1,12 +0,0 @@
-#![allow(missing_docs)]
-use html_to_markdown_rs::ConversionOptions;
-fn main() {
-    let json = r#"{"headingStyle":"","listIndentType":"","listIndentWidth":2,"bullets":"-*+","strongEmSymbol":"*","escapeAsterisks":false,"escapeUnderscores":false,"escapeMisc":false,"escapeAscii":false,"codeLanguage":"","autolinks":true,"defaultTitle":false,"brInTables":false,"highlightStyle":"","extractMetadata":true,"whitespaceMode":"","stripNewlines":false,"wrap":false,"wrapWidth":80,"convertAsInline":false,"subSymbol":"","supSymbol":"","newlineStyle":"spaces","codeBlockStyle":"tildes","keepInlineImagesIn":null,"preprocessing":{"enabled":false,"preset":"","removeNavigation":false,"removeForms":false},"encoding":"utf-8","debug":false,"stripTags":null,"preserveTags":null,"skipImages":false,"linkStyle":"","outputFormat":"","includeDocumentStructure":false,"extractImages":false,"maxImageSize":5242880,"captureSvg":false,"inferDimensions":true}"#;
-    let opts: ConversionOptions = serde_json::from_str(json).unwrap();
-    println!("code_block_style: {:?}", opts.code_block_style);
-    let result = html_to_markdown_rs::convert("<pre><code>some code</code></pre>", Some(opts)).unwrap();
-    println!("result: {:?}", result.content);
-}

data/vendor/html-to-markdown-rs/examples/test_escape.rs DELETED Viewed

@@ -1,58 +0,0 @@
-//! Example: Testing HTML escape sequences and special characters
-fn convert(
-    html: &str,
-    opts: Option<html_to_markdown_rs::ConversionOptions>,
-) -> html_to_markdown_rs::error::Result<String> {
-    html_to_markdown_rs::convert(html, opts).map(|r| r.content.unwrap_or_default())
-}
-fn main() {
-    let html = "<p>Use *wildcards* for search</p>";
-    match convert(html, None) {
-        Ok(markdown) => {
-            println!("Test 1 - Asterisks:");
-            println!("HTML: {html}");
-            println!("Markdown: {markdown}");
-            println!("Expected: Use \\*wildcards\\* for search");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html2 = "<p>variable_name in code</p>";
-    match convert(html2, None) {
-        Ok(markdown) => {
-            println!("Test 2 - Underscores:");
-            println!("HTML: {html2}");
-            println!("Markdown: {markdown}");
-            println!("Expected: variable\\_name in code");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html3 = "<code>use *wildcards* for search</code>";
-    match convert(html3, None) {
-        Ok(markdown) => {
-            println!("Test 3 - Code (no escaping):");
-            println!("HTML: {html3}");
-            println!("Markdown: {markdown}");
-            println!("Expected: `use *wildcards* for search`");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html4 = "<p>List: 1. First item</p>";
-    match convert(html4, None) {
-        Ok(markdown) => {
-            println!("Test 4 - Numbered list escape:");
-            println!("HTML: {html4}");
-            println!("Markdown: {markdown}");
-            println!("Expected: List: 1\\. First item");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-}

data/vendor/html-to-markdown-rs/examples/test_inline_formatting.rs DELETED Viewed

@@ -1,113 +0,0 @@
-#![allow(missing_docs)]
-fn convert(
-    html: &str,
-    opts: Option<html_to_markdown_rs::ConversionOptions>,
-) -> html_to_markdown_rs::error::Result<String> {
-    html_to_markdown_rs::convert(html, opts).map(|r| r.content.unwrap_or_default())
-}
-use html_to_markdown_rs::ConversionOptions;
-fn main() {
-    let html = "<p>This is <mark>highlighted</mark> text</p>";
-    match convert(html, None) {
-        Ok(markdown) => {
-            println!("Test - Mark (default):");
-            println!("HTML: {html}");
-            println!("Markdown: {markdown}");
-            println!("Expected: This is ==highlighted== text");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html2 = "<p>This is <del>deleted</del> and <s>strikethrough</s> text</p>";
-    match convert(html2, None) {
-        Ok(markdown) => {
-            println!("Test - Del/Strike:");
-            println!("HTML: {html2}");
-            println!("Markdown: {markdown}");
-            println!("Expected: This is ~~deleted~~ and ~~strikethrough~~ text");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html3 = "<p>This is <ins>inserted</ins> text</p>";
-    match convert(html3, None) {
-        Ok(markdown) => {
-            println!("Test - Ins:");
-            println!("HTML: {html3}");
-            println!("Markdown: {markdown}");
-            println!("Expected: This is ==inserted== text");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html4 = "<p>Press <kbd>Ctrl+C</kbd> and see <samp>output</samp></p>";
-    match convert(html4, None) {
-        Ok(markdown) => {
-            println!("Test - Kbd/Samp:");
-            println!("HTML: {html4}");
-            println!("Markdown: {markdown}");
-            println!("Expected: Press `Ctrl+C` and see `output`");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html5 = "<p>The variable <var>x</var> is defined</p>";
-    match convert(html5, None) {
-        Ok(markdown) => {
-            println!("Test - Var:");
-            println!("HTML: {html5}");
-            println!("Markdown: {markdown}");
-            println!("Expected: The variable *x* is defined");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let opts = ConversionOptions {
-        sub_symbol: "~".to_string(),
-        sup_symbol: "^".to_string(),
-        ..Default::default()
-    };
-    let html6 = "<p>H<sub>2</sub>O and x<sup>2</sup></p>";
-    match convert(html6, Some(opts)) {
-        Ok(markdown) => {
-            println!("Test - Sub/Sup:");
-            println!("HTML: {html6}");
-            println!("Markdown: {markdown}");
-            println!("Expected: H~2~O and x^2^");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html7 = r#"<p>The <abbr title="World Health Organization">WHO</abbr> announced</p>"#;
-    match convert(html7, None) {
-        Ok(markdown) => {
-            println!("Test - Abbr:");
-            println!("HTML: {html7}");
-            println!("Markdown: {markdown}");
-            println!("Expected: The WHO (World Health Organization) announced");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html8 = "<p>This is <u>underlined</u> and <small>small</small> text</p>";
-    match convert(html8, None) {
-        Ok(markdown) => {
-            println!("Test - U/Small:");
-            println!("HTML: {html8}");
-            println!("Markdown: {markdown}");
-            println!("Expected: This is underlined and small text");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-}

data/vendor/html-to-markdown-rs/examples/test_lists.rs DELETED Viewed

@@ -1,39 +0,0 @@
-//! Example: Testing HTML list conversion (ordered and unordered lists)
-fn convert(
-    html: &str,
-    opts: Option<html_to_markdown_rs::ConversionOptions>,
-) -> html_to_markdown_rs::error::Result<String> {
-    html_to_markdown_rs::convert(html, opts).map(|r| r.content.unwrap_or_default())
-}
-fn main() {
-    let html = "<ol><li>First item</li><li>Second item</li><li>Third item</li></ol>";
-    match convert(html, None) {
-        Ok(markdown) => {
-            println!("Test - Ordered list:");
-            println!("HTML: {html}");
-            println!("Markdown:\n{markdown}");
-            println!("Expected:");
-            println!("1. First item");
-            println!("2. Second item");
-            println!("3. Third item");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html2 = "<ul><li>First item</li><li>Second item</li></ul>";
-    match convert(html2, None) {
-        Ok(markdown) => {
-            println!("Test - Unordered list:");
-            println!("HTML: {html2}");
-            println!("Markdown:\n{markdown}");
-            println!("Expected:");
-            println!("* First item");
-            println!("* Second item");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-}

data/vendor/html-to-markdown-rs/examples/test_semantic_tags.rs DELETED Viewed

@@ -1,89 +0,0 @@
-//! Example: Testing HTML5 semantic tags (article, section, nav, etc.)
-fn convert(
-    html: &str,
-    opts: Option<html_to_markdown_rs::ConversionOptions>,
-) -> html_to_markdown_rs::error::Result<String> {
-    html_to_markdown_rs::convert(html, opts).map(|r| r.content.unwrap_or_default())
-}
-fn main() {
-    let html = r"<article>
-        <header><h1>Title</h1></header>
-        <section><p>Content here</p></section>
-        <footer><p>Footer</p></footer>
-    </article>";
-    match convert(html, None) {
-        Ok(markdown) => {
-            println!("Test - Semantic blocks:");
-            println!("HTML: {html}");
-            println!("\nMarkdown:\n{markdown}");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html2 = r#"<figure>
-        <img src="image.jpg" alt="Diagram">
-        <figcaption>Figure 1: Example diagram</figcaption>
-    </figure>"#;
-    match convert(html2, None) {
-        Ok(markdown) => {
-            println!("Test - Figure/Figcaption:");
-            println!("HTML: {html2}");
-            println!("\nMarkdown:\n{markdown}");
-            println!("Expected: ![Diagram](image.jpg)");
-            println!("          *Figure 1: Example diagram*");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html3 = r"<p>As <cite>Shakespeare</cite> said, <q>To be or not to be</q></p>";
-    match convert(html3, None) {
-        Ok(markdown) => {
-            println!("Test - Cite/Quote:");
-            println!("HTML: {html3}");
-            println!("Markdown: {markdown}");
-            println!("Expected: As *Shakespeare* said, \"To be or not to be\"");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html4 = r"<dl>
-        <dt>Term 1</dt>
-        <dd>Definition 1</dd>
-        <dt>Term 2</dt>
-        <dd>Definition 2</dd>
-    </dl>";
-    match convert(html4, None) {
-        Ok(markdown) => {
-            println!("Test - Definition list:");
-            println!("HTML: {html4}");
-            println!("\nMarkdown:\n{markdown}");
-            println!("Expected:");
-            println!("**Term 1**");
-            println!("Definition 1");
-            println!();
-            println!("**Term 2**");
-            println!("Definition 2");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-    let html5 = r"<hgroup>
-        <h1>Main Title</h1>
-        <h2>Subtitle</h2>
-    </hgroup>";
-    match convert(html5, None) {
-        Ok(markdown) => {
-            println!("Test - Hgroup:");
-            println!("HTML: {html5}");
-            println!("\nMarkdown:\n{markdown}");
-            println!();
-        }
-        Err(e) => eprintln!("Error: {e}"),
-    }
-}