RubyGems - html-to-markdown - Versions diffs - 2.12.0 → 2.13.0 - Mend

html-to-markdown 2.12.0 → 2.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (11) hide show

checksums.yaml +4 -4
data/Gemfile.lock +1 -1
data/METADATA.md +227 -0
data/README.md +34 -0
data/ext/html-to-markdown-rb/native/Cargo.toml +4 -3
data/ext/html-to-markdown-rb/native/src/lib.rs +269 -0
data/lib/html_to_markdown/version.rb +1 -1
data/lib/html_to_markdown.rb +11 -0
data/sig/html_to_markdown.rbs +123 -8
data/spec/metadata_extraction_spec.rb +440 -0
metadata +4 -2

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 64b0bc7bc7ba063d5e3bdcdf4ac3edde703e9358a208ea73f54cfa0d48de53e0
-  data.tar.gz: 3bb71b6453bd3d8fc0095d22e9da317ba2a20aef0ca29bfd2c53b18ec0d212a2
+  metadata.gz: 5643900ad2e99a827fb3814cba27fae1ca60f33dda4d0ff82af524b08afd9ad8
+  data.tar.gz: '029a84a8b9f7e2f08453651807313f0b4d40c83cf9af1ebb1399a8bcfbf1ff00'
 SHA512:
-  metadata.gz: cd5e3ca8fd4c06817a129587b30df66b2a5c114c329047ca4aa0f691627b0a057bcaa8ed930460dc7399808c32779bc7a8f576e6eee8739c482d0fe3bbca4783
-  data.tar.gz: 5f049999f764f98f9980a5aa3bdb057c18f0a7914ed844a80613d88a7ae84b83a3e9824752daa60f82b25458ee871b028bd87c02713e0422e1abbaf0b2d6c709
+  metadata.gz: 8ee229c8b956f68d316ee17a08163d88ed89beb36a912e2ab03a8b52a625d217eb97c6428c12610014f89cc583515dc56531f7622c17ccdf9c66d8f10d3467db
+  data.tar.gz: e4ceff847ce455aa4c925cfd3c7084540649a45e85665948f3faaa1d8f0c4d31e69cb1efdb4eee2b185d6553db1c6eb5bbf376f96e8e5aa394d630c4f46bf93c

data/Gemfile.lock CHANGED Viewed

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    html-to-markdown (2.12.0)
+    html-to-markdown (2.13.0)
       rb_sys (>= 0.9, < 1.0)
 GEM

data/METADATA.md ADDED Viewed

@@ -0,0 +1,227 @@
+# Metadata Extraction for Ruby Bindings
+Complete Ruby Magnus binding implementation for HTML-to-Markdown metadata extraction with full RBS type signatures.
+## Features
+The Ruby binding provides comprehensive metadata extraction during HTML-to-Markdown conversion:
+- **Document Metadata**: title, description, keywords, author, canonical URL, language, text direction
+- **Open Graph & Twitter Card**: social media metadata extraction
+- **Headers**: h1-h6 extraction with hierarchy, ids, and depth tracking
+- **Links**: hyperlink extraction with type classification (anchor, internal, external, email, phone)
+- **Images**: image extraction with source type (data_uri, inline_svg, external, relative) and dimensions
+- **Structured Data**: JSON-LD, Microdata, and RDFa extraction
+## API
+### Basic Usage
+```ruby
+require 'html_to_markdown'
+html = '<html lang="en"><head><title>Test</title></head><body><h1>Hello</h1></body></html>'
+markdown, metadata = HtmlToMarkdown.convert_with_metadata(html)
+puts markdown
+puts metadata[:document][:title]  # "Test"
+puts metadata[:headers].length     # 1
+```
+### With Conversion Options
+```ruby
+conv_opts = { heading_style: :atx_closed }
+metadata_opts = { extract_headers: true, extract_links: false }
+markdown, metadata = HtmlToMarkdown.convert_with_metadata(
+  html,
+  conv_opts,
+  metadata_opts
+)
+```
+### Return Value
+Returns a 2-element array: `[markdown_string, metadata_hash]`
+The metadata hash contains:
+```ruby
+{
+  document: {
+    title: String?,
+    description: String?,
+    keywords: Array[String],
+    author: String?,
+    canonical_url: String?,
+    base_href: String?,
+    language: String?,
+    text_direction: "ltr" | "rtl" | "auto" | nil,
+    open_graph: Hash[String, String],
+    twitter_card: Hash[String, String],
+    meta_tags: Hash[String, String]
+  },
+  headers: [
+    {
+      level: Integer,          # 1-6
+      text: String,
+      id: String?,
+      depth: Integer,
+      html_offset: Integer
+    }
+  ],
+  links: [
+    {
+      href: String,
+      text: String,
+      title: String?,
+      link_type: "anchor" | "internal" | "external" | "email" | "phone" | "other",
+      rel: Array[String],
+      attributes: Hash[String, String]
+    }
+  ],
+  images: [
+    {
+      src: String,
+      alt: String?,
+      title: String?,
+      dimensions: [Integer, Integer]?,
+      image_type: "data_uri" | "inline_svg" | "external" | "relative",
+      attributes: Hash[String, String]
+    }
+  ],
+  structured_data: [
+    {
+      data_type: "json_ld" | "microdata" | "rdfa",
+      raw_json: String,
+      schema_type: String?
+    }
+  ]
+}
+```
+## Metadata Configuration
+Pass a hash with the following options to control which metadata types are extracted:
+```ruby
+config = {
+  extract_headers: true,           # Extract h1-h6 elements (default: true)
+  extract_links: true,             # Extract <a> elements (default: true)
+  extract_images: true,            # Extract <img> elements (default: true)
+  extract_structured_data: true,   # Extract JSON-LD/Microdata/RDFa (default: true)
+  max_structured_data_size: 1_000_000  # Max bytes for structured data (default: 1MB)
+}
+markdown, metadata = HtmlToMarkdown.convert_with_metadata(html, nil, config)
+```
+## Type Signatures
+All types are defined in RBS format in `sig/html_to_markdown.rbs`:
+- `document_metadata` - Document-level metadata structure
+- `header_metadata` - Individual header element
+- `link_metadata` - Individual link element
+- `image_metadata` - Individual image element
+- `structured_data` - Structured data block
+- `extended_metadata` - Complete metadata extraction result
+Uses strict RBS type checking with Steep for full type safety.
+## Implementation Details
+### Architecture
+The Rust implementation uses a single-pass collector pattern for efficient metadata extraction:
+1. **No duplication**: Core logic lives in Rust (`crates/html-to-markdown/src/metadata.rs`)
+2. **Minimal wrapper layer**: Ruby binding in `crates/html-to-markdown-rb/src/lib.rs`
+3. **Type translation**: Rust types → Ruby hashes with proper Magnus bindings
+4. **Hash conversion**: Uses Magnus `RHash` API for efficient Ruby hash construction
+### Hash Conversion Pattern
+Following the inline_images pattern:
+```rust
+fn document_metadata_to_ruby(ruby: &Ruby, doc: RustDocumentMetadata) -> Result<Value, Error> {
+    let hash = ruby.hash_new();
+    hash.aset(ruby.intern("title"), opt_string_to_ruby(ruby, doc.title)?)?;
+    hash.aset(ruby.intern("keywords"), keywords_array)?;
+    // ... more fields
+    Ok(hash.as_value())
+}
+```
+### Feature Flag
+The metadata feature is gated by a Cargo feature in `Cargo.toml`:
+```toml
+[features]
+metadata = ["html-to-markdown-rs/metadata"]
+```
+This ensures:
+- Zero overhead when metadata is not needed
+- Clean integration with feature flag detection
+- Consistent with Python binding implementation
+## Tests
+Comprehensive RSpec test suite in `spec/metadata_extraction_spec.rb`:
+```bash
+cd packages/ruby
+bundle exec rake compile
+bundle exec rspec spec/metadata_extraction_spec.rb
+```
+Tests cover:
+- All metadata types extraction
+- Configuration flags
+- Edge cases (empty HTML, malformed input, special characters)
+- Return value structure validation
+- Integration with conversion options
+## Language Parity
+Implements the same API as the Python binding:
+- Same method signature: `convert_with_metadata(html, options, metadata_config)`
+- Same return type: `[markdown, metadata_dict]`
+- Same metadata structures and field names
+- Same enum values (link_type, image_type, data_type, text_direction)
+Enables seamless migration and multi-language development.
+## Performance
+Single-pass collection during tree traversal:
+- No additional parsing passes
+- Minimal memory overhead
+- Configurable extraction granularity
+- Built-in size limits for safety
+## Building and Testing
+Build the extension with metadata support:
+```bash
+cd packages/ruby
+bundle exec rake compile -- --release --features metadata
+```
+Run type checking:
+```bash
+steep check
+```
+Run tests:
+```bash
+bundle exec rspec spec/metadata_extraction_spec.rb
+```

data/README.md CHANGED Viewed

@@ -184,6 +184,40 @@ result.inline_images.each do |img|
 end
 ```
+### Metadata extraction
+```ruby
+require 'html_to_markdown'
+html = <<~HTML
+  <html>
+    <head>
+      <title>Example</title>
+      <meta name="description" content="Demo page">
+      <link rel="canonical" href="https://example.com/page">
+    </head>
+    <body>
+      <h1 id="welcome">Welcome</h1>
+      <a href="https://example.com" rel="nofollow external">Example link</a>
+      <img src="https://example.com/image.jpg" alt="Hero" width="640" height="480">
+    </body>
+  </html>
+HTML
+markdown, metadata = HtmlToMarkdown.convert_with_metadata(
+  html,
+  { heading_style: :atx },
+  { extract_links: true, extract_images: true, extract_headers: true }
+)
+puts markdown
+puts metadata[:document][:title]         # "Example"
+puts metadata[:links].first[:rel]        # ["nofollow", "external"]
+puts metadata[:images].first[:dimensions] # [640, 480]
+```
+`metadata` contains document tags (title, description, canonical URL, Open Graph/Twitter cards), enriched links (type, text, `rel`, raw attributes), images (alt, title, inferred dimensions, attributes), headers with depth/offsets, and optional structured data when enabled.
 ## CLI
 The gem bundles a small proxy for the Rust CLI binary. Use it when you need parity with the standalone `html-to-markdown` executable.

data/ext/html-to-markdown-rb/native/Cargo.toml CHANGED Viewed

@@ -1,6 +1,6 @@
 [package]
 name = "html-to-markdown-rb"
-version = "2.12.0"
+version = "2.13.0"
 edition = "2024"
 authors = ["Na'aman Hirschfeld <nhirschfeld@gmail.com>"]
 license = "MIT"
@@ -18,10 +18,11 @@ name = "html_to_markdown_rb"
 crate-type = ["cdylib", "rlib"]
 [features]
-default = []
+default = ["metadata"]
+metadata = ["html-to-markdown-rs/metadata"]
 [dependencies]
-html-to-markdown-rs = { version = "2.12.0", features = ["inline-images"] }
+html-to-markdown-rs = { version = "2.13.0", features = ["inline-images"] }
 magnus = { git = "https://github.com/matsadler/magnus", rev = "f6db11769efb517427bf7f121f9c32e18b059b38", features = ["rb-sys"] }
 [dev-dependencies]

data/ext/html-to-markdown-rb/native/src/lib.rs CHANGED Viewed

@@ -4,6 +4,17 @@ use html_to_markdown_rs::{
     PreprocessingPreset, WhitespaceMode, convert as convert_inner,
     convert_with_inline_images as convert_with_inline_images_inner, error::ConversionError, safety::guard_panic,
 };
+#[cfg(feature = "metadata")]
+use html_to_markdown_rs::convert_with_metadata as convert_with_metadata_inner;
+#[cfg(feature = "metadata")]
+use html_to_markdown_rs::metadata::{
+    DocumentMetadata as RustDocumentMetadata, ExtendedMetadata as RustExtendedMetadata,
+    HeaderMetadata as RustHeaderMetadata, ImageMetadata as RustImageMetadata, ImageType as RustImageType,
+    LinkMetadata as RustLinkMetadata, LinkType as RustLinkType, MetadataConfig as RustMetadataConfig,
+    StructuredData as RustStructuredData, StructuredDataType as RustStructuredDataType,
+    TextDirection as RustTextDirection,
+};
 use magnus::prelude::*;
 use magnus::r_hash::ForEach;
 use magnus::{Error, RArray, RHash, Ruby, Symbol, TryConvert, Value, function, scan_args::scan_args};
@@ -423,6 +434,261 @@ fn convert_with_inline_images_fn(ruby: &Ruby, args: &[Value]) -> Result<Value, E
     extraction_to_value(ruby, extraction)
 }
+#[cfg(feature = "metadata")]
+fn build_metadata_config(_ruby: &Ruby, config: Option<Value>) -> Result<RustMetadataConfig, Error> {
+    let mut cfg = RustMetadataConfig::default();
+    let Some(config) = config else {
+        return Ok(cfg);
+    };
+    if config.is_nil() {
+        return Ok(cfg);
+    }
+    let hash = RHash::from_value(config).ok_or_else(|| arg_error("metadata_config must be provided as a Hash"))?;
+    hash.foreach(|key: Value, val: Value| {
+        let key_name = symbol_to_string(key)?;
+        match key_name.as_str() {
+            "extract_headers" => {
+                cfg.extract_headers = bool::try_convert(val)?;
+            }
+            "extract_links" => {
+                cfg.extract_links = bool::try_convert(val)?;
+            }
+            "extract_images" => {
+                cfg.extract_images = bool::try_convert(val)?;
+            }
+            "extract_structured_data" => {
+                cfg.extract_structured_data = bool::try_convert(val)?;
+            }
+            "max_structured_data_size" => {
+                cfg.max_structured_data_size = usize::try_convert(val)?;
+            }
+            _ => {}
+        }
+        Ok(ForEach::Continue)
+    })?;
+    Ok(cfg)
+}
+#[cfg(feature = "metadata")]
+fn opt_string_to_ruby(ruby: &Ruby, opt: Option<String>) -> Result<Value, Error> {
+    match opt {
+        Some(val) => Ok(ruby.str_from_slice(val.as_bytes()).as_value()),
+        None => Ok(ruby.qnil().as_value()),
+    }
+}
+#[cfg(feature = "metadata")]
+fn btreemap_to_ruby_hash(ruby: &Ruby, map: std::collections::BTreeMap<String, String>) -> Result<Value, Error> {
+    let hash = ruby.hash_new();
+    for (k, v) in map {
+        hash.aset(k, v)?;
+    }
+    Ok(hash.as_value())
+}
+#[cfg(feature = "metadata")]
+fn text_direction_to_string(text_direction: Option<RustTextDirection>) -> Option<&'static str> {
+    match text_direction {
+        Some(RustTextDirection::LeftToRight) => Some("ltr"),
+        Some(RustTextDirection::RightToLeft) => Some("rtl"),
+        Some(RustTextDirection::Auto) => Some("auto"),
+        None => None,
+    }
+}
+#[cfg(feature = "metadata")]
+fn link_type_to_string(link_type: &RustLinkType) -> &'static str {
+    match link_type {
+        RustLinkType::Anchor => "anchor",
+        RustLinkType::Internal => "internal",
+        RustLinkType::External => "external",
+        RustLinkType::Email => "email",
+        RustLinkType::Phone => "phone",
+        RustLinkType::Other => "other",
+    }
+}
+#[cfg(feature = "metadata")]
+fn image_type_to_string(image_type: &RustImageType) -> &'static str {
+    match image_type {
+        RustImageType::DataUri => "data_uri",
+        RustImageType::InlineSvg => "inline_svg",
+        RustImageType::External => "external",
+        RustImageType::Relative => "relative",
+    }
+}
+#[cfg(feature = "metadata")]
+fn structured_data_type_to_string(data_type: &RustStructuredDataType) -> &'static str {
+    match data_type {
+        RustStructuredDataType::JsonLd => "json_ld",
+        RustStructuredDataType::Microdata => "microdata",
+        RustStructuredDataType::RDFa => "rdfa",
+    }
+}
+#[cfg(feature = "metadata")]
+fn document_metadata_to_ruby(ruby: &Ruby, doc: RustDocumentMetadata) -> Result<Value, Error> {
+    let hash = ruby.hash_new();
+    hash.aset(ruby.intern("title"), opt_string_to_ruby(ruby, doc.title)?)?;
+    hash.aset(ruby.intern("description"), opt_string_to_ruby(ruby, doc.description)?)?;
+    let keywords = ruby.ary_new();
+    for keyword in doc.keywords {
+        keywords.push(keyword)?;
+    }
+    hash.aset(ruby.intern("keywords"), keywords)?;
+    hash.aset(ruby.intern("author"), opt_string_to_ruby(ruby, doc.author)?)?;
+    hash.aset(
+        ruby.intern("canonical_url"),
+        opt_string_to_ruby(ruby, doc.canonical_url)?,
+    )?;
+    hash.aset(ruby.intern("base_href"), opt_string_to_ruby(ruby, doc.base_href)?)?;
+    hash.aset(ruby.intern("language"), opt_string_to_ruby(ruby, doc.language)?)?;
+    match text_direction_to_string(doc.text_direction) {
+        Some(dir) => hash.aset(ruby.intern("text_direction"), dir)?,
+        None => hash.aset(ruby.intern("text_direction"), ruby.qnil())?,
+    }
+    hash.aset(ruby.intern("open_graph"), btreemap_to_ruby_hash(ruby, doc.open_graph)?)?;
+    hash.aset(
+        ruby.intern("twitter_card"),
+        btreemap_to_ruby_hash(ruby, doc.twitter_card)?,
+    )?;
+    hash.aset(ruby.intern("meta_tags"), btreemap_to_ruby_hash(ruby, doc.meta_tags)?)?;
+    Ok(hash.as_value())
+}
+#[cfg(feature = "metadata")]
+fn headers_to_ruby(ruby: &Ruby, headers: Vec<RustHeaderMetadata>) -> Result<Value, Error> {
+    let array = ruby.ary_new();
+    for header in headers {
+        let hash = ruby.hash_new();
+        hash.aset(ruby.intern("level"), header.level)?;
+        hash.aset(ruby.intern("text"), header.text)?;
+        hash.aset(ruby.intern("id"), opt_string_to_ruby(ruby, header.id)?)?;
+        hash.aset(ruby.intern("depth"), header.depth as i64)?;
+        hash.aset(ruby.intern("html_offset"), header.html_offset as i64)?;
+        array.push(hash)?;
+    }
+    Ok(array.as_value())
+}
+#[cfg(feature = "metadata")]
+fn links_to_ruby(ruby: &Ruby, links: Vec<RustLinkMetadata>) -> Result<Value, Error> {
+    let array = ruby.ary_new();
+    for link in links {
+        let hash = ruby.hash_new();
+        hash.aset(ruby.intern("href"), link.href)?;
+        hash.aset(ruby.intern("text"), link.text)?;
+        hash.aset(ruby.intern("title"), opt_string_to_ruby(ruby, link.title)?)?;
+        hash.aset(ruby.intern("link_type"), link_type_to_string(&link.link_type))?;
+        let rel_array = ruby.ary_new();
+        for r in link.rel {
+            rel_array.push(r)?;
+        }
+        hash.aset(ruby.intern("rel"), rel_array)?;
+        hash.aset(ruby.intern("attributes"), btreemap_to_ruby_hash(ruby, link.attributes)?)?;
+        array.push(hash)?;
+    }
+    Ok(array.as_value())
+}
+#[cfg(feature = "metadata")]
+fn images_to_ruby(ruby: &Ruby, images: Vec<RustImageMetadata>) -> Result<Value, Error> {
+    let array = ruby.ary_new();
+    for image in images {
+        let hash = ruby.hash_new();
+        hash.aset(ruby.intern("src"), image.src)?;
+        hash.aset(ruby.intern("alt"), opt_string_to_ruby(ruby, image.alt)?)?;
+        hash.aset(ruby.intern("title"), opt_string_to_ruby(ruby, image.title)?)?;
+        match image.dimensions {
+            Some((width, height)) => {
+                let dims = ruby.ary_new();
+                dims.push(width as i64)?;
+                dims.push(height as i64)?;
+                hash.aset(ruby.intern("dimensions"), dims)?;
+            }
+            None => {
+                hash.aset(ruby.intern("dimensions"), ruby.qnil())?;
+            }
+        }
+        hash.aset(ruby.intern("image_type"), image_type_to_string(&image.image_type))?;
+        hash.aset(
+            ruby.intern("attributes"),
+            btreemap_to_ruby_hash(ruby, image.attributes)?,
+        )?;
+        array.push(hash)?;
+    }
+    Ok(array.as_value())
+}
+#[cfg(feature = "metadata")]
+fn structured_data_to_ruby(ruby: &Ruby, data: Vec<RustStructuredData>) -> Result<Value, Error> {
+    let array = ruby.ary_new();
+    for item in data {
+        let hash = ruby.hash_new();
+        hash.aset(
+            ruby.intern("data_type"),
+            structured_data_type_to_string(&item.data_type),
+        )?;
+        hash.aset(ruby.intern("raw_json"), item.raw_json)?;
+        hash.aset(ruby.intern("schema_type"), opt_string_to_ruby(ruby, item.schema_type)?)?;
+        array.push(hash)?;
+    }
+    Ok(array.as_value())
+}
+#[cfg(feature = "metadata")]
+fn extended_metadata_to_ruby(ruby: &Ruby, metadata: RustExtendedMetadata) -> Result<Value, Error> {
+    let hash = ruby.hash_new();
+    hash.aset(
+        ruby.intern("document"),
+        document_metadata_to_ruby(ruby, metadata.document)?,
+    )?;
+    hash.aset(ruby.intern("headers"), headers_to_ruby(ruby, metadata.headers)?)?;
+    hash.aset(ruby.intern("links"), links_to_ruby(ruby, metadata.links)?)?;
+    hash.aset(ruby.intern("images"), images_to_ruby(ruby, metadata.images)?)?;
+    hash.aset(
+        ruby.intern("structured_data"),
+        structured_data_to_ruby(ruby, metadata.structured_data)?,
+    )?;
+    Ok(hash.as_value())
+}
+#[cfg(feature = "metadata")]
+fn convert_with_metadata_fn(ruby: &Ruby, args: &[Value]) -> Result<Value, Error> {
+    let parsed = scan_args::<(String,), (Option<Value>, Option<Value>), (), (), (), ()>(args)?;
+    let html = parsed.required.0;
+    let options = build_conversion_options(ruby, parsed.optional.0)?;
+    let metadata_config = build_metadata_config(ruby, parsed.optional.1)?;
+    let (markdown, metadata) =
+        guard_panic(|| convert_with_metadata_inner(&html, Some(options), metadata_config)).map_err(conversion_error)?;
+    // Convert to Ruby array [markdown, metadata_hash]
+    let array = ruby.ary_new();
+    array.push(markdown)?;
+    array.push(extended_metadata_to_ruby(ruby, metadata)?)?;
+    Ok(array.as_value())
+}
 #[magnus::init]
 fn init(ruby: &Ruby) -> Result<(), Error> {
     let module = ruby.define_module("HtmlToMarkdown")?;
@@ -434,5 +700,8 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
         function!(convert_with_inline_images_fn, -1),
     )?;
+    #[cfg(feature = "metadata")]
+    module.define_singleton_method("convert_with_metadata", function!(convert_with_metadata_fn, -1))?;
     Ok(())
 }

data/lib/html_to_markdown/version.rb CHANGED Viewed

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module HtmlToMarkdown
-  VERSION = '2.12.0'
+  VERSION = '2.13.0'
 end

data/lib/html_to_markdown.rb CHANGED Viewed

@@ -14,6 +14,7 @@ module HtmlToMarkdown
     alias native_convert_with_inline_images convert_with_inline_images
     alias native_options options
     alias native_convert_with_options convert_with_options
+    alias native_convert_with_metadata convert_with_metadata
   end
   module_function
@@ -33,4 +34,14 @@ module HtmlToMarkdown
   def options(options_hash = nil)
     native_options(options_hash)
   end
+  # Convert HTML to Markdown with comprehensive metadata extraction
+  #
+  # @param html [String] HTML string to convert
+  # @param options [Hash, nil] Optional conversion configuration
+  # @param metadata_config [Hash, nil] Optional metadata extraction configuration
+  # @return [Array<String, Hash>] Array containing [markdown_string, metadata_hash]
+  def convert_with_metadata(html, options = nil, metadata_config = nil)
+    native_convert_with_metadata(html.to_s, options, metadata_config)
+  end
 end

data/sig/html_to_markdown.rbs CHANGED Viewed

@@ -87,6 +87,74 @@ module HtmlToMarkdown
     warnings: Array[inline_image_warning]
   }
+  type metadata_config = {
+    extract_headers?: bool,
+    extract_links?: bool,
+    extract_images?: bool,
+    extract_structured_data?: bool,
+    max_structured_data_size?: Integer
+  }
+  type text_direction = "ltr" | "rtl" | "auto" | nil
+  type document_metadata = {
+    title: String?,
+    description: String?,
+    keywords: Array[String],
+    author: String?,
+    canonical_url: String?,
+    base_href: String?,
+    language: String?,
+    text_direction: text_direction,
+    open_graph: Hash[String, String],
+    twitter_card: Hash[String, String],
+    meta_tags: Hash[String, String]
+  }
+  type header_metadata = {
+    level: Integer,
+    text: String,
+    id: String?,
+    depth: Integer,
+    html_offset: Integer
+  }
+  type link_type = "anchor" | "internal" | "external" | "email" | "phone" | "other"
+  type link_metadata = {
+    href: String,
+    text: String,
+    title: String?,
+    link_type: link_type,
+    rel: Array[String],
+    attributes: Hash[String, String]
+  }
+  type image_type = "data_uri" | "inline_svg" | "external" | "relative"
+  type image_metadata = {
+    src: String,
+    alt: String?,
+    title: String?,
+    dimensions: [Integer, Integer]?,
+    image_type: image_type,
+    attributes: Hash[String, String]
+  }
+  type structured_data = {
+    data_type: "json_ld" | "microdata" | "rdfa",
+    raw_json: String,
+    schema_type: String?
+  }
+  type extended_metadata = {
+    document: document_metadata,
+    headers: Array[header_metadata],
+    links: Array[link_metadata],
+    images: Array[image_metadata],
+    structured_data: Array[structured_data]
+  }
   # Native methods (implemented in Rust via Magnus/rb-sys)
   # These are aliased from the Rust extension and available as both module and instance methods
   private
@@ -99,6 +167,11 @@ module HtmlToMarkdown
     conversion_options? options,
     inline_image_config? image_config
   ) -> html_extraction
+  def self.native_convert_with_metadata: (
+    String html,
+    conversion_options? options,
+    metadata_config? metadata_config
+  ) -> [String, extended_metadata]
   def native_convert: (String html, conversion_options? options) -> String
   def native_options: (conversion_options? options_hash) -> Options
@@ -108,14 +181,19 @@ module HtmlToMarkdown
     conversion_options? options,
     inline_image_config? image_config
   ) -> html_extraction
+  def native_convert_with_metadata: (
+    String html,
+    conversion_options? options,
+    metadata_config? metadata_config
+  ) -> [String, extended_metadata]
   public
   # Convert HTML to Markdown with optional configuration
-  def self.convert: (String html, ?conversion_options? options) -> String
+  def self.convert: (String html, ?conversion_options options) -> String
   # Create a reusable options handle for performance
-  def self.options: (?conversion_options? options_hash) -> Options
+  def self.options: (?conversion_options options_hash) -> Options
   # Convert HTML using a pre-built options handle
   def self.convert_with_options: (String html, Options options_handle) -> String
@@ -123,17 +201,54 @@ module HtmlToMarkdown
   # Convert HTML with inline image extraction
   def self.convert_with_inline_images: (
     String html,
-    ?conversion_options? options,
-    ?inline_image_config? image_config
+    ?conversion_options options,
+    ?inline_image_config image_config
   ) -> html_extraction
+  # Convert HTML to Markdown with metadata extraction
+  #
+  # Extracts comprehensive metadata (headers, links, images, structured data) during conversion.
+  #
+  # Args:
+  #   html: HTML string to convert
+  #   options: Optional conversion configuration
+  #   metadata_config: Optional metadata extraction configuration
+  #
+  # Returns:
+  #   Array containing:
+  #   - [0] markdown: String - Converted markdown output
+  #   - [1] metadata: Hash - Extracted metadata with document, headers, links, images, structured_data
+  #
+  # The metadata hash contains:
+  #   - document: Document-level metadata (title, description, lang, etc.)
+  #   - headers: List of header elements with hierarchy
+  #   - links: List of extracted hyperlinks with classification
+  #   - images: List of extracted images with metadata
+  #   - structured_data: List of JSON-LD, Microdata, or RDFa blocks
+  #
+  # Example:
+  #   html = '<html lang="en"><head><title>Test</title></head><body><h1>Hello</h1></body></html>'
+  #   markdown, metadata = HtmlToMarkdown.convert_with_metadata(html)
+  #   puts "Title: #{metadata['document']['title']}"
+  #   puts "Headers: #{metadata['headers'].length}"
+  def self.convert_with_metadata: (
+    String html,
+    ?conversion_options options,
+    ?metadata_config metadata_config
+  ) -> [String, extended_metadata]
   # Instance method versions (created by module_function)
-  def convert: (String html, ?conversion_options? options) -> String
-  def options: (?conversion_options? options_hash) -> Options
+  def convert: (String html, ?conversion_options options) -> String
+  def options: (?conversion_options options_hash) -> Options
   def convert_with_options: (String html, Options options_handle) -> String
   def convert_with_inline_images: (
     String html,
-    ?conversion_options? options,
-    ?inline_image_config? image_config
+    ?conversion_options options,
+    ?inline_image_config image_config
   ) -> html_extraction
+  def convert_with_metadata: (
+    String html,
+    ?conversion_options options,
+    ?metadata_config metadata_config
+  ) -> [String, extended_metadata]
 end

data/spec/metadata_extraction_spec.rb ADDED Viewed

@@ -0,0 +1,440 @@
+# frozen_string_literal: true
+require 'spec_helper'
+RSpec.describe HtmlToMarkdown do
+  describe '.convert_with_metadata' do
+    it 'returns array with markdown and metadata' do
+      html = '<html><head><title>Test</title></head><body><p>Content</p></body></html>'
+      result = described_class.convert_with_metadata(html)
+      expect(result).to be_an(Array)
+      expect(result.length).to eq(2)
+      expect(result[0]).to be_a(String)
+      expect(result[1]).to be_a(Hash)
+    end
+    context 'when extracting document metadata' do
+      it 'extracts title' do
+        html = '<html><head><title>My Page Title</title></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:title]).to eq('My Page Title')
+      end
+      it 'extracts description' do
+        html = <<~HTML
+          <html>
+            <head><meta name="description" content="Page description"></head>
+            <body><p>Content</p></body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:description]).to eq('Page description')
+      end
+      it 'extracts keywords' do
+        html = <<~HTML
+          <html>
+            <head><meta name="keywords" content="keyword1, keyword2, keyword3"></head>
+            <body><p>Content</p></body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:keywords]).to include('keyword1', 'keyword2', 'keyword3')
+      end
+      it 'extracts author' do
+        html = '<html><head><meta name="author" content="John Doe"></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:author]).to eq('John Doe')
+      end
+      it 'extracts base href' do
+        html = '<html><head><base href="https://example.com/"></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:base_href]).to eq('https://example.com/')
+      end
+      it 'extracts canonical URL' do
+        html = '<html><head><link rel="canonical" href="https://example.com/page"></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:canonical_url]).to eq('https://example.com/page')
+      end
+      it 'extracts language' do
+        html = '<html lang="en"><head></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:language]).to eq('en')
+      end
+      it 'extracts text direction' do
+        html = '<html dir="ltr"><head></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:text_direction]).to eq('ltr')
+      end
+      it 'extracts open graph metadata' do
+        html = <<~HTML
+          <html>
+            <head>
+              <meta property="og:title" content="OG Title">
+              <meta property="og:description" content="OG Description">
+              <meta property="og:image" content="https://example.com/image.jpg">
+            </head>
+            <body><p>Content</p></body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:open_graph]).to include(
+          'title' => 'OG Title',
+          'description' => 'OG Description',
+          'image' => 'https://example.com/image.jpg'
+        )
+      end
+      it 'extracts twitter card metadata' do
+        html = <<~HTML
+          <html>
+            <head>
+              <meta name="twitter:card" content="summary_large_image">
+              <meta name="twitter:title" content="Twitter Title">
+            </head>
+            <body><p>Content</p></body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:twitter_card]).to include(
+          'card' => 'summary_large_image',
+          'title' => 'Twitter Title'
+        )
+      end
+      it 'returns empty arrays and hashes for missing metadata' do
+        html = '<p>Content</p>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:title]).to be_nil
+        expect(metadata[:document][:description]).to be_nil
+        expect(metadata[:document][:keywords]).to eq([])
+        expect(metadata[:document][:open_graph]).to eq({})
+        expect(metadata[:document][:twitter_card]).to eq({})
+        expect(metadata[:document][:meta_tags]).to eq({})
+      end
+    end
+    context 'when extracting header metadata' do
+      it 'extracts headers with hierarchy' do
+        html = <<~HTML
+          <html>
+            <body>
+              <h1>Main Title</h1>
+              <h2>Section</h2>
+              <h3>Subsection</h3>
+            </body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:headers].length).to eq(3)
+        expect(metadata[:headers][0][:level]).to eq(1)
+        expect(metadata[:headers][0][:text]).to eq('Main Title')
+        expect(metadata[:headers][1][:level]).to eq(2)
+        expect(metadata[:headers][1][:text]).to eq('Section')
+        expect(metadata[:headers][2][:level]).to eq(3)
+        expect(metadata[:headers][2][:text]).to eq('Subsection')
+      end
+      it 'includes header id' do
+        html = '<html><body><h1 id="main-title">Title</h1></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:headers][0][:id]).to eq('main-title')
+      end
+      it 'includes depth and html_offset' do
+        html = '<html><body><h1>Title</h1></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        header = metadata[:headers][0]
+        expect(header).to include(:depth, :html_offset)
+        expect(header[:depth]).to be_a(Integer)
+        expect(header[:html_offset]).to be_a(Integer)
+      end
+    end
+    context 'when extracting link metadata' do
+      it 'extracts links with classification' do
+        html = <<~HTML
+          <html>
+            <body>
+              <a href="#section">Anchor</a>
+              <a href="https://example.com">External</a>
+              <a href="/page">Internal</a>
+              <a href="mailto:test@example.com">Email</a>
+              <a href="tel:+1234567890">Phone</a>
+            </body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        links = metadata[:links]
+        expect(links.length).to eq(5)
+        expect(links[0][:link_type]).to eq('anchor')
+        expect(links[1][:link_type]).to eq('external')
+        expect(links[2][:link_type]).to eq('internal')
+        expect(links[3][:link_type]).to eq('email')
+        expect(links[4][:link_type]).to eq('phone')
+      end
+      it 'includes link text and href' do
+        html = '<html><body><a href="https://example.com">Click here</a></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        link = metadata[:links][0]
+        expect(link[:href]).to eq('https://example.com')
+        expect(link[:text]).to eq('Click here')
+      end
+      it 'includes link title attribute' do
+        html = '<html><body><a href="https://example.com" title="Example Site">Link</a></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        link = metadata[:links][0]
+        expect(link[:title]).to eq('Example Site')
+      end
+      it 'includes link rel attributes' do
+        html = '<html><body><a href="https://example.com" rel="nofollow external">Link</a></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        link = metadata[:links][0]
+        expect(link[:rel]).to include('nofollow', 'external')
+      end
+      it 'includes link attributes' do
+        html = '<html><body><a href="https://example.com" data-custom="value">Link</a></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        link = metadata[:links][0]
+        expect(link[:attributes]).to include('data-custom' => 'value')
+      end
+    end
+    context 'when extracting image metadata' do
+      it 'extracts images with source type' do
+        html = <<~HTML
+          <html>
+            <body>
+              <img src="https://example.com/image.jpg" alt="External">
+              <img src="/images/local.jpg" alt="Relative">
+              <img src="data:image/png;base64,..." alt="Data URI">
+            </body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        images = metadata[:images]
+        expect(images.length).to eq(3)
+        expect(images[0][:image_type]).to eq('external')
+        expect(images[1][:image_type]).to eq('relative')
+        expect(images[2][:image_type]).to eq('data_uri')
+      end
+      it 'includes image alt and title' do
+        html = '<html><body><img src="image.jpg" alt="Alt text" title="Image title"></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        image = metadata[:images][0]
+        expect(image[:alt]).to eq('Alt text')
+        expect(image[:title]).to eq('Image title')
+      end
+      it 'includes image dimensions' do
+        html = '<html><body><img src="image.jpg" width="800" height="600"></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        image = metadata[:images][0]
+        expect(image[:dimensions]).to be_an(Array)
+        expect(image[:dimensions].length).to eq(2)
+      end
+      it 'handles missing image attributes' do
+        html = '<html><body><img src="image.jpg"></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        image = metadata[:images][0]
+        expect(image[:alt]).to be_nil
+        expect(image[:title]).to be_nil
+      end
+    end
+    context 'with metadata configuration flags' do
+      it 'respects extract_headers flag' do
+        html = '<html><body><h1>Title</h1><p>Content</p></body></html>'
+        config = { extract_headers: false }
+        _, metadata = described_class.convert_with_metadata(html, nil, config)
+        expect(metadata[:headers]).to eq([])
+      end
+      it 'respects extract_links flag' do
+        html = '<html><body><a href="https://example.com">Link</a></body></html>'
+        config = { extract_links: false }
+        _, metadata = described_class.convert_with_metadata(html, nil, config)
+        expect(metadata[:links]).to eq([])
+      end
+      it 'respects extract_images flag' do
+        html = '<html><body><img src="image.jpg" alt="test"></body></html>'
+        config = { extract_images: false }
+        _, metadata = described_class.convert_with_metadata(html, nil, config)
+        expect(metadata[:images]).to eq([])
+      end
+      it 'respects extract_structured_data flag' do
+        html = '<html><body><script type="application/ld+json">{"@type":"Article"}</script></body></html>'
+        config = { extract_structured_data: false }
+        _, metadata = described_class.convert_with_metadata(html, nil, config)
+        expect(metadata[:structured_data]).to eq([])
+      end
+    end
+    context 'with conversion options and metadata config' do
+      it 'accepts both conversion options and metadata config' do
+        html = '<html><head><title>Test</title></head><body><h1>Heading</h1></body></html>'
+        conv_opts = { heading_style: :atx_closed }
+        meta_opts = { extract_headers: true }
+        markdown, metadata = described_class.convert_with_metadata(html, conv_opts, meta_opts)
+        expect(markdown).to include('# Heading #')
+        expect(metadata[:headers].length).to eq(1)
+      end
+      it 'works with nil options' do
+        html = '<html><head><title>Test</title></head><body><p>Content</p></body></html>'
+        result = described_class.convert_with_metadata(html, nil, nil)
+        expect(result).to be_an(Array)
+        expect(result.length).to eq(2)
+      end
+    end
+    context 'when extracting structured data' do
+      it 'extracts JSON-LD blocks' do
+        html = <<~HTML
+          <html>
+            <head>
+              <script type="application/ld+json">
+                {"@context":"https://schema.org","@type":"Article","headline":"Test"}
+              </script>
+            </head>
+            <body><p>Content</p></body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        # Structured data extraction may vary by implementation
+        expect(metadata[:structured_data]).to be_an(Array)
+      end
+    end
+    context 'with edge cases' do
+      it 'handles empty HTML' do
+        html = ''
+        markdown, metadata = described_class.convert_with_metadata(html)
+        expect(markdown).to be_a(String)
+        expect(metadata).to be_a(Hash)
+      end
+      it 'handles malformed HTML' do
+        html = '<html><head><title>Unclosed'
+        markdown, metadata = described_class.convert_with_metadata(html)
+        expect(markdown).to be_a(String)
+        expect(metadata).to be_a(Hash)
+      end
+      it 'handles special characters in metadata' do
+        html = '<html><head><title>Title with "quotes" & <brackets></title></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata[:document][:title]).to be_a(String)
+      end
+      it 'handles whitespace in metadata' do
+        html = '<html><head><title>  Title with   spaces  </title></head><body><p>Content</p></body></html>'
+        _, metadata = described_class.convert_with_metadata(html)
+        # Whitespace may be normalized
+        expect(metadata[:document][:title]).to match(/Title.*spaces/)
+      end
+      it 'handles multiple values for same metadata key' do
+        html = <<~HTML
+          <html>
+            <head>
+              <meta name="author" content="Author 1">
+              <meta name="author" content="Author 2">
+            </head>
+            <body><p>Content</p></body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        # Last value typically wins, but implementation may vary
+        expect(metadata[:document][:author]).to be_a(String)
+      end
+    end
+    context 'when returning value structure' do
+      it 'returns proper metadata hash structure' do
+        html = <<~HTML
+          <html>
+            <head><title>Test</title><base href="https://example.com"></head>
+            <body><h1>H1</h1><a href="link">Link</a><img src="img.jpg"></body>
+          </html>
+        HTML
+        _, metadata = described_class.convert_with_metadata(html)
+        expect(metadata).to include(
+          :document,
+          :headers,
+          :links,
+          :images,
+          :structured_data
+        )
+        expect(metadata[:document]).to include(
+          :title,
+          :description,
+          :keywords,
+          :author,
+          :canonical_url,
+          :base_href,
+          :language,
+          :text_direction,
+          :open_graph,
+          :twitter_card,
+          :meta_tags
+        )
+      end
+    end
+  end
+end

metadata CHANGED Viewed

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: html-to-markdown
 version: !ruby/object:Gem::Version
-  version: 2.12.0
+  version: 2.13.0
 platform: ruby
 authors:
 - Na'aman Hirschfeld
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2025-12-08 00:00:00.000000000 Z
+date: 2025-12-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rb_sys
@@ -46,6 +46,7 @@ files:
 - ".rubocop.yml"
 - Gemfile
 - Gemfile.lock
+- METADATA.md
 - README.md
 - Rakefile
 - Steepfile
@@ -67,6 +68,7 @@ files:
 - sig/open3.rbs
 - spec/cli_proxy_spec.rb
 - spec/convert_spec.rb
+- spec/metadata_extraction_spec.rb
 - spec/spec_helper.rb
 homepage: https://github.com/Goldziher/html-to-markdown
 licenses: