kreuzberg 4.0.0.pre.rc.29 → 4.0.0.rc1

data/README.md

@@ -1,781 +1,421 @@
-# Ruby
-
-<div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;">
-  <!-- Language Bindings -->
-  <a href="https://crates.io/crates/kreuzberg">
-    <img src="https://img.shields.io/crates/v/kreuzberg?label=Rust&color=007ec6" alt="Rust">
-  </a>
-  <a href="https://hex.pm/packages/kreuzberg">
-    <img src="https://img.shields.io/hexpm/v/kreuzberg?label=Elixir&color=007ec6" alt="Elixir">
-  </a>
-  <a href="https://pypi.org/project/kreuzberg/">
-    <img src="https://img.shields.io/pypi/v/kreuzberg?label=Python&color=007ec6" alt="Python">
-  </a>
-  <a href="https://www.npmjs.com/package/@kreuzberg/node">
-    <img src="https://img.shields.io/npm/v/@kreuzberg/node?label=Node.js&color=007ec6" alt="Node.js">
-  </a>
-  <a href="https://www.npmjs.com/package/@kreuzberg/wasm">
-    <img src="https://img.shields.io/npm/v/@kreuzberg/wasm?label=WASM&color=007ec6" alt="WASM">
-  </a>
-
-<a href="https://central.sonatype.com/artifact/dev.kreuzberg/kreuzberg">
-    <img src="https://img.shields.io/maven-central/v/dev.kreuzberg/kreuzberg?label=Java&color=007ec6" alt="Java">
-  </a>
-  <a href="https://github.com/kreuzberg-dev/kreuzberg/releases">
-    <img src="https://img.shields.io/github/v/tag/kreuzberg-dev/kreuzberg?label=Go&color=007ec6&filter=v4.0.0-*" alt="Go">
-  </a>
-  <a href="https://www.nuget.org/packages/Kreuzberg/">
-    <img src="https://img.shields.io/nuget/v/Kreuzberg?label=C%23&color=007ec6" alt="C#">
-  </a>
-  <a href="https://packagist.org/packages/kreuzberg/kreuzberg">
-    <img src="https://img.shields.io/packagist/v/kreuzberg/kreuzberg?label=PHP&color=007ec6" alt="PHP">
-  </a>
-  <a href="https://rubygems.org/gems/kreuzberg">
-    <img src="https://img.shields.io/gem/v/kreuzberg?label=Ruby&color=007ec6" alt="Ruby">
-  </a>
-
-<!-- Project Info -->
-
-<a href="https://github.com/kreuzberg-dev/kreuzberg/blob/main/LICENSE">
-    <img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="License">
-  </a>
-  <a href="https://docs.kreuzberg.dev">
-    <img src="https://img.shields.io/badge/docs-kreuzberg.dev-blue" alt="Documentation">
-  </a>
-</div>
-
-<img width="1128" height="191" alt="Banner2" src="https://github.com/user-attachments/assets/419fc06c-8313-4324-b159-4b4d3cfce5c0" />
-
-<div align="center" style="margin-top: 20px;">
-  <a href="https://discord.gg/pXxagNK2zN">
-      <img height="22" src="https://img.shields.io/badge/Discord-Join%20our%20community-7289da?logo=discord&logoColor=white" alt="Discord">
-  </a>
-</div>
-
-Extract text, tables, images, and metadata from 56 file formats including PDF, Office documents, and images. Ruby bindings with idiomatic Ruby API and native performance.
-
-> **Version 4.0.0 Release Candidate**
-> Kreuzberg v4.0.0 is in **Release Candidate** stage. Bugs and breaking changes are expected.
-> This is a pre-release version. Please test the library and [report any issues](https://github.com/kreuzberg-dev/kreuzberg/issues) you encounter.
+# Kreuzberg for Ruby
 
-## Installation
+[![RubyGems](https://img.shields.io/gem/v/kreuzberg)](https://rubygems.org/gems/kreuzberg)
+[![Crates.io](https://img.shields.io/crates/v/kreuzberg)](https://crates.io/crates/kreuzberg)
+[![PyPI](https://img.shields.io/pypi/v/kreuzberg)](https://pypi.org/project/kreuzberg/)
+[![npm](https://img.shields.io/npm/v/@goldziher/kreuzberg)](https://www.npmjs.com/package/@goldziher/kreuzberg)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Documentation](https://img.shields.io/badge/docs-kreuzberg.dev-blue)](https://kreuzberg.dev)
 
-### Package Installation
+High-performance document intelligence for Ruby, powered by Rust.
 
-Install via one of the supported package managers:
+Extract text, tables, images, and metadata from 30+ file formats including PDF, DOCX, PPTX, XLSX, images, and more.
 
-**gem:**
+> **🚀 Version 4.0.0 Release Candidate**
+> This is a pre-release version. We invite you to test the library and [report any issues](https://github.com/Goldziher/kreuzberg/issues) you encounter.
 
-```bash
-gem install kreuzberg
-```
-
-**Bundler:**
-
-```ruby
-gem 'kreuzberg'
-```
+## Features
 
-### System Requirements
+- **30+ File Formats**: PDF, DOCX, PPTX, XLSX, images, HTML, Markdown, XML, JSON, and more
+- **OCR Support**: Built-in Tesseract OCR for scanned documents and images
+- **High Performance**: Rust-powered extraction for native-level performance
+- **Table Extraction**: Extract structured tables from documents
+- **Language Detection**: Automatic language detection for extracted text
+- **Text Chunking**: Split long documents into manageable chunks
+- **Caching**: Built-in result caching for faster repeated extractions
+- **Type-Safe**: Comprehensive typed configuration and result objects
 
-- **Ruby 2.7+** required
-- Optional: [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) version 1.22.x for embeddings support
-- Optional: [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) for OCR functionality
+## Requirements
 
-### Platform Support
+- Ruby 3.2 or higher
+- Rust toolchain (for building from source)
 
-Precompiled native extensions are available for the following platforms, providing instant installation without compilation:
+### Optional System Dependencies
 
-- ✅ Linux x86_64
-- ✅ Linux aarch64 (ARM64)
-- ✅ macOS aarch64 (Apple Silicon)
+- **Tesseract**: For OCR functionality
+  - macOS: `brew install tesseract`
+  - Ubuntu: `sudo apt-get install tesseract-ocr`
+  - Windows: Download from [GitHub](https://github.com/tesseract-ocr/tesseract)
 
-On these platforms, no C compiler or Rust toolchain is required for installation.
+- **LibreOffice**: For legacy MS Office formats (.doc, .ppt)
+  - macOS: `brew install libreoffice`
+  - Ubuntu: `sudo apt-get install libreoffice`
 
-## Quick Start
+- **Pandoc**: For advanced document conversion
+  - macOS: `brew install pandoc`
+  - Ubuntu: `sudo apt-get install pandoc`
 
-### Basic Extraction
+## Installation
 
-Extract text, metadata, and structure from any supported document format:
+Add to your Gemfile:
 
 ```ruby
-require 'kreuzberg'
-
-result = Kreuzberg.extract_file_sync(path: 'document.pdf')
-
-puts "Content:"
-puts result.content
-
-puts "\nMetadata:"
-puts "Title: #{result.metadata&.dig('title')}"
-puts "Author: #{result.metadata&.dig('author')}"
-
-puts "\nTables found: #{result.tables.length}"
-puts "Images found: #{result.images.length}"
+gem 'kreuzberg'
 ```
 
-### Common Use Cases
-
-#### Extract with Custom Configuration
+Then run:
 
-Most use cases benefit from configuration to control extraction behavior:
-
-**With OCR (for scanned documents):**
-
-```ruby
-require 'kreuzberg'
-
-ocr_config = Kreuzberg::Config::OCR.new(
-  backend: 'tesseract',
-  language: 'eng'
-)
+```bash
+bundle install
+```
 
-config = Kreuzberg::Config::Extraction.new(ocr: ocr_config)
-result = Kreuzberg.extract_file_sync(path: 'scanned.pdf', config: config)
+Or install directly:
 
-puts "Extracted text from scanned document:"
-puts result.content
-puts "Used OCR backend: tesseract"
+```bash
+gem install kreuzberg
 ```
 
-#### Table Extraction
-
-See [Table Extraction Guide](https://kreuzberg.dev/features/table-extraction/) for detailed examples.
+## Quick Start
 
-#### Processing Multiple Files
+### Basic Extraction
 
 ```ruby
 require 'kreuzberg'
 
-puts "Kreuzberg version: #{Kreuzberg::VERSION}"
-puts "FFI bindings loaded successfully"
-
-result = Kreuzberg.extract_file_sync(path: 'sample.pdf')
-puts "Installation verified! Extracted #{result.content.length} characters"
+# Extract from a file
+result = Kreuzberg.extract_file_sync("document.pdf")
+puts result.content
+puts "MIME type: #{result.mime_type}"
 ```
 
-#### Async Processing
-
-For non-blocking document processing:
+### With Configuration
 
 ```ruby
-require 'kreuzberg'
-
+# Create configuration
 config = Kreuzberg::Config::Extraction.new(
   use_cache: true,
-  enable_quality_processing: true
+  force_ocr: false
 )
 
-result = Kreuzberg.extract_file_sync(path: 'contract.pdf', config: config)
-
-puts "Extracted #{result.content.length} characters"
-puts "Quality score: #{result.metadata&.dig('quality_score')}"
-puts "Processing time: #{result.metadata&.dig('processing_time')}ms"
+result = Kreuzberg.extract_file_sync("document.pdf", config: config)
 ```
 
-### Next Steps
-
-- **[Installation Guide](https://kreuzberg.dev/getting-started/installation/)** - Platform-specific setup
-- **[API Documentation](https://kreuzberg.dev/api/)** - Complete API reference
-- **[Examples & Guides](https://kreuzberg.dev/guides/)** - Full code examples and usage guides
-- **[Configuration Guide](https://kreuzberg.dev/configuration/)** - Advanced configuration options
-- **[Troubleshooting](https://kreuzberg.dev/troubleshooting/)** - Common issues and solutions
-
-## Features
-
-### Supported File Formats (56+)
-
-56 file formats across 8 major categories with intelligent format detection and comprehensive metadata extraction.
-
-#### Office Documents
-
-| Category | Formats | Capabilities |
-|----------|---------|--------------|
-| **Word Processing** | `.docx`, `.odt` | Full text, tables, images, metadata, styles |
-| **Spreadsheets** | `.xlsx`, `.xlsm`, `.xlsb`, `.xls`, `.xla`, `.xlam`, `.xltm`, `.ods` | Sheet data, formulas, cell metadata, charts |
-| **Presentations** | `.pptx`, `.ppt`, `.ppsx` | Slides, speaker notes, images, metadata |
-| **PDF** | `.pdf` | Text, tables, images, metadata, OCR support |
-| **eBooks** | `.epub`, `.fb2` | Chapters, metadata, embedded resources |
-
-#### Images (OCR-Enabled)
-
-| Category | Formats | Features |
-|----------|---------|----------|
-| **Raster** | `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.bmp`, `.tiff`, `.tif` | OCR, table detection, EXIF metadata, dimensions, color space |
-| **Advanced** | `.jp2`, `.jpx`, `.jpm`, `.mj2`, `.pnm`, `.pbm`, `.pgm`, `.ppm` | OCR, table detection, format-specific metadata |
-| **Vector** | `.svg` | DOM parsing, embedded text, graphics metadata |
-
-#### Web & Data
-
-| Category | Formats | Features |
-|----------|---------|----------|
-| **Markup** | `.html`, `.htm`, `.xhtml`, `.xml`, `.svg` | DOM parsing, metadata (Open Graph, Twitter Card), link extraction |
-| **Structured Data** | `.json`, `.yaml`, `.yml`, `.toml`, `.csv`, `.tsv` | Schema detection, nested structures, validation |
-| **Text & Markdown** | `.txt`, `.md`, `.markdown`, `.rst`, `.org`, `.rtf` | CommonMark, GFM, reStructuredText, Org Mode |
-
-#### Email & Archives
-
-| Category | Formats | Features |
-|----------|---------|----------|
-| **Email** | `.eml`, `.msg` | Headers, body (HTML/plain), attachments, threading |
-| **Archives** | `.zip`, `.tar`, `.tgz`, `.gz`, `.7z` | File listing, nested archives, metadata |
-
-#### Academic & Scientific
-
-| Category | Formats | Features |
-|----------|---------|----------|
-| **Citations** | `.bib`, `.biblatex`, `.ris`, `.enw`, `.csl` | Bibliography parsing, citation extraction |
-| **Scientific** | `.tex`, `.latex`, `.typst`, `.jats`, `.ipynb`, `.docbook` | LaTeX, Jupyter notebooks, PubMed JATS |
-| **Documentation** | `.opml`, `.pod`, `.mdoc`, `.troff` | Technical documentation formats |
-
-**[Complete Format Reference](https://kreuzberg.dev/reference/formats/)**
-
-### Key Capabilities
-
-- **Text Extraction** - Extract all text content with position and formatting information
-
-- **Metadata Extraction** - Retrieve document properties, creation date, author, etc.
-
-- **Table Extraction** - Parse tables with structure and cell content preservation
-
-- **Image Extraction** - Extract embedded images and render page previews
-
-- **OCR Support** - Integrate multiple OCR backends for scanned documents
-
-- **Async/Await** - Non-blocking document processing with concurrent operations
-
-- **Plugin System** - Extensible post-processing for custom text transformation
-
-- **Embeddings** - Generate vector embeddings using ONNX Runtime models
-
-- **Batch Processing** - Efficiently process multiple documents in parallel
-
-- **Memory Efficient** - Stream large files without loading entirely into memory
-
-- **Language Detection** - Detect and support multiple languages in documents
-
-- **Configuration** - Fine-grained control over extraction behavior
-
-### Performance Characteristics
-
-| Format | Speed | Memory | Notes |
-|--------|-------|--------|-------|
-| **PDF (text)** | 10-100 MB/s | ~50MB per doc | Fastest extraction |
-| **Office docs** | 20-200 MB/s | ~100MB per doc | DOCX, XLSX, PPTX |
-| **Images (OCR)** | 1-5 MB/s | Variable | Depends on OCR backend |
-| **Archives** | 5-50 MB/s | ~200MB per doc | ZIP, TAR, etc. |
-| **Web formats** | 50-200 MB/s | Streaming | HTML, XML, JSON |
-
-## OCR Support
-
-Kreuzberg supports multiple OCR backends for extracting text from scanned documents and images:
-
-- **Tesseract**
-
-### OCR Configuration Example
+### With OCR
 
 ```ruby
-require 'kreuzberg'
-
+# Configure OCR
 ocr_config = Kreuzberg::Config::OCR.new(
-  backend: 'tesseract',
-  language: 'eng'
+  backend: "tesseract",
+  language: "eng",
+  preprocessing: true
 )
 
 config = Kreuzberg::Config::Extraction.new(ocr: ocr_config)
-result = Kreuzberg.extract_file_sync(path: 'scanned.pdf', config: config)
-
-puts "Extracted text from scanned document:"
-puts result.content
-puts "Used OCR backend: tesseract"
+result = Kreuzberg.extract_file_sync("scanned.pdf", config: config)
 ```
 
-## Async Support
-
-This binding provides full async/await support for non-blocking document processing:
+### Extract from Bytes
 
 ```ruby
-require 'kreuzberg'
-
-config = Kreuzberg::Config::Extraction.new(
-  use_cache: true,
-  enable_quality_processing: true
-)
-
-result = Kreuzberg.extract_file_sync(path: 'contract.pdf', config: config)
-
-puts "Extracted #{result.content.length} characters"
-puts "Quality score: #{result.metadata&.dig('quality_score')}"
-puts "Processing time: #{result.metadata&.dig('processing_time')}ms"
+data = File.binread("document.pdf")
+result = Kreuzberg.extract_bytes_sync(data, "application/pdf")
+puts result.content
 ```
 
-## Plugin System
-
-Kreuzberg supports extensible post-processing plugins for custom text transformation and filtering.
-
-For detailed plugin documentation, visit [Plugin System Guide](https://kreuzberg.dev/plugins/).
-
-## Embeddings Support
-
-Generate vector embeddings for extracted text using the built-in ONNX Runtime support. Requires ONNX Runtime installation.
-
-**[Embeddings Guide](https://kreuzberg.dev/features/#embeddings)**
+### Batch Processing
 
-## Advanced Examples
+```ruby
+paths = ["doc1.pdf", "doc2.docx", "doc3.xlsx"]
+results = Kreuzberg.batch_extract_files_sync(paths)
 
-### Embeddings with Model Configuration
+results.each do |result|
+  puts "Content: #{result.content[0..100]}"
+  puts "MIME: #{result.mime_type}"
+end
+```
 
-Generate embeddings for document chunks with custom model configuration:
+### Structured Results (Chunks & Images)
 
 ```ruby
-require 'kreuzberg'
-
-# Configure embedding model with custom parameters
-embedding_config = Kreuzberg::Config::Embedding.new(
-  model: { type: :preset, name: 'balanced' },
-  normalize: true,
-  batch_size: 32,
-  show_download_progress: false
-)
+result = Kreuzberg.extract_file_sync("long-report.pdf", config: {
+  chunking: { max_chars: 750 },
+  image_extraction: { extract_images: true }
+})
 
-# Enable chunking with embeddings
-chunking_config = Kreuzberg::Config::Chunking.new(
-  max_chars: 1024,
-  max_overlap: 256,
-  embedding: embedding_config
-)
+result.chunks&.each do |chunk|
+  puts "[#{chunk.chunk_index + 1}/#{chunk.total_chunks}] #{chunk.content[0..80]}"
+end
 
-config = Kreuzberg::Config::Extraction.new(chunking: chunking_config)
-result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: config)
-
-# Access chunks with embeddings
-result.chunks.each_with_index do |chunk, idx|
-  puts "Chunk #{idx}:"
-  puts "  Content: #{chunk.content[0..50]}..."
-  puts "  Tokens: #{chunk.token_count}"
-  puts "  Pages: #{chunk.first_page}-#{chunk.last_page}"
-  if chunk.embedding
-    puts "  Embedding dimensions: #{chunk.embedding.length}"
+result.images&.each do |image|
+  File.binwrite("image-#{image.image_index}.#{image.format}", image.data)
+  if image.ocr_result
+    puts "Embedded OCR content: #{image.ocr_result.content[0..60]}"
   end
 end
 ```
 
-### Keywords Extraction (YAKE and RAKE)
+## Configuration
 
-Extract keywords using YAKE and RAKE algorithms:
+### Load From File
 
 ```ruby
-require 'kreuzberg'
+config = Kreuzberg::Config::Extraction.from_file("config.toml")
+result = Kreuzberg.extract_file_sync("report.pdf", config: config)
+```
 
-# Extract keywords using YAKE algorithm
-yake_config = Kreuzberg::Config::Keywords.new(
-  algorithm: 'yake',
-  max_keywords: 10,
-  min_score: 0.1,
-  yake_params: Kreuzberg::Config::KeywordYakeParams.new(window_size: 3)
-)
+### Extraction Configuration
 
-config = Kreuzberg::Config::Extraction.new(keywords: yake_config)
-result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: config)
-
-# Extract keywords using RAKE algorithm
-rake_config = Kreuzberg::Config::Keywords.new(
-  algorithm: 'rake',
-  max_keywords: 15,
-  language: 'english',
-  rake_params: Kreuzberg::Config::KeywordRakeParams.new(
-    min_word_length: 3,
-    max_words_per_phrase: 5
-  )
+```ruby
+config = Kreuzberg::Config::Extraction.new(
+  use_cache: true,                      # Enable result caching
+  enable_quality_processing: false,     # Enable text quality processing
+  force_ocr: false                      # Force OCR even for digital PDFs
 )
-
-config = Kreuzberg::Config::Extraction.new(keywords: rake_config)
-result = Kreuzberg.extract_file_sync(path: 'report.docx', config: config)
-
-puts "Keywords extracted for document"
 ```
 
-### Pages Extraction with PageConfig
-
-Extract and organize content by pages:
+### OCR Configuration
 
 ```ruby
-require 'kreuzberg'
-
-# Enable per-page extraction with markers
-page_config = Kreuzberg::Config::PageConfig.new(
-  extract_pages: true,
-  insert_page_markers: true,
-  marker_format: "\n\n=== PAGE {page_num} ===\n\n"
+ocr = Kreuzberg::Config::OCR.new(
+  backend: "tesseract",           # OCR backend (tesseract, easyocr, paddleocr)
+  language: "eng",                # Language code (eng, deu, fra, etc.)
+  tesseract_config: {
+    psm: 6,
+    enable_table_detection: true,
+    preprocessing: Kreuzberg::Config::ImagePreprocessing.new(auto_rotate: true).to_h
+  }
 )
 
-config = Kreuzberg::Config::Extraction.new(pages: page_config)
-result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: config)
-
-# Access extracted pages
-if result.pages
-  result.pages.each do |page|
-    puts "Page #{page.page_number}:"
-    puts "  Content length: #{page.content.length}"
-    puts "  Tables: #{page.tables.length}"
-    puts "  Images: #{page.images.length}"
-  end
-end
-
-puts "Total pages: #{result.page_count}"
+config = Kreuzberg::Config::Extraction.new(ocr: ocr)
 ```
 
-### Custom PostProcessor Implementation
-
-Create and register custom post-processors for text transformation:
+### Chunking Configuration
 
 ```ruby
-require 'kreuzberg'
-
-# Define a custom post-processor class
-class MarkdownEnhancerPostProcessor
-  include Kreuzberg::PostProcessorProtocol
-
-  def call(result)
-    # Enhance extracted content with markdown formatting
-    enhanced = result.dup
-
-    if enhanced['content']
-      # Add markdown headers for detected structure
-      enhanced['content'] = enhance_with_markdown(enhanced['content'])
-    end
-
-    enhanced
-  end
+chunking = Kreuzberg::Config::Chunking.new(
+  enabled: true,
+  chunk_size: 1000,       # Characters per chunk
+  chunk_overlap: 200,     # Overlap between chunks
+  embedding: {
+    model: { type: :preset, name: "balanced" },
+    normalize: true
+  }
+)
 
-  private
+config = Kreuzberg::Config::Extraction.new(chunking: chunking)
+result = Kreuzberg.extract_file_sync("long_document.pdf", config: config)
 
-  def enhance_with_markdown(content)
-    # Example: Convert section breaks to markdown headers
-    content
-      .split("\n\n")
-      .map { |paragraph| paragraph.length > 100 ? "## #{paragraph[0..30]}...\n\n#{paragraph}" : paragraph }
-      .join("\n\n")
-  end
+result.chunks.each do |chunk|
+  puts "Chunk: #{chunk.content}"
+  puts "Tokens: #{chunk.token_count}"
 end
-
-# Use custom post-processor in configuration
-processor = MarkdownEnhancerPostProcessor.new
-postprocessor_config = Kreuzberg::Config::PostProcessor.new(enabled: true)
-config = Kreuzberg::Config::Extraction.new(postprocessor: postprocessor_config)
-
-result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: config)
-puts result.content
 ```
 
-### Custom Validator Implementation
-
-Create and register validators to ensure extraction quality:
+### HTML Conversion Options
 
 ```ruby
-require 'kreuzberg'
-
-# Define a custom validator class
-class ContentQualityValidator
-  include Kreuzberg::ValidatorProtocol
-
-  MIN_CONTENT_LENGTH = 100
-  MIN_METADATA_FIELDS = 2
-
-  def call(result)
-    # Validate extracted content meets quality standards
-    content = result['content'].to_s
-    metadata = result['metadata'].to_h
-
-    if content.length < MIN_CONTENT_LENGTH
-      raise Kreuzberg::Errors::ValidationError,
-            "Content too short: #{content.length} bytes (minimum: #{MIN_CONTENT_LENGTH})"
-    end
-
-    if metadata.length < MIN_METADATA_FIELDS
-      raise Kreuzberg::Errors::ValidationError,
-            "Insufficient metadata: #{metadata.length} fields (minimum: #{MIN_METADATA_FIELDS})"
-    end
-
-    # Validation passed
-    nil
-  end
-end
-
-# Use validator in extraction workflow
-validator = ContentQualityValidator.new
-config = Kreuzberg::Config::Extraction.new(enable_quality_processing: true)
+html_options = Kreuzberg::Config::HtmlOptions.new(
+  heading_style: :atx_closed,
+  wrap: true,
+  wrap_width: 100,
+  preprocessing: { enabled: true, preset: :standard }
+)
 
-begin
-  result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: config)
-  validator.call(result.to_h)
-  puts "Extraction passed quality validation"
-rescue Kreuzberg::Errors::ValidationError => e
-  puts "Validation failed: #{e.message}"
-end
+config = Kreuzberg::Config::Extraction.new(html_options: html_options)
+result = Kreuzberg.extract_file_sync("page.html", config: config)
 ```
 
-### Config File Loading (from_file and discover)
-
-Load configuration from TOML, YAML, or JSON files:
+### Keyword Extraction
 
 ```ruby
-require 'kreuzberg'
+keywords = Kreuzberg::Config::Keywords.new(
+  algorithm: :yake,
+  max_keywords: 8,
+  min_score: 0.2,
+  ngram_range: [1, 3]
+)
 
-# Load configuration from a specific file
-# Supports: .toml, .yaml/.yml, .json
-config = Kreuzberg::Config::Extraction.from_file('config/kreuzberg.toml')
-
-# Example: config/kreuzberg.toml
-# use_cache = true
-# force_ocr = false
-# enable_quality_processing = true
-#
-# [chunking]
-# max_chars = 1024
-# max_overlap = 256
-#
-# [ocr]
-# backend = "tesseract"
-# language = "eng"
-#
-# [language_detection]
-# enabled = true
-# min_confidence = 0.7
-
-result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: config)
-puts "Extracted with config from file"
-
-# Auto-discover configuration in project hierarchy
-discovered_config = Kreuzberg::Config::Extraction.discover
-if discovered_config
-  puts "Found configuration at project root"
-  result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: discovered_config)
-else
-  puts "No configuration file found, using defaults"
-  result = Kreuzberg.extract_file_sync(path: 'document.pdf')
-end
+config = Kreuzberg::Config::Extraction.new(keywords: keywords)
+result = Kreuzberg.extract_file_sync("research.pdf", config: config)
 ```
 
-### Fiber-Based Async Patterns
-
-Use Ruby Fibers for efficient async extraction workflows:
+### Language Detection
 
 ```ruby
-require 'kreuzberg'
-
-# Create async extraction workflow using Fibers
-def extract_documents_async(file_paths)
-  fibers = file_paths.map do |path|
-    Fiber.new do
-      config = Kreuzberg::Config::Extraction.new(
-        use_cache: true,
-        enable_quality_processing: true
-      )
-
-      # Extract asynchronously
-      result = Kreuzberg.extract_file(path: path, config: config)
-
-      {
-        path: path,
-        content_length: result.content.length,
-        tables: result.tables.length,
-        languages: result.detected_languages
-      }
-    end
-  end
+lang_detection = Kreuzberg::Config::LanguageDetection.new(
+  enabled: true,
+  min_confidence: 0.8,
+  detect_multiple: true
+)
 
-  # Resume all fibers and collect results
-  results = fibers.map do |fiber|
-    Fiber.yield fiber.resume if fiber.alive?
-  end
+config = Kreuzberg::Config::Extraction.new(language_detection: lang_detection)
+result = Kreuzberg.extract_file_sync("multilingual.pdf", config: config)
 
-  results.compact
+result.detected_languages&.each do |lang|
+  puts "Language: #{lang.lang}, Confidence: #{lang.confidence}"
 end
+```
 
-# Usage
-file_paths = ['document1.pdf', 'document2.docx', 'document3.xlsx']
-results = extract_documents_async(file_paths)
+### PDF Options
 
-results.each do |result|
-  puts "#{result[:path]}: #{result[:content_length]} characters"
-end
-```
+```ruby
+pdf_options = Kreuzberg::Config::PDF.new(
+  extract_images: true,
+  image_min_size: 10000,    # Minimum image size in bytes
+  password: "secret"        # PDF password
+)
 
-### Table Extraction Detailed Usage
+config = Kreuzberg::Config::Extraction.new(pdf_options: pdf_options)
+```
 
-Extract and access table structure and cell data:
+## Working with Results
 
 ```ruby
-require 'kreuzberg'
+result = Kreuzberg.extract_file_sync("invoice.pdf")
 
-# Configure table extraction
-config = Kreuzberg::Config::Extraction.new(
-  pdf_options: Kreuzberg::Config::PDF.new(extract_images: true)
-)
+# Access extracted text
+puts result.content
+
+# Access MIME type
+puts result.mime_type
 
-result = Kreuzberg.extract_file_sync(path: 'spreadsheet.pdf', config: config)
+# Access metadata
+puts result.metadata.inspect
 
 # Access extracted tables
-result.tables.each_with_index do |table, table_idx|
-  puts "Table #{table_idx} (Page #{table.page_number}):"
-
-  # Access table cells (2D array)
-  table.cells.each_with_index do |row, row_idx|
-    puts "  Row #{row_idx}:"
-    row.each_with_index do |cell, col_idx|
-      puts "    [#{col_idx}] #{cell}"
-    end
+result.tables.each do |table|
+  puts "Headers: #{table.headers.join(', ')}"
+  table.rows.each do |row|
+    puts row.join(', ')
   end
+end
 
-  # Access markdown representation
-  puts "\nMarkdown format:"
-  puts table.markdown
+# Access text chunks and metadata
+result.chunks&.each do |chunk|
+  puts "Chunk #{chunk.chunk_index + 1}/#{chunk.total_chunks}"
+  puts "Chars: #{chunk.char_start}-#{chunk.char_end}"
+  puts "Embedding length: #{chunk.embedding&.length}"
 end
 
-# Extract tables from specific pages
-page_config = Kreuzberg::Config::PageConfig.new(extract_pages: true)
-config = Kreuzberg::Config::Extraction.new(pages: page_config)
-result = Kreuzberg.extract_file_sync(path: 'data.xlsx', config: config)
-
-if result.pages
-  result.pages.each do |page|
-    page.tables.each do |table|
-      puts "Table on page #{page.page_number}:"
-      puts "  Dimensions: #{table.cells.length} rows x #{table.cells.first&.length || 0} columns"
-    end
-  end
+# Access extracted images
+result.images&.each do |image|
+  File.binwrite("image-\#{image.image_index}.#{image.format}", image.data)
+  puts "Image #{image.image_index} on page #{image.page_number}"
 end
-```
 
-### Image Extraction and Saving
+# Convert to hash
+hash = result.to_h
 
-Extract images and save them to disk:
+# Convert to JSON
+json = result.to_json
+```
 
-```ruby
-require 'kreuzberg'
+## CLI Usage
 
-# Configure image extraction with high DPI
-image_config = Kreuzberg::Config::ImageExtraction.new(
-  extract_images: true,
-  target_dpi: 300,
-  max_image_dimension: 2000,
-  auto_adjust_dpi: true
-)
+Kreuzberg provides a Ruby wrapper for the CLI:
 
-config = Kreuzberg::Config::Extraction.new(image_extraction: image_config)
-result = Kreuzberg.extract_file_sync(path: 'document.pdf', config: config)
+```ruby
+# Extract content
+output = Kreuzberg::CLI.extract("document.pdf", output: "text")
 
-# Save extracted images
-output_dir = 'extracted_images'
-Dir.mkdir(output_dir) unless Dir.exist?(output_dir)
+# Detect MIME type
+mime_type = Kreuzberg::CLI.detect("document.pdf")
 
-result.images.each_with_index do |image, idx|
-  # Generate filename
-  filename = "image_p#{image.page_number}_#{image.image_index}.#{image.format}"
-  filepath = File.join(output_dir, filename)
+# Get version
+version = Kreuzberg::CLI.version
+```
 
-  # Save image data
-  File.write(filepath, image.data, mode: 'wb')
+## API Server
 
-  puts "Saved: #{filename}"
-  puts "  Page: #{image.page_number}"
-  puts "  Format: #{image.format}"
-  puts "  Dimensions: #{image.width}x#{image.height}"
-  puts "  Colorspace: #{image.colorspace}"
+Start an API server (requires kreuzberg CLI):
 
-  # Process OCR result if available
-  if image.ocr_result
-    puts "  OCR Text: #{image.ocr_result['text'][0..50]}..."
-  end
+```ruby
+Kreuzberg::APIProxy.run(port: 8000) do |server|
+  # Server runs in background
+  # Make HTTP requests to http://localhost:8000
 end
 ```
 
-### Language Detection Configuration
+## MCP Server
 
-Configure and use language detection:
+Start a Model Context Protocol server for Claude Desktop:
 
 ```ruby
-require 'kreuzberg'
-
-# Enable language detection with confidence threshold
-lang_detection_config = Kreuzberg::Config::LanguageDetection.new(
-  enabled: true,
-  min_confidence: 0.8,
-  detect_multiple: true
-)
-
-config = Kreuzberg::Config::Extraction.new(
-  language_detection: lang_detection_config
-)
-
-result = Kreuzberg.extract_file_sync(path: 'multilingual.pdf', config: config)
+server = Kreuzberg::MCPProxy::Server.new(transport: 'stdio')
+server.start
 
-# Access detected languages
-puts "Primary language: #{result.detected_language}"
-puts "All detected languages: #{result.detected_languages.join(', ')}"
-
-# Access language from metadata
-if result.metadata.is_a?(Hash)
-  puts "Language from metadata: #{result.metadata['language']}"
-end
+# Use with Claude Desktop integration
+```
 
-# Combine with keyword extraction for specific language
-keywords_config = Kreuzberg::Config::Keywords.new(
-  algorithm: 'yake',
-  language: 'de',  # German keywords
-  max_keywords: 10
-)
+## Cache Management
 
-config = Kreuzberg::Config::Extraction.new(
-  language_detection: lang_detection_config,
-  keywords: keywords_config
-)
+```ruby
+# Get cache statistics
+stats = Kreuzberg.cache_stats
+puts "Entries: #{stats[:total_entries]}"
+puts "Size: #{stats[:total_size_bytes]} bytes"
 
-result = Kreuzberg.extract_file_sync(path: 'german_document.pdf', config: config)
-puts "Keywords extracted for: #{result.detected_language}"
+# Clear cache
+Kreuzberg.clear_cache
 ```
 
-## Batch Processing
-
-Process multiple documents efficiently:
+## Error Handling
 
 ```ruby
-require 'kreuzberg'
+begin
+  result = Kreuzberg.extract_file_sync("document.pdf")
+rescue Kreuzberg::Errors::ParsingError => e
+  puts "Parsing failed: #{e.message}"
+  puts "Context: #{e.context}"
+rescue Kreuzberg::Errors::OCRError => e
+  puts "OCR failed: #{e.message}"
+rescue Kreuzberg::Errors::MissingDependencyError => e
+  puts "Missing dependency: #{e.dependency}"
+rescue Kreuzberg::Errors::Error => e
+  puts "Kreuzberg error: #{e.message}"
+end
+```
 
-puts "Kreuzberg version: #{Kreuzberg::VERSION}"
-puts "FFI bindings loaded successfully"
+## Supported Formats
 
-result = Kreuzberg.extract_file_sync(path: 'sample.pdf')
-puts "Installation verified! Extracted #{result.content.length} characters"
-```
+- **Documents**: PDF, DOCX, DOC, PPTX, PPT, ODT, ODP
+- **Spreadsheets**: XLSX, XLS, ODS, CSV
+- **Images**: PNG, JPEG, TIFF, BMP, GIF
+- **Web**: HTML, MHTML, Markdown
+- **Data**: JSON, YAML, TOML, XML
+- **Email**: EML, MSG
+- **Archives**: ZIP, TAR, 7Z
+- **Text**: TXT, RTF, MD
 
-## Configuration
+## Performance
 
-For advanced configuration options including language detection, table extraction, OCR settings, and more:
+Kreuzberg's Rust core provides significant performance improvements:
 
-**[Configuration Guide](https://kreuzberg.dev/configuration/)**
+- **PDF extraction**: 10-50x faster than pure Ruby solutions
+- **Batch processing**: Parallel extraction with Tokio async runtime
+- **Memory efficient**: Streaming parsers for large files
+- **Caching**: Automatic result caching for repeated extractions
 
-## Documentation
+## Development
 
-- **[Official Documentation](https://kreuzberg.dev/)**
-- **[API Reference](https://kreuzberg.dev/reference/api-ruby/)**
-- **[Examples & Guides](https://kreuzberg.dev/guides/)**
+```bash
+# Clone the repository
+git clone https://github.com/Goldziher/kreuzberg.git
+cd kreuzberg/packages/ruby
 
-## Troubleshooting
+# Install dependencies
+bundle install
 
-For common issues and solutions, visit [Troubleshooting Guide](https://kreuzberg.dev/troubleshooting/).
+# Build the Rust extension
+bundle exec rake compile
 
-## Contributing
+# Run tests
+bundle exec rspec
 
-Contributions are welcome! See [Contributing Guide](https://github.com/kreuzberg-dev/kreuzberg/blob/main/CONTRIBUTING.md).
+# Run RuboCop
+bundle exec rubocop
+```
 
 ## License
 
-MIT License - see LICENSE file for details.
+MIT License. See [LICENSE](../../LICENSE) for details.
+
+## Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](../../CONTRIBUTING.md) for guidelines.
 
-## Support
+## Links
 
-- **Discord Community**: [Join our Discord](https://discord.gg/pXxagNK2zN)
-- **GitHub Issues**: [Report bugs](https://github.com/kreuzberg-dev/kreuzberg/issues)
-- **Discussions**: [Ask questions](https://github.com/kreuzberg-dev/kreuzberg/discussions)
+- **Documentation**: https://docs.kreuzberg.dev
+- **GitHub**: https://github.com/Goldziher/kreuzberg
+- **Issues**: https://github.com/Goldziher/kreuzberg/issues