@kreuzberg/node 4.0.0-rc.21 → 4.0.0-rc.24

package/README.md

@@ -1,705 +1,516 @@
-# Kreuzberg
+# TypeScript (Node.js)
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;">
+  <!-- Language Bindings -->
+  <a href="https://crates.io/crates/kreuzberg">
+    <img src="https://img.shields.io/crates/v/kreuzberg?label=Rust&color=007ec6" alt="Rust">
+  </a>
+  <a href="https://hex.pm/packages/kreuzberg">
+    <img src="https://img.shields.io/hexpm/v/kreuzberg?label=Elixir&color=007ec6" alt="Elixir">
+  </a>
+  <a href="https://pypi.org/project/kreuzberg/">
+    <img src="https://img.shields.io/pypi/v/kreuzberg?label=Python&color=007ec6" alt="Python">
+  </a>
+  <a href="https://www.npmjs.com/package/@kreuzberg/node">
+    <img src="https://img.shields.io/npm/v/@kreuzberg/node?label=Node.js&color=007ec6" alt="Node.js">
+  </a>
+  <a href="https://www.npmjs.com/package/@kreuzberg/wasm">
+    <img src="https://img.shields.io/npm/v/@kreuzberg/wasm?label=WASM&color=007ec6" alt="WASM">
+  </a>
+
+  <a href="https://central.sonatype.com/artifact/dev.kreuzberg/kreuzberg">
+    <img src="https://img.shields.io/maven-central/v/dev.kreuzberg/kreuzberg?label=Java&color=007ec6" alt="Java">
+  </a>
+  <a href="https://github.com/kreuzberg-dev/kreuzberg/releases">
+    <img src="https://img.shields.io/github/v/tag/kreuzberg-dev/kreuzberg?label=Go&color=007ec6&filter=v4.0.0-*" alt="Go">
+  </a>
+  <a href="https://www.nuget.org/packages/Kreuzberg/">
+    <img src="https://img.shields.io/nuget/v/Kreuzberg?label=C%23&color=007ec6" alt="C#">
+  </a>
+  <a href="https://packagist.org/packages/kreuzberg/kreuzberg">
+    <img src="https://img.shields.io/packagist/v/kreuzberg/kreuzberg?label=PHP&color=007ec6" alt="PHP">
+  </a>
+  <a href="https://rubygems.org/gems/kreuzberg">
+    <img src="https://img.shields.io/gem/v/kreuzberg?label=Ruby&color=007ec6" alt="Ruby">
+  </a>
+
+  <!-- Project Info -->
+  <a href="https://github.com/kreuzberg-dev/kreuzberg/blob/main/LICENSE">
+    <img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="License">
+  </a>
+  <a href="https://docs.kreuzberg.dev">
+    <img src="https://img.shields.io/badge/docs-kreuzberg.dev-blue" alt="Documentation">
+  </a>
+</div>
+
+<img width="1128" height="191" alt="Banner2" src="https://github.com/user-attachments/assets/419fc06c-8313-4324-b159-4b4d3cfce5c0" />
+
+<div align="center" style="margin-top: 20px;">
+  <a href="https://discord.gg/pXxagNK2zN">
+      <img height="22" src="https://img.shields.io/badge/Discord-Join%20our%20community-7289da?logo=discord&logoColor=white" alt="Discord">
+  </a>
+</div>
+
+
+Extract text, tables, images, and metadata from 56 file formats including PDF, Office documents, and images. Native NAPI-RS bindings for Node.js with superior performance, async/await support, and TypeScript type definitions.
 
-[![Rust](https://img.shields.io/crates/v/kreuzberg?label=Rust)](https://crates.io/crates/kreuzberg)
-[![Python](https://img.shields.io/pypi/v/kreuzberg?label=Python)](https://pypi.org/project/kreuzberg/)
-[![TypeScript](https://img.shields.io/npm/v/@kreuzberg/node?label=TypeScript)](https://www.npmjs.com/package/@kreuzberg/node)
-[![WASM](https://img.shields.io/npm/v/@kreuzberg/wasm?label=WASM)](https://www.npmjs.com/package/@kreuzberg/wasm)
-[![Ruby](https://img.shields.io/gem/v/kreuzberg?label=Ruby)](https://rubygems.org/gems/kreuzberg)
-[![Java](https://img.shields.io/maven-central/v/dev.kreuzberg/kreuzberg?label=Java)](https://central.sonatype.com/artifact/dev.kreuzberg/kreuzberg)
-[![Go](https://img.shields.io/github/v/tag/kreuzberg-dev/kreuzberg?label=Go)](https://pkg.go.dev/github.com/kreuzberg-dev/kreuzberg)
-[![C#](https://img.shields.io/nuget/v/Goldziher.Kreuzberg?label=C%23)](https://www.nuget.org/packages/Goldziher.Kreuzberg/)
 
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Documentation](https://img.shields.io/badge/docs-kreuzberg.dev-blue)](https://kreuzberg.dev/)
-[![Discord](https://img.shields.io/badge/Discord-Join%20our%20community-7289da)](https://discord.gg/pXxagNK2zN)
+> **Version 4.0.0 Release Candidate**
+> Kreuzberg v4.0.0 is in **Release Candidate** stage. Bugs and breaking changes are expected.
+> This is a pre-release version. Please test the library and [report any issues](https://github.com/kreuzberg-dev/kreuzberg/issues) you encounter.
 
-High-performance document intelligence for Node.js and TypeScript, powered by Rust.
+## Installation
 
-Extract text, tables, images, and metadata from 56 file formats including PDF, DOCX, PPTX, XLSX, images, and more.
+### Package Installation
 
-> **Recommended for Node.js and Bun** - Native NAPI-RS bindings provide the best performance (2-3x faster than WASM).
->
-> For browser, Deno, or Cloudflare Workers, use [@kreuzberg/wasm](../kreuzberg-wasm/) instead.
 
-> **Version 4.0.0 Release Candidate**
-> This is a pre-release version. We invite you to test the library and [report any issues](https://github.com/kreuzberg-dev/kreuzberg/issues) you encounter.
+Install via one of the supported package managers:
 
-## Features
 
-- **56 File Formats**: PDF, DOCX, PPTX, XLSX, images, HTML, Markdown, XML, JSON, and more
-- **OCR Support**: Built-in Tesseract, EasyOCR, and PaddleOCR backends for scanned documents
-- **Table Extraction**: Advanced table detection and structured data extraction
-- **Native Performance**: 2-3x faster than WASM; 10-50x faster than pure JavaScript
-- **Zero-Copy Operations**: Direct system calls and minimal data copying
-- **Type-Safe**: Full TypeScript definitions for all methods, configurations, and return types
-- **Async/Sync APIs**: Both asynchronous and synchronous extraction methods
-- **Batch Processing**: Process multiple documents in parallel with optimized concurrency
-- **Language Detection**: Automatic language detection for extracted text
-- **Text Chunking**: Split long documents into manageable chunks for LLM processing
-- **Caching**: Built-in result caching for faster repeated extractions
-- **Zero Configuration**: Works out of the box with sensible defaults
 
-## Why Use This Package?
+**npm:**
+```bash
+npm install @kreuzberg/node
+```
 
-Choose `@kreuzberg/node` if you're building with:
 
-- **Node.js 18+** - Native bindings provide direct access to system resources
-- **Bun** - Full compatibility with Bun's Node.js API
-- **Performance-critical applications** - Processing large document batches or real-time extraction
-- **Server-side extraction** - APIs, microservices, document processing pipelines
 
-### Comparison with @kreuzberg/wasm
 
-| Aspect | `@kreuzberg/node` | `@kreuzberg/wasm` |
-|--------|------------------|-------------------|
-| **Performance** | 2-3x faster (native) | Standard baseline |
-| **Environment** | Node.js, Bun | Browser, Deno, Workers, Node.js |
-| **Bundle Size** | 10-15 MB (prebuilt binary) | 2-4 MB (WASM module) |
-| **System Access** | Direct system calls | Sandboxed via WASM |
-| **Best For** | Server-side, batch processing | Client-side, edge computing |
+**pnpm:**
+```bash
+pnpm add @kreuzberg/node
+```
 
-Use `@kreuzberg/wasm` for browser applications, Cloudflare Workers, Deno, or when you need a smaller bundle size.
 
-## Requirements
 
-- Node.js 18 or higher
-- Native bindings are prebuilt for:
-  - macOS (x64, arm64)
-  - Linux (x64, arm64, armv7)
-  - Windows (x64, arm64)
 
-### Optional System Dependencies
+**yarn:**
+```bash
+yarn add @kreuzberg/node
+```
 
-- **ONNX Runtime**: For embeddings functionality
-  - macOS: `brew install onnxruntime`
-  - Ubuntu: `sudo apt-get install libonnxruntime libonnxruntime-dev`
-  - Windows: `scoop install onnxruntime` or download from [GitHub](https://github.com/microsoft/onnxruntime/releases)
 
-- **Tesseract**: For OCR functionality
-  - macOS: `brew install tesseract`
-  - Ubuntu: `sudo apt-get install tesseract-ocr`
-  - Windows: Download from [GitHub](https://github.com/tesseract-ocr/tesseract)
 
-- **LibreOffice**: For legacy MS Office formats (.doc, .ppt)
-  - macOS: `brew install libreoffice`
-  - Ubuntu: `sudo apt-get install libreoffice`
 
-- **Pandoc**: For advanced document conversion
-  - macOS: `brew install pandoc`
-  - Ubuntu: `sudo apt-get install pandoc`
 
-## Installation
+### System Requirements
 
-```bash
-npm install @kreuzberg/node
-```
+- **Node.js 22+** required (NAPI-RS native bindings)
+- Optional: [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) version 1.22.x for embeddings support
+- Optional: [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) for OCR functionality
 
-Or with pnpm:
+- Optional: [LibreOffice](https://www.libreoffice.org/download/download/) for legacy Office formats (DOC, XLS, PPT, RTF, ODT, ODS, ODP)
 
-```bash
-pnpm add @kreuzberg/node
-```
+**Format Support Notes:**
+- Modern Office formats (DOCX, XLSX, PPTX) work without LibreOffice
+- Legacy formats (DOC, XLS, PPT) require LibreOffice installation
+- WASM binding does NOT support LibreOffice formats (use Node.js for full format support)
+
+
+
+### Platform Support
+
+Pre-built binaries available for:
+- macOS (arm64, x64)
+- Linux (x64)
+- Windows (x64)
 
-Or with yarn:
 
-```bash
-yarn add @kreuzberg/node
-```
 
-The package includes prebuilt native binaries for major platforms. No additional build steps required.
 
 ## Quick Start
 
 ### Basic Extraction
 
+Extract text, metadata, and structure from any supported document format:
+
 ```typescript
 import { extractFileSync } from '@kreuzberg/node';
 
-// Synchronous extraction
-const result = extractFileSync('document.pdf');
+const config = {
+	useCache: true,
+	enableQualityProcessing: true,
+};
+
+const result = extractFileSync('document.pdf', null, config);
+
 console.log(result.content);
-console.log(result.metadata);
+console.log(`MIME Type: ${result.mimeType}`);
 ```
 
-### Async Extraction (Recommended)
+
+### Common Use Cases
+
+#### Extract with Custom Configuration
+
+Most use cases benefit from configuration to control extraction behavior:
+
+
+**With OCR (for scanned documents):**
 
 ```typescript
 import { extractFile } from '@kreuzberg/node';
 
-// Asynchronous extraction
-const result = await extractFile('document.pdf');
+const config = {
+	ocr: {
+		backend: 'tesseract',
+		language: 'eng+fra',
+		tesseractConfig: {
+			psm: 3,
+		},
+	},
+};
+
+const result = await extractFile('document.pdf', null, config);
 console.log(result.content);
-console.log(result.tables);
 ```
 
-### With Full Type Safety
 
-```typescript
-import {
-  extractFile,
-  type ExtractionConfig,
-  type ExtractionResult
-} from '@kreuzberg/node';
-
-const config: ExtractionConfig = {
-  useCache: true,
-  enableQualityProcessing: true
-};
 
-const result: ExtractionResult = await extractFile('invoice.pdf', config);
 
-// Type-safe access to all properties
-console.log(result.content);
-console.log(result.mimeType);
-console.log(result.metadata);
+#### Table Extraction
 
-if (result.tables) {
-  for (const table of result.tables) {
-    console.log(table.markdown);
-  }
+
+```typescript
+import { extractFileSync } from '@kreuzberg/node';
+
+const result = extractFileSync('document.pdf');
+
+for (const table of result.tables) {
+	console.log(`Table with ${table.cells.length} rows`);
+	console.log(`Page: ${table.pageNumber}`);
+	console.log(table.markdown);
 }
 ```
 
-## Configuration
 
-### OCR Configuration
 
-```typescript
-import { extractFile, type ExtractionConfig, type OcrConfig } from '@kreuzberg/node';
-
-const config: ExtractionConfig = {
-  ocr: {
-    backend: 'tesseract',
-    language: 'eng',
-    tesseractConfig: {
-      enableTableDetection: true,
-      psm: 6,
-      minConfidence: 50.0
-    }
-  } as OcrConfig
-};
 
-const result = await extractFile('scanned.pdf', config);
-console.log(result.content);
-```
+#### Processing Multiple Files
 
-### PDF Password Protection
 
 ```typescript
-import { extractFile, type PdfConfig } from '@kreuzberg/node';
+import { batchExtractFilesSync } from '@kreuzberg/node';
 
-const config = {
-  pdfOptions: {
-    passwords: ['password1', 'password2'],
-    extractImages: true,
-    extractMetadata: true
-  } as PdfConfig
-};
+const files = ['doc1.pdf', 'doc2.docx', 'doc3.pptx'];
+const results = batchExtractFilesSync(files);
 
-const result = await extractFile('protected.pdf', config);
+results.forEach((result, i) => {
+	console.log(`File ${i + 1}: ${result.content.length} characters`);
+});
 ```
 
-### Extract Tables
+
+
+
+
+#### Async Processing
+
+For non-blocking document processing:
 
 ```typescript
 import { extractFile } from '@kreuzberg/node';
 
-const result = await extractFile('financial-report.pdf');
+const result = await extractFile('document.pdf');
+console.log(result.content);
+```
 
-if (result.tables) {
-  for (const table of result.tables) {
-    console.log('Table as Markdown:');
-    console.log(table.markdown);
 
-    console.log('Table cells:');
-    console.log(JSON.stringify(table.cells, null, 2));
-  }
-}
-```
 
-### Text Chunking
 
-```typescript
-import { extractFile, type ChunkingConfig } from '@kreuzberg/node';
 
-const config = {
-  chunking: {
-    maxChars: 1000,
-    maxOverlap: 200
-  } as ChunkingConfig
-};
+#### Configuration Discovery
 
-const result = await extractFile('long-document.pdf', config);
+```typescript
+import { discoverExtractionConfig, extractFile } from '@kreuzberg/node';
 
-if (result.chunks) {
-  for (const chunk of result.chunks) {
-    console.log(`Chunk ${chunk.index}: ${chunk.text.substring(0, 100)}...`);
-  }
+const config = discoverExtractionConfig();
+if (config) {
+  console.log('Found configuration file');
+  const result = await extractFile('document.pdf', null, config);
+  console.log(result.content);
+} else {
+  console.log('No configuration file found, using defaults');
+  const result = await extractFile('document.pdf');
+  console.log(result.content);
 }
 ```
 
-### Language Detection
+
+
+
+
+#### Worker Thread Pool
 
 ```typescript
-import { extractFile, type LanguageDetectionConfig } from '@kreuzberg/node';
+import { createWorkerPool, extractFileInWorker, batchExtractFilesInWorker, closeWorkerPool } from '@kreuzberg/node';
 
-const config = {
-  languageDetection: {
-    enabled: true,
-    minConfidence: 0.8,
-    detectMultiple: false
-  } as LanguageDetectionConfig
-};
+// Create a pool with 4 worker threads
+const pool = createWorkerPool(4);
 
-const result = await extractFile('multilingual.pdf', config);
+try {
+  // Extract single file in worker
+  const result = await extractFileInWorker(pool, 'document.pdf', null, {
+    useCache: true
+  });
+  console.log(result.content);
 
-if (result.language) {
-  console.log(`Detected language: ${result.language.code}`);
-  console.log(`Confidence: ${result.language.confidence}`);
+  // Extract multiple files concurrently
+  const files = ['doc1.pdf', 'doc2.docx', 'doc3.xlsx'];
+  const results = await batchExtractFilesInWorker(pool, files, {
+    useCache: true
+  });
+
+  results.forEach((result, i) => {
+    console.log(`File ${i + 1}: ${result.content.length} characters`);
+  });
+} finally {
+  // Always close the pool when done
+  await closeWorkerPool(pool);
 }
 ```
 
-### Image Extraction
 
-```typescript
-import { extractFile, type ImageExtractionConfig } from '@kreuzberg/node';
-import { writeFile } from 'fs/promises';
+**Performance Benefits:**
+- **Parallel Processing**: Multiple documents extracted simultaneously
+- **CPU Utilization**: Maximizes multi-core CPU usage for large batches
+- **Queue Management**: Automatically distributes work across available workers
+- **Resource Control**: Prevents thread exhaustion with configurable pool size
 
-const config = {
-  images: {
-    extractImages: true,
-    targetDpi: 300,
-    maxImageDimension: 4096,
-    autoAdjustDpi: true
-  } as ImageExtractionConfig
-};
+**Best Practices:**
+- Use worker pools for batches of 10+ documents
+- Set pool size to number of CPU cores (default behavior)
+- Always close pools with `closeWorkerPool()` to prevent resource leaks
+- Reuse pools across multiple batch operations for efficiency
 
-const result = await extractFile('document-with-images.pdf', config);
 
-if (result.images) {
-  for (let i = 0; i < result.images.length; i++) {
-    const image = result.images[i];
-    await writeFile(`image-${i}.${image.format}`, Buffer.from(image.data));
-  }
-}
-```
 
-### Complete Configuration Example
+### Next Steps
 
-```typescript
-import {
-  extractFile,
-  type ExtractionConfig,
-  type OcrConfig,
-  type ChunkingConfig,
-  type ImageExtractionConfig,
-  type PdfConfig,
-  type TokenReductionConfig,
-  type LanguageDetectionConfig
-} from '@kreuzberg/node';
-
-const config: ExtractionConfig = {
-  useCache: true,
-  enableQualityProcessing: true,
-  forceOcr: false,
-  maxConcurrentExtractions: 8,
-
-  ocr: {
-    backend: 'tesseract',
-    language: 'eng',
-    preprocessing: true,
-    tesseractConfig: {
-      enableTableDetection: true,
-      psm: 6,
-      oem: 3,
-      minConfidence: 50.0
-    }
-  } as OcrConfig,
-
-  chunking: {
-    maxChars: 1000,
-    maxOverlap: 200
-  } as ChunkingConfig,
-
-  images: {
-    extractImages: true,
-    targetDpi: 300,
-    maxImageDimension: 4096,
-    autoAdjustDpi: true
-  } as ImageExtractionConfig,
-
-  pdfOptions: {
-    extractImages: true,
-    passwords: [],
-    extractMetadata: true
-  } as PdfConfig,
-
-  tokenReduction: {
-    mode: 'moderate',
-    preserveImportantWords: true
-  } as TokenReductionConfig,
-
-  languageDetection: {
-    enabled: true,
-    minConfidence: 0.8,
-    detectMultiple: false
-  } as LanguageDetectionConfig
-};
+- **[Installation Guide](https://kreuzberg.dev/getting-started/installation/)** - Platform-specific setup
+- **[API Documentation](https://kreuzberg.dev/api/)** - Complete API reference
+- **[Examples & Guides](https://kreuzberg.dev/guides/)** - Full code examples and usage guides
+- **[Configuration Guide](https://kreuzberg.dev/configuration/)** - Advanced configuration options
+- **[Troubleshooting](https://kreuzberg.dev/troubleshooting/)** - Common issues and solutions
 
-const result = await extractFile('document.pdf', config);
-```
 
-## Advanced Usage
 
-### Extract from Buffer
+## NAPI-RS Implementation Details
 
-```typescript
-import { extractBytes } from '@kreuzberg/node';
-import { readFile } from 'fs/promises';
+### Native Performance
 
-const buffer = await readFile('document.pdf');
-const result = await extractBytes(buffer, 'application/pdf');
-console.log(result.content);
-```
+This binding uses NAPI-RS to provide native Node.js bindings with:
 
-### Batch Processing
+- **Zero-copy data transfer** between JavaScript and Rust layers
+- **Native thread pool** for concurrent document processing
+- **Direct memory management** for efficient large document handling
+- **Binary-compatible** pre-built native modules across platforms
 
-```typescript
-import { batchExtractFiles } from '@kreuzberg/node';
+### Threading Model
 
-const files = [
-  'document1.pdf',
-  'document2.docx',
-  'document3.xlsx'
-];
+- Single documents are processed synchronously or asynchronously in a dedicated thread
+- Batch operations distribute work across available CPU cores
+- Thread count is configurable but defaults to system CPU count
+- Long-running extractions block the event loop unless using async APIs
 
-const results = await batchExtractFiles(files);
+### Memory Management
 
-for (const result of results) {
-  console.log(`${result.mimeType}: ${result.content.length} characters`);
-}
-```
+- Large documents (> 100 MB) are streamed to avoid loading entirely into memory
+- Temporary files are created in system temp directory for extraction
+- Memory is automatically released after extraction completion
+- ONNX models are cached in memory for repeated embeddings operations
 
-### Batch Processing with Custom Concurrency
 
-```typescript
-import { batchExtractFiles } from '@kreuzberg/node';
 
-const config = {
-  maxConcurrentExtractions: 4  // Process 4 files at a time
-};
+## Features
 
-const files = Array.from({ length: 20 }, (_, i) => `file-${i}.pdf`);
-const results = await batchExtractFiles(files, config);
+### Supported File Formats (56+)
 
-console.log(`Processed ${results.length} files`);
-```
+56 file formats across 8 major categories with intelligent format detection and comprehensive metadata extraction.
 
-### Extract with Metadata
+#### Office Documents
 
-```typescript
-import { extractFile } from '@kreuzberg/node';
+| Category | Formats | Capabilities |
+|----------|---------|--------------|
+| **Word Processing** | `.docx`, `.odt` | Full text, tables, images, metadata, styles |
+| **Spreadsheets** | `.xlsx`, `.xlsm`, `.xlsb`, `.xls`, `.xla`, `.xlam`, `.xltm`, `.ods` | Sheet data, formulas, cell metadata, charts |
+| **Presentations** | `.pptx`, `.ppt`, `.ppsx` | Slides, speaker notes, images, metadata |
+| **PDF** | `.pdf` | Text, tables, images, metadata, OCR support |
+| **eBooks** | `.epub`, `.fb2` | Chapters, metadata, embedded resources |
 
-const result = await extractFile('document.pdf');
+#### Images (OCR-Enabled)
 
-if (result.metadata) {
-  console.log('Title:', result.metadata.title);
-  console.log('Author:', result.metadata.author);
-  console.log('Creation Date:', result.metadata.creationDate);
-  console.log('Page Count:', result.metadata.pageCount);
-  console.log('Word Count:', result.metadata.wordCount);
-}
-```
+| Category | Formats | Features |
+|----------|---------|----------|
+| **Raster** | `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.bmp`, `.tiff`, `.tif` | OCR, table detection, EXIF metadata, dimensions, color space |
+| **Advanced** | `.jp2`, `.jpx`, `.jpm`, `.mj2`, `.pnm`, `.pbm`, `.pgm`, `.ppm` | OCR, table detection, format-specific metadata |
+| **Vector** | `.svg` | DOM parsing, embedded text, graphics metadata |
 
-### Token Reduction for LLM Processing
+#### Web & Data
 
-```typescript
-import { extractFile, type TokenReductionConfig } from '@kreuzberg/node';
+| Category | Formats | Features |
+|----------|---------|----------|
+| **Markup** | `.html`, `.htm`, `.xhtml`, `.xml`, `.svg` | DOM parsing, metadata (Open Graph, Twitter Card), link extraction |
+| **Structured Data** | `.json`, `.yaml`, `.yml`, `.toml`, `.csv`, `.tsv` | Schema detection, nested structures, validation |
+| **Text & Markdown** | `.txt`, `.md`, `.markdown`, `.rst`, `.org`, `.rtf` | CommonMark, GFM, reStructuredText, Org Mode |
 
-const config = {
-  tokenReduction: {
-    mode: 'aggressive',  // Options: 'light', 'moderate', 'aggressive'
-    preserveImportantWords: true
-  } as TokenReductionConfig
-};
+#### Email & Archives
 
-const result = await extractFile('long-document.pdf', config);
+| Category | Formats | Features |
+|----------|---------|----------|
+| **Email** | `.eml`, `.msg` | Headers, body (HTML/plain), attachments, threading |
+| **Archives** | `.zip`, `.tar`, `.tgz`, `.gz`, `.7z` | File listing, nested archives, metadata |
 
-// Reduced token count while preserving meaning
-console.log(`Original length: ${result.content.length}`);
-console.log(`Processed for LLM context window`);
-```
+#### Academic & Scientific
 
-## Error Handling
+| Category | Formats | Features |
+|----------|---------|----------|
+| **Citations** | `.bib`, `.biblatex`, `.ris`, `.enw`, `.csl` | Bibliography parsing, citation extraction |
+| **Scientific** | `.tex`, `.latex`, `.typst`, `.jats`, `.ipynb`, `.docbook` | LaTeX, Jupyter notebooks, PubMed JATS |
+| **Documentation** | `.opml`, `.pod`, `.mdoc`, `.troff` | Technical documentation formats |
 
-```typescript
-import {
-  extractFile,
-  KreuzbergError,
-  ValidationError,
-  ParsingError,
-  OCRError,
-  MissingDependencyError
-} from '@kreuzberg/node';
+**[Complete Format Reference](https://kreuzberg.dev/reference/formats/)**
 
-try {
-  const result = await extractFile('document.pdf');
-  console.log(result.content);
-} catch (error) {
-  if (error instanceof ValidationError) {
-    console.error('Invalid configuration or input:', error.message);
-  } else if (error instanceof ParsingError) {
-    console.error('Failed to parse document:', error.message);
-  } else if (error instanceof OCRError) {
-    console.error('OCR processing failed:', error.message);
-  } else if (error instanceof MissingDependencyError) {
-    console.error(`Missing dependency: ${error.dependency}`);
-    console.error('Installation instructions:', error.message);
-  } else if (error instanceof KreuzbergError) {
-    console.error('Kreuzberg error:', error.message);
-  } else {
-    throw error;
-  }
-}
-```
+### Key Capabilities
 
-## API Reference
-
-### Extraction Functions
-
-#### `extractFile(filePath: string, config?: ExtractionConfig): Promise<ExtractionResult>`
-Asynchronously extract content from a file.
-
-#### `extractFileSync(filePath: string, config?: ExtractionConfig): ExtractionResult`
-Synchronously extract content from a file.
-
-#### `extractBytes(data: Buffer, mimeType: string, config?: ExtractionConfig): Promise<ExtractionResult>`
-Asynchronously extract content from a buffer.
-
-#### `extractBytesSync(data: Buffer, mimeType: string, config?: ExtractionConfig): ExtractionResult`
-Synchronously extract content from a buffer.
-
-#### `batchExtractFiles(paths: string[], config?: ExtractionConfig): Promise<ExtractionResult[]>`
-Asynchronously extract content from multiple files in parallel.
-
-#### `batchExtractFilesSync(paths: string[], config?: ExtractionConfig): ExtractionResult[]`
-Synchronously extract content from multiple files.
-
-### Types
-
-#### `ExtractionResult`
-Main result object containing:
-- `content: string` - Extracted text content
-- `mimeType: string` - MIME type of the document
-- `metadata?: Metadata` - Document metadata
-- `tables?: Table[]` - Extracted tables
-- `images?: ImageData[]` - Extracted images
-- `chunks?: Chunk[]` - Text chunks (if chunking enabled)
-- `language?: LanguageInfo` - Detected language (if enabled)
-
-#### `ExtractionConfig`
-Configuration object for extraction:
-- `useCache?: boolean` - Enable result caching
-- `enableQualityProcessing?: boolean` - Enable text quality improvements
-- `forceOcr?: boolean` - Force OCR even for text-based PDFs
-- `maxConcurrentExtractions?: number` - Max parallel extractions
-- `ocr?: OcrConfig` - OCR settings
-- `chunking?: ChunkingConfig` - Text chunking settings
-- `images?: ImageExtractionConfig` - Image extraction settings
-- `pdfOptions?: PdfConfig` - PDF-specific options
-- `tokenReduction?: TokenReductionConfig` - Token reduction settings
-- `languageDetection?: LanguageDetectionConfig` - Language detection settings
-
-#### `OcrConfig`
-OCR configuration:
-- `backend: string` - OCR backend ('tesseract', 'easyocr', 'paddleocr')
-- `language: string` - Language code (e.g., 'eng', 'fra', 'deu')
-- `preprocessing?: boolean` - Enable image preprocessing
-- `tesseractConfig?: TesseractConfig` - Tesseract-specific options
-
-#### `Table`
-Extracted table structure:
-- `markdown: string` - Table in Markdown format
-- `cells: TableCell[][]` - 2D array of table cells
-- `rowCount: number` - Number of rows
-- `columnCount: number` - Number of columns
-
-### Exceptions
-
-All Kreuzberg exceptions extend the base `KreuzbergError` class:
-
-- `KreuzbergError` - Base error class for all Kreuzberg errors
-- `ValidationError` - Invalid configuration, missing required fields, or invalid input
-- `ParsingError` - Document parsing failure or corrupted file
-- `OCRError` - OCR processing failure
-- `MissingDependencyError` - Missing optional system dependency (includes installation instructions)
-
-## Supported Formats
-
-| Category | Formats |
-|----------|---------|
-| **Documents** | PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, ODT, ODP, ODS, RTF |
-| **Images** | PNG, JPEG, JPG, WEBP, BMP, TIFF, GIF |
-| **Web** | HTML, XHTML, XML |
-| **Text** | TXT, MD, CSV, TSV, JSON, YAML, TOML |
-| **Email** | EML, MSG |
-| **Archives** | ZIP, TAR, 7Z |
-| **Other** | And 30+ more formats |
-
-## Performance
-
-Kreuzberg is built with a native Rust core, providing significant performance improvements over pure JavaScript solutions:
-
-- **10-50x faster** text extraction compared to pure Node.js libraries
-- **Native multithreading** for batch processing
-- **Optimized memory usage** with streaming for large files
-- **Zero-copy operations** where possible
-- **Efficient caching** to avoid redundant processing
-
-### Benchmarks
-
-Processing 100 mixed documents (PDF, DOCX, XLSX):
-
-| Library | Time | Memory |
-|---------|------|--------|
-| Kreuzberg | 2.3s | 145 MB |
-| pdf-parse + mammoth | 23.1s | 890 MB |
-| textract | 45.2s | 1.2 GB |
+- **Text Extraction** - Extract all text content with position and formatting information
+- **Metadata Extraction** - Retrieve document properties, creation date, author, etc.
+- **Table Extraction** - Parse tables with structure and cell content preservation
+- **Image Extraction** - Extract embedded images and render page previews
+- **OCR Support** - Integrate multiple OCR backends for scanned documents
 
-## Troubleshooting
+- **Async/Await** - Non-blocking document processing with concurrent operations
 
-### Native Module Not Found
 
-If you encounter errors about missing native modules:
+- **Plugin System** - Extensible post-processing for custom text transformation
 
-```bash
-npm rebuild @kreuzberg/node
-```
 
-### OCR Not Working
+- **Embeddings** - Generate vector embeddings using ONNX Runtime models
 
-Ensure Tesseract is installed and available in PATH:
+- **Batch Processing** - Efficiently process multiple documents in parallel
+- **Memory Efficient** - Stream large files without loading entirely into memory
+- **Language Detection** - Detect and support multiple languages in documents
+- **Configuration** - Fine-grained control over extraction behavior
 
-```bash
-tesseract --version
-```
+### Performance Characteristics
+
+| Format | Speed | Memory | Notes |
+|--------|-------|--------|-------|
+| **PDF (text)** | 10-100 MB/s | ~50MB per doc | Fastest extraction |
+| **Office docs** | 20-200 MB/s | ~100MB per doc | DOCX, XLSX, PPTX |
+| **Images (OCR)** | 1-5 MB/s | Variable | Depends on OCR backend |
+| **Archives** | 5-50 MB/s | ~200MB per doc | ZIP, TAR, etc. |
+| **Web formats** | 50-200 MB/s | Streaming | HTML, XML, JSON |
 
-If Tesseract is not found:
-- macOS: `brew install tesseract`
-- Ubuntu: `sudo apt-get install tesseract-ocr`
-- Windows: Download from [tesseract-ocr/tesseract](https://github.com/tesseract-ocr/tesseract)
 
-### Memory Issues with Large PDFs
 
-For very large PDFs, use chunking to reduce memory usage:
+## OCR Support
+
+Kreuzberg supports multiple OCR backends for extracting text from scanned documents and images:
+
+
+- **Tesseract**
+
+- **Guten**
+
+
+### OCR Configuration Example
 
 ```typescript
+import { extractFile } from '@kreuzberg/node';
+
 const config = {
-  chunking: { maxChars: 1000 }
+	ocr: {
+		backend: 'tesseract',
+		language: 'eng+fra',
+		tesseractConfig: {
+			psm: 3,
+		},
+	},
 };
-const result = await extractFile('large.pdf', config);
+
+const result = await extractFile('document.pdf', null, config);
+console.log(result.content);
 ```
 
-### TypeScript Types Not Resolving
 
-Make sure you're using:
-- Node.js 18 or higher
-- TypeScript 5.0 or higher
 
-The package includes built-in type definitions.
 
-### Performance Optimization
+## Async Support
 
-For maximum performance when processing many files:
+This binding provides full async/await support for non-blocking document processing:
 
 ```typescript
-// Use batch processing instead of sequential
-const results = await batchExtractFiles(files, {
-  maxConcurrentExtractions: 8  // Tune based on CPU cores
-});
+import { extractFile } from '@kreuzberg/node';
+
+const result = await extractFile('document.pdf');
+console.log(result.content);
 ```
 
-## Examples
 
-### Extract Invoice Data
 
-```typescript
-import { extractFile } from '@kreuzberg/node';
 
-const result = await extractFile('invoice.pdf');
+## Plugin System
 
-// Access tables for line items
-if (result.tables && result.tables.length > 0) {
-  const lineItems = result.tables[0];
-  console.log(lineItems.markdown);
-}
+Kreuzberg supports extensible post-processing plugins for custom text transformation and filtering.
 
-// Access metadata for invoice details
-if (result.metadata) {
-  console.log('Invoice Date:', result.metadata.creationDate);
-}
-```
+For detailed plugin documentation, visit [Plugin System Guide](https://kreuzberg.dev/plugins/).
 
-### Process Scanned Documents
 
-```typescript
-import { extractFile } from '@kreuzberg/node';
 
-const config = {
-  forceOcr: true,
-  ocr: {
-    backend: 'tesseract',
-    language: 'eng',
-    preprocessing: true
-  }
-};
 
-const result = await extractFile('scanned-contract.pdf', config);
-console.log(result.content);
-```
+## Embeddings Support
+
+Generate vector embeddings for extracted text using the built-in ONNX Runtime support. Requires ONNX Runtime installation.
+
+**[Embeddings Guide](https://kreuzberg.dev/features/#embeddings)**
+
+
+
+## Batch Processing
 
-### Build a Document Search Index
+Process multiple documents efficiently:
 
 ```typescript
-import { batchExtractFiles } from '@kreuzberg/node';
-import { glob } from 'glob';
+import { batchExtractFilesSync } from '@kreuzberg/node';
 
-// Find all documents
-const files = await glob('documents/**/*.{pdf,docx,xlsx}');
+const files = ['doc1.pdf', 'doc2.docx', 'doc3.pptx'];
+const results = batchExtractFilesSync(files);
 
-// Extract in batches
-const results = await batchExtractFiles(files, {
-  maxConcurrentExtractions: 8,
-  enableQualityProcessing: true
+results.forEach((result, i) => {
+	console.log(`File ${i + 1}: ${result.content.length} characters`);
 });
+```
 
-// Build search index
-const searchIndex = results.map((result, i) => ({
-  path: files[i],
-  content: result.content,
-  metadata: result.metadata
-}));
 
-console.log(`Indexed ${searchIndex.length} documents`);
-```
+
+
+## Configuration
+
+For advanced configuration options including language detection, table extraction, OCR settings, and more:
+
+**[Configuration Guide](https://kreuzberg.dev/configuration/)**
 
 ## Documentation
 
-For comprehensive documentation, visit [https://kreuzberg.dev](https://kreuzberg.dev)
+- **[Official Documentation](https://kreuzberg.dev/)**
+- **[API Reference](https://kreuzberg.dev/reference/api-typescript/)**
+- **[Examples & Guides](https://kreuzberg.dev/guides/)**
+
+## Troubleshooting
+
+For common issues and solutions, visit [Troubleshooting Guide](https://kreuzberg.dev/troubleshooting/).
 
 ## Contributing
 
-We welcome contributions! Please see our [Contributing Guide](../../CONTRIBUTING.md) for details.
+Contributions are welcome! See [Contributing Guide](https://github.com/kreuzberg-dev/kreuzberg/blob/main/CONTRIBUTING.md).
 
 ## License
 
-MIT
+MIT License - see LICENSE file for details.
 
-## Links
+## Support
 
-- [Website](https://kreuzberg.dev)
-- [Documentation](https://kreuzberg.dev)
-- [GitHub](https://github.com/kreuzberg-dev/kreuzberg)
-- [Issue Tracker](https://github.com/kreuzberg-dev/kreuzberg/issues)
-- [Changelog](https://github.com/kreuzberg-dev/kreuzberg/blob/main/CHANGELOG.md)
-- [npm Package](https://www.npmjs.com/package/@kreuzberg/node)
+- **Discord Community**: [Join our Discord](https://discord.gg/pXxagNK2zN)
+- **GitHub Issues**: [Report bugs](https://github.com/kreuzberg-dev/kreuzberg/issues)
+- **Discussions**: [Ask questions](https://github.com/kreuzberg-dev/kreuzberg/discussions)