kreuzberg 4.0.0.pre.rc.13 → 4.0.0.pre.rc.14

data/vendor/kreuzberg-ffi/README.md

@@ -1,851 +1,851 @@
-# kreuzberg-ffi
-
-C Foreign Function Interface (FFI) bindings for the Kreuzberg document intelligence library.
-
-## Overview
-
-This crate provides a C-compatible API that bridges the high-performance Rust core of Kreuzberg with multiple programming languages and FFI systems. It is the foundation for language bindings in Java (Panama FFI), Go (cgo), C# (P/Invoke), and other languages with C interoperability.
-
-The FFI exposes extraction functions, configuration management, plugin registration, and error handling through a stable C interface with thread-safe callbacks.
-
-## Architecture
-
-### FFI Bridge Layers
-
-```
-Language-Specific Bindings
-    ↓
-Kreuzberg FFI C Library (crates/kreuzberg-ffi) ← This crate
-    ↓
-Rust Core Library (crates/kreuzberg)
-    ↓
-Document Extraction Engines
-```
-
-### Binding Support
-
-This FFI layer is consumed by:
-
-- **Java** (packages/java): Using Java 25 Foreign Function & Memory API (Panama FFI)
-- **Go** (packages/go): Using cgo wrapper bindings
-- **C#** (packages/csharp): Using P/Invoke interop
-- **Zig** and other C-compatible languages
-
-### Key Components
-
-- **Core Extraction** (`extract_file`, `extract_bytes`): Document text and data extraction
-- **Batch Operations**: Parallel processing of multiple documents
-- **MIME Detection**: File format identification
-- **Configuration Management**: Loading and applying extraction settings
-- **Plugin System**: OCR backend registration and callbacks
-- **Error Handling**: Thread-local error message storage
-- **Memory Management**: Safe pointer handling and FFI boundaries
-
-## Installation
-
-### Build from Source
-
-```bash
-cargo build --release -p kreuzberg-ffi
-```
-
-### Output Artifacts
-
-After building, you will have:
-
-- **Dynamic Library**: `target/release/libkreuzberg_ffi.{so,dylib,dll}`
-  - For loading at runtime
-  - Platform-specific extensions (`.so` Linux, `.dylib` macOS, `.dll` Windows)
-
-- **Static Library**: `target/release/libkreuzberg_ffi.{a,lib}`
-  - For static linking into applications
-  - Platform-specific extensions (`.a` Unix, `.lib` Windows)
-
-- **Header File**: Auto-generated via `cbindgen` during build
-
-### Header Generation
-
-The C header file is automatically generated during the build process via `cbindgen`:
-
-```bash
-cargo build --release -p kreuzberg-ffi
-# Header is generated at build time based on #[no_mangle] functions
-```
-
-For manual header generation:
-
-```bash
-cargo build --features html,embeddings -p kreuzberg-ffi
-```
-
-### pkg-config File Generation
-
-The build process automatically generates pkg-config files for library discovery:
-
-```bash
-cargo build --release -p kreuzberg-ffi
-```
-
-This creates two variants in `crates/kreuzberg-ffi/`:
-- **kreuzberg-ffi.pc**: Development version (prefix points to repository)
-- **kreuzberg-ffi-install.pc**: Installation version (prefix=/usr/local)
-
-The development variant enables monorepo developers to use pkg-config:
-
-```bash
-export PKG_CONFIG_PATH="$PWD/crates/kreuzberg-ffi:$PKG_CONFIG_PATH"
-pkg-config --cflags kreuzberg-ffi  # Returns -I/path/to/repo/crates/kreuzberg-ffi
-pkg-config --libs kreuzberg-ffi    # Returns -L/path/to/repo/target/release -lkreuzberg_ffi
-```
-
-The installation variant is used in release artifacts for third-party use.
-
-### Installing from Release Artifacts
-
-Pre-built binaries are available for Linux, macOS, and Windows (MinGW) from the [releases page](https://github.com/kreuzberg-dev/kreuzberg/releases).
-
-Each `go-ffi-{platform}.tar.gz` archive contains:
-- `lib/`: Shared libraries (kreuzberg-ffi, pdfium, onnxruntime)
-- `include/`: C header file (kreuzberg.h)
-- `share/pkgconfig/`: pkg-config file for library discovery
-- `README.md`: Installation instructions
-
-Installation:
-
-```bash
-# Download and extract
-tar -xzf go-ffi-linux-x86_64.tar.gz
-cd kreuzberg-ffi
-
-# System-wide installation (requires sudo)
-sudo cp -r lib/* /usr/local/lib/
-sudo cp -r include/* /usr/local/include/
-sudo cp -r share/* /usr/local/share/
-sudo ldconfig  # Linux only
-
-# Verify
-pkg-config --modversion kreuzberg-ffi
-```
-
-For user-local installation or custom prefix, see the README.md included in the archive.
-
-## Quick Start: C Example
-
-### Basic Extraction
-
-```c
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-// Include the auto-generated Kreuzberg FFI header
-#include "kreuzberg_ffi.h"
-
-int main() {
-    // Extract text from a PDF file
-    const char* file_path = "document.pdf";
-    const char* mime_type = NULL;  // Auto-detect
-    const char* config_json = "{}";  // Empty config uses defaults
-
-    // Perform extraction
-    ExtractionResult result = kreuzberg_extract_file(
-        file_path,
-        mime_type,
-        config_json
-    );
-
-    // Check for errors
-    if (result.error != NULL) {
-        fprintf(stderr, "Extraction failed: %s\n", result.error);
-        kreuzberg_free_string(result.error);
-        return 1;
-    }
-
-    // Process the result
-    printf("Extracted content:\n%s\n", result.content);
-    printf("MIME Type: %s\n", result.mime_type);
-
-    // Free resources
-    kreuzberg_free_extraction_result(result);
-
-    return 0;
-}
-```
-
-### Compilation
-
-```bash
-# Link against the dynamic library
-gcc -o extract_example extract.c \
-    -L/path/to/kreuzberg/target/release \
-    -lkreuzberg_ffi
-```
-
-## API Reference
-
-### Core Extraction Functions
-
-All functions return results via out-parameters or result structs. Error messages are accessible via `kreuzberg_get_last_error()`.
-
-#### `ExtractionResult kreuzberg_extract_file(const char *path, const char *mime_type, const char *config_json)`
-
-Extract text and metadata from a file.
-
-**Parameters:**
-- `path`: Absolute or relative file path (must exist)
-- `mime_type`: Optional MIME type hint (e.g., "application/pdf"). Pass NULL for auto-detection.
-- `config_json`: JSON string with extraction configuration
-
-**Returns:**
-- `ExtractionResult`: Contains `content` (extracted text), `mime_type`, `metadata` (JSON), and optional `error`
-
-**Example:**
-
-```c
-const char* config = "{\"use_cache\": true, \"enable_quality_processing\": true}";
-ExtractionResult result = kreuzberg_extract_file("document.pdf", NULL, config);
-
-if (result.error) {
-    printf("Error: %s\n", result.error);
-} else {
-    printf("Content: %s\n", result.content);
-    printf("MIME: %s\n", result.mime_type);
-}
-
-kreuzberg_free_extraction_result(result);
-```
-
-#### `ExtractionResult kreuzberg_extract_bytes(const char *data, size_t data_len, const char *mime_type, const char *config_json)`
-
-Extract from a byte buffer (e.g., file in memory).
-
-**Parameters:**
-- `data`: Pointer to byte buffer
-- `data_len`: Buffer length in bytes
-- `mime_type`: Optional MIME type hint (required if auto-detection not desired)
-- `config_json`: JSON extraction configuration
-
-**Returns:**
-- `ExtractionResult`: Same as `extract_file`
-
-**Example:**
-
-```c
-// Read file into buffer
-FILE* f = fopen("document.pdf", "rb");
-fseek(f, 0, SEEK_END);
-size_t size = ftell(f);
-fseek(f, 0, SEEK_SET);
-unsigned char* buffer = malloc(size);
-fread(buffer, 1, size, f);
-fclose(f);
-
-// Extract from buffer
-const char* config = "{}";
-ExtractionResult result = kreuzberg_extract_bytes(
-    (const char*)buffer,
-    size,
-    "application/pdf",
-    config
-);
-
-if (!result.error) {
-    printf("Content: %s\n", result.content);
-}
-
-kreuzberg_free_extraction_result(result);
-free(buffer);
-```
-
-#### `BatchExtractionResult kreuzberg_batch_extract_files(const char **paths, size_t paths_count, const char *config_json)`
-
-Process multiple files in parallel.
-
-**Parameters:**
-- `paths`: Array of file path strings
-- `paths_count`: Number of paths in array
-- `config_json`: JSON extraction configuration (applied to all files)
-
-**Returns:**
-- `BatchExtractionResult`: Contains array of `ExtractionResult` and error status
-
-**Example:**
-
-```c
-const char* files[] = {"doc1.pdf", "doc2.docx", "doc3.xlsx"};
-size_t count = 3;
-
-BatchExtractionResult batch_result = kreuzberg_batch_extract_files(
-    files,
-    count,
-    "{}"
-);
-
-if (batch_result.error) {
-    printf("Batch error: %s\n", batch_result.error);
-} else {
-    for (size_t i = 0; i < batch_result.count; i++) {
-        printf("File %zu: %s\n", i, batch_result.results[i].mime_type);
-    }
-}
-
-kreuzberg_free_batch_extraction_result(batch_result);
-```
-
-#### `char* kreuzberg_detect_mime_type(const char *path, bool use_cache)`
-
-Identify the MIME type of a file using magic bytes and file signatures.
-
-**Parameters:**
-- `path`: File path to analyze
-- `use_cache`: Whether to use cached results for the same path
-
-**Returns:**
-- `char*`: MIME type string (must be freed with `kreuzberg_free_string`)
-
-**Example:**
-
-```c
-char* mime = kreuzberg_detect_mime_type("unknown-file", true);
-if (mime) {
-    printf("MIME Type: %s\n", mime);
-    kreuzberg_free_string(mime);
-}
-```
-
-### Configuration Management
-
-#### `char* kreuzberg_get_default_config_json()`
-
-Get default extraction configuration as JSON.
-
-**Returns:**
-- `char*`: JSON string (must be freed with `kreuzberg_free_string`)
-
-**Example:**
-
-```c
-char* default_config = kreuzberg_get_default_config_json();
-printf("Default config: %s\n", default_config);
-kreuzberg_free_string(default_config);
-```
-
-#### Configuration JSON Schema
-
-```json
-{
-  "use_cache": true,
-  "enable_quality_processing": false,
-  "force_ocr": false,
-  "ocr": {
-    "backend": "tesseract",
-    "language": "eng",
-    "tesseract_config": {
-      "enable_table_detection": true,
-      "psm": 6,
-      "min_confidence": 50.0
-    }
-  },
-  "chunking": {
-    "max_chars": 1000,
-    "max_overlap": 200
-  },
-  "language_detection": {
-    "enabled": false,
-    "min_confidence": 0.8,
-    "detect_multiple": false
-  }
-}
-```
-
-### Plugin System
-
-#### `void kreuzberg_register_ocr_backend(const char *backend_name, OcrBackendCallback callback)`
-
-Register a custom OCR backend implemented in the calling language.
-
-**Parameters:**
-- `backend_name`: Unique identifier for the backend (e.g., "custom-ocr")
-- `callback`: Function pointer to OCR processing implementation
-
-**Example (C):**
-
-```c
-// Define the callback function
-OcrBackendResult custom_ocr_callback(
-    const char* image_data,
-    size_t image_size,
-    const char* language,
-    void* user_data
-) {
-    // Call out to Python, Go, C#, etc. to perform OCR
-    // Implementation depends on the host language
-
-    OcrBackendResult result = {0};
-    result.content = "Extracted text from OCR";
-    result.metadata = "{}";
-    return result;
-}
-
-// Register the backend
-kreuzberg_register_ocr_backend("easyocr", custom_ocr_callback);
-
-// Now use it in extraction config:
-const char* config = "{\"ocr\": {\"backend\": \"easyocr\", \"language\": \"eng\"}}";
-ExtractionResult result = kreuzberg_extract_file("scanned.pdf", NULL, config);
-```
-
-### Error Handling
-
-#### `char* kreuzberg_get_last_error()`
-
-Retrieve the last error message.
-
-**Returns:**
-- `char*`: Error message string (NULL if no error)
-
-Note: Error messages are thread-local and persist until the next Kreuzberg function call.
-
-**Example:**
-
-```c
-ExtractionResult result = kreuzberg_extract_file("invalid.pdf", NULL, "{}");
-
-if (result.error == NULL) {
-    // Also check global error state
-    char* error = kreuzberg_get_last_error();
-    if (error) {
-        printf("Error: %s\n", error);
-    }
-}
-```
-
-### Memory Management
-
-#### `void kreuzberg_free_string(char *ptr)`
-
-Free a string allocated by FFI functions.
-
-**Note:** Only use for strings returned by FFI functions, not for caller-allocated strings.
-
-**Example:**
-
-```c
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-kreuzberg_free_string(mime);
-
-char* error = kreuzberg_get_last_error();
-if (error) {
-    kreuzberg_free_string(error);
-}
-```
-
-#### `void kreuzberg_free_extraction_result(ExtractionResult result)`
-
-Free memory associated with an extraction result.
-
-**Example:**
-
-```c
-ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
-// ... use result ...
-kreuzberg_free_extraction_result(result);
-```
-
-#### `void kreuzberg_free_batch_extraction_result(BatchExtractionResult result)`
-
-Free memory associated with a batch extraction result.
-
-**Example:**
-
-```c
-BatchExtractionResult batch = kreuzberg_batch_extract_files(files, 3, "{}");
-// ... use batch.results ...
-kreuzberg_free_batch_extraction_result(batch);
-```
-
-## Type Definitions
-
-### ExtractionResult
-
-```c
-typedef struct {
-    char* content;           // Extracted text
-    char* mime_type;         // MIME type of document
-    char* metadata;          // JSON metadata string
-    char* error;             // Error message (NULL if success)
-    // ... tables, images, chunks as needed
-} ExtractionResult;
-```
-
-### BatchExtractionResult
-
-```c
-typedef struct {
-    ExtractionResult* results;  // Array of extraction results
-    size_t count;               // Number of results
-    char* error;                // Batch-level error (NULL if success)
-} BatchExtractionResult;
-```
-
-### OcrBackendCallback
-
-```c
-typedef OcrBackendResult (*OcrBackendCallback)(
-    const char* image_data,
-    size_t image_size,
-    const char* language,
-    void* user_data
-);
-```
-
-## FFI Safety Guidelines
-
-### Thread Safety
-
-All FFI functions are **thread-safe**. The Kreuzberg core uses Arc, Mutex, and RwLock for safe concurrent access:
-
-```c
-// Safe to call from multiple threads
-for (int i = 0; i < 10; i++) {
-    pthread_t thread;
-    pthread_create(&thread, NULL, extract_worker, (void*)(intptr_t)i);
-}
-```
-
-### Pointer Validation
-
-The FFI layer validates all input pointers:
-
-```c
-// Safe: NULL pointers are handled gracefully
-ExtractionResult result = kreuzberg_extract_file(NULL, NULL, "{}");
-// Returns error in result.error
-
-// Safe: Invalid paths return errors
-ExtractionResult result = kreuzberg_extract_file("/nonexistent", NULL, "{}");
-// Returns error in result.error
-```
-
-### Memory Lifetime
-
-- **Returned strings** must be freed with `kreuzberg_free_string()`
-- **Result structs** must be freed with appropriate cleanup functions
-- **Input parameters** are copied internally; caller retains ownership
-
-```c
-// Correct: Free returned values
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-kreuzberg_free_string(mime);
-
-// Correct: Input paths can be freed immediately after call
-{
-    char path[256];
-    snprintf(path, sizeof(path), "document.pdf");
-    ExtractionResult result = kreuzberg_extract_file(path, NULL, "{}");
-    // path can be freed here; FFI has already copied it
-}
-
-// Correct: Free result structures
-ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
-// ... use result ...
-kreuzberg_free_extraction_result(result);
-```
-
-### Error Handling Pattern
-
-```c
-ExtractionResult result = kreuzberg_extract_file(path, NULL, config);
-
-// Check return value first
-if (result.error != NULL) {
-    fprintf(stderr, "FFI Error: %s\n", result.error);
-    kreuzberg_free_extraction_result(result);
-    return;
-}
-
-// Process successful result
-printf("Content: %s\n", result.content);
-printf("MIME: %s\n", result.mime_type);
-
-// Always cleanup
-kreuzberg_free_extraction_result(result);
-```
-
-## Building from C
-
-### Static Linking
-
-```bash
-# Build the library
-cargo build --release -p kreuzberg-ffi
-
-# Create your C program
-gcc -c -o myapp.o myapp.c
-
-# Link statically
-gcc -o myapp myapp.o \
-    -L/path/to/kreuzberg/target/release \
-    -l:libkreuzberg_ffi.a \
-    -pthread -ldl  # May need additional system libs
-
-# Run
-./myapp
-```
-
-### Dynamic Linking
-
-```bash
-# Build the library
-cargo build --release -p kreuzberg-ffi
-
-# Create your C program
-gcc -c -o myapp.o myapp.c
-
-# Link dynamically
-gcc -o myapp myapp.o \
-    -L/path/to/kreuzberg/target/release \
-    -lkreuzberg_ffi
-
-# Set library path and run
-export LD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$LD_LIBRARY_PATH
-./myapp
-```
-
-### macOS Considerations
-
-```bash
-# Build
-cargo build --release -p kreuzberg-ffi
-
-# Link
-gcc -o myapp myapp.o \
-    -L/path/to/kreuzberg/target/release \
-    -lkreuzberg_ffi
-
-# Set runtime path
-export DYLD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$DYLD_LIBRARY_PATH
-./myapp
-```
-
-## Language Binding Integration
-
-### Java Integration (Panama FFI)
-
-The FFI is wrapped in Java 25's Foreign Function & Memory API:
-
-```java
-// Java code that calls FFI
-Arena arena = Arena.ofConfined();
-MemorySegment path = arena.allocateUtf8String("document.pdf");
-MemorySegment config = arena.allocateUtf8String("{}");
-
-ExtractionResult result = KreuzbergFFI.extract_file(path, MemorySegment.NULL, config);
-```
-
-### Go Integration (cgo)
-
-The FFI is exposed through cgo bindings:
-
-```go
-// Go code that calls FFI
-C.kreuzberg_extract_file(C.CString("document.pdf"), nil, C.CString("{}"))
-```
-
-### C# Integration (P/Invoke)
-
-The FFI is declared in C# through P/Invoke:
-
-```csharp
-[DllImport("kreuzberg_ffi", CharSet = CharSet.Ansi)]
-private static extern IntPtr kreuzberg_extract_file(
-    string path,
-    string mimeType,
-    string configJson
-);
-```
-
-## Supported Features
-
-### Default Features
-- `html`: HTML to Markdown conversion support
-- `embeddings`: Text embedding extraction via fastembed-rs (requires ONNX Runtime - must be installed separately)
-
-### System Requirements for Embeddings
-
-If using the `embeddings` feature, ONNX Runtime must be installed on the system:
-
-```bash
-# macOS
-brew install onnxruntime
-
-# Ubuntu/Debian
-sudo apt install libonnxruntime libonnxruntime-dev
-
-# Windows (MSVC)
-scoop install onnxruntime
-# OR download from https://github.com/microsoft/onnxruntime/releases
-```
-
-Without ONNX Runtime, embeddings functionality will raise errors at runtime.
-
-### Core Feature (Windows MinGW Compatibility)
-- `core`: Minimal feature set for cross-platform compatibility
-  - Includes: `html` (HTML to Markdown conversion)
-  - Excludes: `embeddings` (ONNX Runtime not available on MinGW)
-  - Use case: Windows Go bindings with MinGW toolchain
-
-### Platform-Specific Build Requirements
-
-**Windows MinGW (Go bindings):**
-
-The Windows ONNX Runtime library only provides MSVC-compatible .lib files. MinGW cannot link against these, requiring the core feature:
-
-```bash
-# Windows MinGW - Use core feature
-cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-gnu --no-default-features --features core
-
-# Windows MSVC - Full features available
-cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-msvc
-
-# Unix (Linux/macOS) - Full features available
-cargo build --release -p kreuzberg-ffi
-```
-
-**Why MinGW Requires core Feature:**
-- ONNX Runtime distributes Windows binaries compiled with MSVC toolchain
-- MSVC .lib files use different name mangling and linking conventions than MinGW
-- MinGW's GNU toolchain cannot consume MSVC import libraries
-- The `core` feature excludes the `embeddings` dependency, which depends on ort-sys (ONNX Runtime)
-- HTML support (via html-to-markdown-rs) is pure Rust and works on all platforms
-
-### Building with Features
-
-```bash
-# Build with HTML and embeddings support (default)
-cargo build --release -p kreuzberg-ffi
-
-# Build with core feature only (Windows MinGW compatibility)
-cargo build --release -p kreuzberg-ffi --no-default-features --features core
-
-# Build without any features (minimal FFI)
-cargo build --release -p kreuzberg-ffi --no-default-features
-```
-
-## Performance Characteristics
-
-- **Single extraction**: 10-100ms (varies by file size and format)
-- **Batch processing**: Near-linear scaling with CPU cores
-- **OCR processing**: 100-500ms per page
-- **Memory overhead**: ~2-5MB per extraction operation
-- **Thread safety**: Zero synchronization overhead on single-threaded extraction
-
-## Key Files
-
-- `src/lib.rs`: FFI function implementations and plugin system
-- `Cargo.toml`: Dependencies and features
-- Generated header: Auto-created by cbindgen during build
-
-## Building
-
-### Development Build
-
-```bash
-cargo build -p kreuzberg-ffi
-```
-
-### Release Build
-
-```bash
-cargo build --release -p kreuzberg-ffi
-```
-
-### With All Features
-
-```bash
-cargo build --release -p kreuzberg-ffi --features html,embeddings
-```
-
-## Testing
-
-```bash
-# Run FFI tests
-cargo test -p kreuzberg-ffi
-
-# With logging
-RUST_LOG=debug cargo test -p kreuzberg-ffi -- --nocapture
-```
-
-## Troubleshooting
-
-### Library Not Found
-
-Ensure the built library is in the linker search path:
-
-```bash
-# Check for built libraries
-ls -la target/release/libkreuzberg_ffi*
-
-# Add to library path
-export LD_LIBRARY_PATH=target/release:$LD_LIBRARY_PATH
-export DYLD_LIBRARY_PATH=target/release:$DYLD_LIBRARY_PATH  # macOS
-```
-
-### Undefined Reference Errors
-
-Ensure you're linking against the FFI library, not the core library:
-
-```bash
-# Correct
-gcc -o app app.c -lkreuzberg_ffi
-
-# Incorrect
-gcc -o app app.c -lkreuzberg  # Wrong library
-```
-
-### Memory Leaks
-
-Always free returned strings and result structures:
-
-```c
-// Problem: Memory leak
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-printf("%s\n", mime);
-// mime not freed!
-
-// Solution: Free the string
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-printf("%s\n", mime);
-kreuzberg_free_string(mime);
-```
-
-### Thread-Local Error Messages
-
-Each thread has its own error message storage. Check both return values and `kreuzberg_get_last_error()`:
-
-```c
-// Safe across threads
-#pragma omp parallel for
-for (int i = 0; i < 10; i++) {
-    ExtractionResult result = kreuzberg_extract_file(files[i], NULL, "{}");
-    if (result.error) {
-        printf("Thread %d error: %s\n", i, result.error);
-    }
-}
-```
-
-## References
-
-- **Kreuzberg Core**: `../kreuzberg/`
-- **C FFI Standards**: https://en.cppreference.com/w/c
-- **cbindgen Documentation**: https://rust-lang.github.io/cbindgen/
-- **Project Homepage**: https://kreuzberg.dev
-- **GitHub Repository**: https://github.com/kreuzberg-dev/kreuzberg
-
-## Contributing
-
-We welcome contributions! Please see the main Kreuzberg repository for contribution guidelines.
-
-## License
-
-MIT
+# kreuzberg-ffi
+
+C Foreign Function Interface (FFI) bindings for the Kreuzberg document intelligence library.
+
+## Overview
+
+This crate provides a C-compatible API that bridges the high-performance Rust core of Kreuzberg with multiple programming languages and FFI systems. It is the foundation for language bindings in Java (Panama FFI), Go (cgo), C# (P/Invoke), and other languages with C interoperability.
+
+The FFI exposes extraction functions, configuration management, plugin registration, and error handling through a stable C interface with thread-safe callbacks.
+
+## Architecture
+
+### FFI Bridge Layers
+
+```
+Language-Specific Bindings
+    ↓
+Kreuzberg FFI C Library (crates/kreuzberg-ffi) ← This crate
+    ↓
+Rust Core Library (crates/kreuzberg)
+    ↓
+Document Extraction Engines
+```
+
+### Binding Support
+
+This FFI layer is consumed by:
+
+- **Java** (packages/java): Using Java 25 Foreign Function & Memory API (Panama FFI)
+- **Go** (packages/go): Using cgo wrapper bindings
+- **C#** (packages/csharp): Using P/Invoke interop
+- **Zig** and other C-compatible languages
+
+### Key Components
+
+- **Core Extraction** (`extract_file`, `extract_bytes`): Document text and data extraction
+- **Batch Operations**: Parallel processing of multiple documents
+- **MIME Detection**: File format identification
+- **Configuration Management**: Loading and applying extraction settings
+- **Plugin System**: OCR backend registration and callbacks
+- **Error Handling**: Thread-local error message storage
+- **Memory Management**: Safe pointer handling and FFI boundaries
+
+## Installation
+
+### Build from Source
+
+```bash
+cargo build --release -p kreuzberg-ffi
+```
+
+### Output Artifacts
+
+After building, you will have:
+
+- **Dynamic Library**: `target/release/libkreuzberg_ffi.{so,dylib,dll}`
+  - For loading at runtime
+  - Platform-specific extensions (`.so` Linux, `.dylib` macOS, `.dll` Windows)
+
+- **Static Library**: `target/release/libkreuzberg_ffi.{a,lib}`
+  - For static linking into applications
+  - Platform-specific extensions (`.a` Unix, `.lib` Windows)
+
+- **Header File**: Auto-generated via `cbindgen` during build
+
+### Header Generation
+
+The C header file is automatically generated during the build process via `cbindgen`:
+
+```bash
+cargo build --release -p kreuzberg-ffi
+# Header is generated at build time based on #[no_mangle] functions
+```
+
+For manual header generation:
+
+```bash
+cargo build --features html,embeddings -p kreuzberg-ffi
+```
+
+### pkg-config File Generation
+
+The build process automatically generates pkg-config files for library discovery:
+
+```bash
+cargo build --release -p kreuzberg-ffi
+```
+
+This creates two variants in `crates/kreuzberg-ffi/`:
+- **kreuzberg-ffi.pc**: Development version (prefix points to repository)
+- **kreuzberg-ffi-install.pc**: Installation version (prefix=/usr/local)
+
+The development variant enables monorepo developers to use pkg-config:
+
+```bash
+export PKG_CONFIG_PATH="$PWD/crates/kreuzberg-ffi:$PKG_CONFIG_PATH"
+pkg-config --cflags kreuzberg-ffi  # Returns -I/path/to/repo/crates/kreuzberg-ffi
+pkg-config --libs kreuzberg-ffi    # Returns -L/path/to/repo/target/release -lkreuzberg_ffi
+```
+
+The installation variant is used in release artifacts for third-party use.
+
+### Installing from Release Artifacts
+
+Pre-built binaries are available for Linux, macOS, and Windows (MinGW) from the [releases page](https://github.com/kreuzberg-dev/kreuzberg/releases).
+
+Each `go-ffi-{platform}.tar.gz` archive contains:
+- `lib/`: Shared libraries (kreuzberg-ffi, pdfium, onnxruntime)
+- `include/`: C header file (kreuzberg.h)
+- `share/pkgconfig/`: pkg-config file for library discovery
+- `README.md`: Installation instructions
+
+Installation:
+
+```bash
+# Download and extract
+tar -xzf go-ffi-linux-x86_64.tar.gz
+cd kreuzberg-ffi
+
+# System-wide installation (requires sudo)
+sudo cp -r lib/* /usr/local/lib/
+sudo cp -r include/* /usr/local/include/
+sudo cp -r share/* /usr/local/share/
+sudo ldconfig  # Linux only
+
+# Verify
+pkg-config --modversion kreuzberg-ffi
+```
+
+For user-local installation or custom prefix, see the README.md included in the archive.
+
+## Quick Start: C Example
+
+### Basic Extraction
+
+```c
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+// Include the auto-generated Kreuzberg FFI header
+#include "kreuzberg_ffi.h"
+
+int main() {
+    // Extract text from a PDF file
+    const char* file_path = "document.pdf";
+    const char* mime_type = NULL;  // Auto-detect
+    const char* config_json = "{}";  // Empty config uses defaults
+
+    // Perform extraction
+    ExtractionResult result = kreuzberg_extract_file(
+        file_path,
+        mime_type,
+        config_json
+    );
+
+    // Check for errors
+    if (result.error != NULL) {
+        fprintf(stderr, "Extraction failed: %s\n", result.error);
+        kreuzberg_free_string(result.error);
+        return 1;
+    }
+
+    // Process the result
+    printf("Extracted content:\n%s\n", result.content);
+    printf("MIME Type: %s\n", result.mime_type);
+
+    // Free resources
+    kreuzberg_free_extraction_result(result);
+
+    return 0;
+}
+```
+
+### Compilation
+
+```bash
+# Link against the dynamic library
+gcc -o extract_example extract.c \
+    -L/path/to/kreuzberg/target/release \
+    -lkreuzberg_ffi
+```
+
+## API Reference
+
+### Core Extraction Functions
+
+All functions return results via out-parameters or result structs. Error messages are accessible via `kreuzberg_get_last_error()`.
+
+#### `ExtractionResult kreuzberg_extract_file(const char *path, const char *mime_type, const char *config_json)`
+
+Extract text and metadata from a file.
+
+**Parameters:**
+- `path`: Absolute or relative file path (must exist)
+- `mime_type`: Optional MIME type hint (e.g., "application/pdf"). Pass NULL for auto-detection.
+- `config_json`: JSON string with extraction configuration
+
+**Returns:**
+- `ExtractionResult`: Contains `content` (extracted text), `mime_type`, `metadata` (JSON), and optional `error`
+
+**Example:**
+
+```c
+const char* config = "{\"use_cache\": true, \"enable_quality_processing\": true}";
+ExtractionResult result = kreuzberg_extract_file("document.pdf", NULL, config);
+
+if (result.error) {
+    printf("Error: %s\n", result.error);
+} else {
+    printf("Content: %s\n", result.content);
+    printf("MIME: %s\n", result.mime_type);
+}
+
+kreuzberg_free_extraction_result(result);
+```
+
+#### `ExtractionResult kreuzberg_extract_bytes(const char *data, size_t data_len, const char *mime_type, const char *config_json)`
+
+Extract from a byte buffer (e.g., file in memory).
+
+**Parameters:**
+- `data`: Pointer to byte buffer
+- `data_len`: Buffer length in bytes
+- `mime_type`: Optional MIME type hint (required if auto-detection not desired)
+- `config_json`: JSON extraction configuration
+
+**Returns:**
+- `ExtractionResult`: Same as `extract_file`
+
+**Example:**
+
+```c
+// Read file into buffer
+FILE* f = fopen("document.pdf", "rb");
+fseek(f, 0, SEEK_END);
+size_t size = ftell(f);
+fseek(f, 0, SEEK_SET);
+unsigned char* buffer = malloc(size);
+fread(buffer, 1, size, f);
+fclose(f);
+
+// Extract from buffer
+const char* config = "{}";
+ExtractionResult result = kreuzberg_extract_bytes(
+    (const char*)buffer,
+    size,
+    "application/pdf",
+    config
+);
+
+if (!result.error) {
+    printf("Content: %s\n", result.content);
+}
+
+kreuzberg_free_extraction_result(result);
+free(buffer);
+```
+
+#### `BatchExtractionResult kreuzberg_batch_extract_files(const char **paths, size_t paths_count, const char *config_json)`
+
+Process multiple files in parallel.
+
+**Parameters:**
+- `paths`: Array of file path strings
+- `paths_count`: Number of paths in array
+- `config_json`: JSON extraction configuration (applied to all files)
+
+**Returns:**
+- `BatchExtractionResult`: Contains array of `ExtractionResult` and error status
+
+**Example:**
+
+```c
+const char* files[] = {"doc1.pdf", "doc2.docx", "doc3.xlsx"};
+size_t count = 3;
+
+BatchExtractionResult batch_result = kreuzberg_batch_extract_files(
+    files,
+    count,
+    "{}"
+);
+
+if (batch_result.error) {
+    printf("Batch error: %s\n", batch_result.error);
+} else {
+    for (size_t i = 0; i < batch_result.count; i++) {
+        printf("File %zu: %s\n", i, batch_result.results[i].mime_type);
+    }
+}
+
+kreuzberg_free_batch_extraction_result(batch_result);
+```
+
+#### `char* kreuzberg_detect_mime_type(const char *path, bool use_cache)`
+
+Identify the MIME type of a file using magic bytes and file signatures.
+
+**Parameters:**
+- `path`: File path to analyze
+- `use_cache`: Whether to use cached results for the same path
+
+**Returns:**
+- `char*`: MIME type string (must be freed with `kreuzberg_free_string`)
+
+**Example:**
+
+```c
+char* mime = kreuzberg_detect_mime_type("unknown-file", true);
+if (mime) {
+    printf("MIME Type: %s\n", mime);
+    kreuzberg_free_string(mime);
+}
+```
+
+### Configuration Management
+
+#### `char* kreuzberg_get_default_config_json()`
+
+Get default extraction configuration as JSON.
+
+**Returns:**
+- `char*`: JSON string (must be freed with `kreuzberg_free_string`)
+
+**Example:**
+
+```c
+char* default_config = kreuzberg_get_default_config_json();
+printf("Default config: %s\n", default_config);
+kreuzberg_free_string(default_config);
+```
+
+#### Configuration JSON Schema
+
+```json
+{
+  "use_cache": true,
+  "enable_quality_processing": false,
+  "force_ocr": false,
+  "ocr": {
+    "backend": "tesseract",
+    "language": "eng",
+    "tesseract_config": {
+      "enable_table_detection": true,
+      "psm": 6,
+      "min_confidence": 50.0
+    }
+  },
+  "chunking": {
+    "max_chars": 1000,
+    "max_overlap": 200
+  },
+  "language_detection": {
+    "enabled": false,
+    "min_confidence": 0.8,
+    "detect_multiple": false
+  }
+}
+```
+
+### Plugin System
+
+#### `void kreuzberg_register_ocr_backend(const char *backend_name, OcrBackendCallback callback)`
+
+Register a custom OCR backend implemented in the calling language.
+
+**Parameters:**
+- `backend_name`: Unique identifier for the backend (e.g., "custom-ocr")
+- `callback`: Function pointer to OCR processing implementation
+
+**Example (C):**
+
+```c
+// Define the callback function
+OcrBackendResult custom_ocr_callback(
+    const char* image_data,
+    size_t image_size,
+    const char* language,
+    void* user_data
+) {
+    // Call out to Python, Go, C#, etc. to perform OCR
+    // Implementation depends on the host language
+
+    OcrBackendResult result = {0};
+    result.content = "Extracted text from OCR";
+    result.metadata = "{}";
+    return result;
+}
+
+// Register the backend
+kreuzberg_register_ocr_backend("easyocr", custom_ocr_callback);
+
+// Now use it in extraction config:
+const char* config = "{\"ocr\": {\"backend\": \"easyocr\", \"language\": \"eng\"}}";
+ExtractionResult result = kreuzberg_extract_file("scanned.pdf", NULL, config);
+```
+
+### Error Handling
+
+#### `char* kreuzberg_get_last_error()`
+
+Retrieve the last error message.
+
+**Returns:**
+- `char*`: Error message string (NULL if no error)
+
+Note: Error messages are thread-local and persist until the next Kreuzberg function call.
+
+**Example:**
+
+```c
+ExtractionResult result = kreuzberg_extract_file("invalid.pdf", NULL, "{}");
+
+if (result.error == NULL) {
+    // Also check global error state
+    char* error = kreuzberg_get_last_error();
+    if (error) {
+        printf("Error: %s\n", error);
+    }
+}
+```
+
+### Memory Management
+
+#### `void kreuzberg_free_string(char *ptr)`
+
+Free a string allocated by FFI functions.
+
+**Note:** Only use for strings returned by FFI functions, not for caller-allocated strings.
+
+**Example:**
+
+```c
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+kreuzberg_free_string(mime);
+
+char* error = kreuzberg_get_last_error();
+if (error) {
+    kreuzberg_free_string(error);
+}
+```
+
+#### `void kreuzberg_free_extraction_result(ExtractionResult result)`
+
+Free memory associated with an extraction result.
+
+**Example:**
+
+```c
+ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
+// ... use result ...
+kreuzberg_free_extraction_result(result);
+```
+
+#### `void kreuzberg_free_batch_extraction_result(BatchExtractionResult result)`
+
+Free memory associated with a batch extraction result.
+
+**Example:**
+
+```c
+BatchExtractionResult batch = kreuzberg_batch_extract_files(files, 3, "{}");
+// ... use batch.results ...
+kreuzberg_free_batch_extraction_result(batch);
+```
+
+## Type Definitions
+
+### ExtractionResult
+
+```c
+typedef struct {
+    char* content;           // Extracted text
+    char* mime_type;         // MIME type of document
+    char* metadata;          // JSON metadata string
+    char* error;             // Error message (NULL if success)
+    // ... tables, images, chunks as needed
+} ExtractionResult;
+```
+
+### BatchExtractionResult
+
+```c
+typedef struct {
+    ExtractionResult* results;  // Array of extraction results
+    size_t count;               // Number of results
+    char* error;                // Batch-level error (NULL if success)
+} BatchExtractionResult;
+```
+
+### OcrBackendCallback
+
+```c
+typedef OcrBackendResult (*OcrBackendCallback)(
+    const char* image_data,
+    size_t image_size,
+    const char* language,
+    void* user_data
+);
+```
+
+## FFI Safety Guidelines
+
+### Thread Safety
+
+All FFI functions are **thread-safe**. The Kreuzberg core uses Arc, Mutex, and RwLock for safe concurrent access:
+
+```c
+// Safe to call from multiple threads
+for (int i = 0; i < 10; i++) {
+    pthread_t thread;
+    pthread_create(&thread, NULL, extract_worker, (void*)(intptr_t)i);
+}
+```
+
+### Pointer Validation
+
+The FFI layer validates all input pointers:
+
+```c
+// Safe: NULL pointers are handled gracefully
+ExtractionResult result = kreuzberg_extract_file(NULL, NULL, "{}");
+// Returns error in result.error
+
+// Safe: Invalid paths return errors
+ExtractionResult result = kreuzberg_extract_file("/nonexistent", NULL, "{}");
+// Returns error in result.error
+```
+
+### Memory Lifetime
+
+- **Returned strings** must be freed with `kreuzberg_free_string()`
+- **Result structs** must be freed with appropriate cleanup functions
+- **Input parameters** are copied internally; caller retains ownership
+
+```c
+// Correct: Free returned values
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+kreuzberg_free_string(mime);
+
+// Correct: Input paths can be freed immediately after call
+{
+    char path[256];
+    snprintf(path, sizeof(path), "document.pdf");
+    ExtractionResult result = kreuzberg_extract_file(path, NULL, "{}");
+    // path can be freed here; FFI has already copied it
+}
+
+// Correct: Free result structures
+ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
+// ... use result ...
+kreuzberg_free_extraction_result(result);
+```
+
+### Error Handling Pattern
+
+```c
+ExtractionResult result = kreuzberg_extract_file(path, NULL, config);
+
+// Check return value first
+if (result.error != NULL) {
+    fprintf(stderr, "FFI Error: %s\n", result.error);
+    kreuzberg_free_extraction_result(result);
+    return;
+}
+
+// Process successful result
+printf("Content: %s\n", result.content);
+printf("MIME: %s\n", result.mime_type);
+
+// Always cleanup
+kreuzberg_free_extraction_result(result);
+```
+
+## Building from C
+
+### Static Linking
+
+```bash
+# Build the library
+cargo build --release -p kreuzberg-ffi
+
+# Create your C program
+gcc -c -o myapp.o myapp.c
+
+# Link statically
+gcc -o myapp myapp.o \
+    -L/path/to/kreuzberg/target/release \
+    -l:libkreuzberg_ffi.a \
+    -pthread -ldl  # May need additional system libs
+
+# Run
+./myapp
+```
+
+### Dynamic Linking
+
+```bash
+# Build the library
+cargo build --release -p kreuzberg-ffi
+
+# Create your C program
+gcc -c -o myapp.o myapp.c
+
+# Link dynamically
+gcc -o myapp myapp.o \
+    -L/path/to/kreuzberg/target/release \
+    -lkreuzberg_ffi
+
+# Set library path and run
+export LD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$LD_LIBRARY_PATH
+./myapp
+```
+
+### macOS Considerations
+
+```bash
+# Build
+cargo build --release -p kreuzberg-ffi
+
+# Link
+gcc -o myapp myapp.o \
+    -L/path/to/kreuzberg/target/release \
+    -lkreuzberg_ffi
+
+# Set runtime path
+export DYLD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$DYLD_LIBRARY_PATH
+./myapp
+```
+
+## Language Binding Integration
+
+### Java Integration (Panama FFI)
+
+The FFI is wrapped in Java 25's Foreign Function & Memory API:
+
+```java
+// Java code that calls FFI
+Arena arena = Arena.ofConfined();
+MemorySegment path = arena.allocateUtf8String("document.pdf");
+MemorySegment config = arena.allocateUtf8String("{}");
+
+ExtractionResult result = KreuzbergFFI.extract_file(path, MemorySegment.NULL, config);
+```
+
+### Go Integration (cgo)
+
+The FFI is exposed through cgo bindings:
+
+```go
+// Go code that calls FFI
+C.kreuzberg_extract_file(C.CString("document.pdf"), nil, C.CString("{}"))
+```
+
+### C# Integration (P/Invoke)
+
+The FFI is declared in C# through P/Invoke:
+
+```csharp
+[DllImport("kreuzberg_ffi", CharSet = CharSet.Ansi)]
+private static extern IntPtr kreuzberg_extract_file(
+    string path,
+    string mimeType,
+    string configJson
+);
+```
+
+## Supported Features
+
+### Default Features
+- `html`: HTML to Markdown conversion support
+- `embeddings`: Text embedding extraction via fastembed-rs (requires ONNX Runtime - must be installed separately)
+
+### System Requirements for Embeddings
+
+If using the `embeddings` feature, ONNX Runtime must be installed on the system:
+
+```bash
+# macOS
+brew install onnxruntime
+
+# Ubuntu/Debian
+sudo apt install libonnxruntime libonnxruntime-dev
+
+# Windows (MSVC)
+scoop install onnxruntime
+# OR download from https://github.com/microsoft/onnxruntime/releases
+```
+
+Without ONNX Runtime, embeddings functionality will raise errors at runtime.
+
+### Core Feature (Windows MinGW Compatibility)
+- `core`: Minimal feature set for cross-platform compatibility
+  - Includes: `html` (HTML to Markdown conversion)
+  - Excludes: `embeddings` (ONNX Runtime not available on MinGW)
+  - Use case: Windows Go bindings with MinGW toolchain
+
+### Platform-Specific Build Requirements
+
+**Windows MinGW (Go bindings):**
+
+The Windows ONNX Runtime library only provides MSVC-compatible .lib files. MinGW cannot link against these, requiring the core feature:
+
+```bash
+# Windows MinGW - Use core feature
+cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-gnu --no-default-features --features core
+
+# Windows MSVC - Full features available
+cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-msvc
+
+# Unix (Linux/macOS) - Full features available
+cargo build --release -p kreuzberg-ffi
+```
+
+**Why MinGW Requires core Feature:**
+- ONNX Runtime distributes Windows binaries compiled with MSVC toolchain
+- MSVC .lib files use different name mangling and linking conventions than MinGW
+- MinGW's GNU toolchain cannot consume MSVC import libraries
+- The `core` feature excludes the `embeddings` dependency, which depends on ort-sys (ONNX Runtime)
+- HTML support (via html-to-markdown-rs) is pure Rust and works on all platforms
+
+### Building with Features
+
+```bash
+# Build with HTML and embeddings support (default)
+cargo build --release -p kreuzberg-ffi
+
+# Build with core feature only (Windows MinGW compatibility)
+cargo build --release -p kreuzberg-ffi --no-default-features --features core
+
+# Build without any features (minimal FFI)
+cargo build --release -p kreuzberg-ffi --no-default-features
+```
+
+## Performance Characteristics
+
+- **Single extraction**: 10-100ms (varies by file size and format)
+- **Batch processing**: Near-linear scaling with CPU cores
+- **OCR processing**: 100-500ms per page
+- **Memory overhead**: ~2-5MB per extraction operation
+- **Thread safety**: Zero synchronization overhead on single-threaded extraction
+
+## Key Files
+
+- `src/lib.rs`: FFI function implementations and plugin system
+- `Cargo.toml`: Dependencies and features
+- Generated header: Auto-created by cbindgen during build
+
+## Building
+
+### Development Build
+
+```bash
+cargo build -p kreuzberg-ffi
+```
+
+### Release Build
+
+```bash
+cargo build --release -p kreuzberg-ffi
+```
+
+### With All Features
+
+```bash
+cargo build --release -p kreuzberg-ffi --features html,embeddings
+```
+
+## Testing
+
+```bash
+# Run FFI tests
+cargo test -p kreuzberg-ffi
+
+# With logging
+RUST_LOG=debug cargo test -p kreuzberg-ffi -- --nocapture
+```
+
+## Troubleshooting
+
+### Library Not Found
+
+Ensure the built library is in the linker search path:
+
+```bash
+# Check for built libraries
+ls -la target/release/libkreuzberg_ffi*
+
+# Add to library path
+export LD_LIBRARY_PATH=target/release:$LD_LIBRARY_PATH
+export DYLD_LIBRARY_PATH=target/release:$DYLD_LIBRARY_PATH  # macOS
+```
+
+### Undefined Reference Errors
+
+Ensure you're linking against the FFI library, not the core library:
+
+```bash
+# Correct
+gcc -o app app.c -lkreuzberg_ffi
+
+# Incorrect
+gcc -o app app.c -lkreuzberg  # Wrong library
+```
+
+### Memory Leaks
+
+Always free returned strings and result structures:
+
+```c
+// Problem: Memory leak
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+printf("%s\n", mime);
+// mime not freed!
+
+// Solution: Free the string
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+printf("%s\n", mime);
+kreuzberg_free_string(mime);
+```
+
+### Thread-Local Error Messages
+
+Each thread has its own error message storage. Check both return values and `kreuzberg_get_last_error()`:
+
+```c
+// Safe across threads
+#pragma omp parallel for
+for (int i = 0; i < 10; i++) {
+    ExtractionResult result = kreuzberg_extract_file(files[i], NULL, "{}");
+    if (result.error) {
+        printf("Thread %d error: %s\n", i, result.error);
+    }
+}
+```
+
+## References
+
+- **Kreuzberg Core**: `../kreuzberg/`
+- **C FFI Standards**: https://en.cppreference.com/w/c
+- **cbindgen Documentation**: https://rust-lang.github.io/cbindgen/
+- **Project Homepage**: https://kreuzberg.dev
+- **GitHub Repository**: https://github.com/kreuzberg-dev/kreuzberg
+
+## Contributing
+
+We welcome contributions! Please see the main Kreuzberg repository for contribution guidelines.
+
+## License
+
+MIT