kreuzberg 4.0.0.rc2 → 4.0.0

data/vendor/kreuzberg-ffi/README.md

@@ -0,0 +1,851 @@
+# kreuzberg-ffi
+
+C Foreign Function Interface (FFI) bindings for the Kreuzberg document intelligence library.
+
+## Overview
+
+This crate provides a C-compatible API that bridges the high-performance Rust core of Kreuzberg with multiple programming languages and FFI systems. It is the foundation for language bindings in Java (Panama FFI), Go (cgo), C# (P/Invoke), and other languages with C interoperability.
+
+The FFI exposes extraction functions, configuration management, plugin registration, and error handling through a stable C interface with thread-safe callbacks.
+
+## Architecture
+
+### FFI Bridge Layers
+
+```
+Language-Specific Bindings
+    ↓
+Kreuzberg FFI C Library (crates/kreuzberg-ffi) ← This crate
+    ↓
+Rust Core Library (crates/kreuzberg)
+    ↓
+Document Extraction Engines
+```
+
+### Binding Support
+
+This FFI layer is consumed by:
+
+- **Java** (packages/java): Using Java 25 Foreign Function & Memory API (Panama FFI)
+- **Go** (packages/go): Using cgo wrapper bindings
+- **C#** (packages/csharp): Using P/Invoke interop
+- **Zig** and other C-compatible languages
+
+### Key Components
+
+- **Core Extraction** (`extract_file`, `extract_bytes`): Document text and data extraction
+- **Batch Operations**: Parallel processing of multiple documents
+- **MIME Detection**: File format identification
+- **Configuration Management**: Loading and applying extraction settings
+- **Plugin System**: OCR backend registration and callbacks
+- **Error Handling**: Thread-local error message storage
+- **Memory Management**: Safe pointer handling and FFI boundaries
+
+## Installation
+
+### Build from Source
+
+```bash
+cargo build --release -p kreuzberg-ffi
+```
+
+### Output Artifacts
+
+After building, you will have:
+
+- **Dynamic Library**: `target/release/libkreuzberg_ffi.{so,dylib,dll}`
+  - For loading at runtime
+  - Platform-specific extensions (`.so` Linux, `.dylib` macOS, `.dll` Windows)
+
+- **Static Library**: `target/release/libkreuzberg_ffi.{a,lib}`
+  - For static linking into applications
+  - Platform-specific extensions (`.a` Unix, `.lib` Windows)
+
+- **Header File**: Auto-generated via `cbindgen` during build
+
+### Header Generation
+
+The C header file is automatically generated during the build process via `cbindgen`:
+
+```bash
+cargo build --release -p kreuzberg-ffi
+# Header is generated at build time based on #[no_mangle] functions
+```
+
+For manual header generation:
+
+```bash
+cargo build --features html,embeddings -p kreuzberg-ffi
+```
+
+### pkg-config File Generation
+
+The build process automatically generates pkg-config files for library discovery:
+
+```bash
+cargo build --release -p kreuzberg-ffi
+```
+
+This creates two variants in `crates/kreuzberg-ffi/`:
+- **kreuzberg-ffi.pc**: Development version (prefix points to repository)
+- **kreuzberg-ffi-install.pc**: Installation version (prefix=/usr/local)
+
+The development variant enables monorepo developers to use pkg-config:
+
+```bash
+export PKG_CONFIG_PATH="$PWD/crates/kreuzberg-ffi:$PKG_CONFIG_PATH"
+pkg-config --cflags kreuzberg-ffi  # Returns -I/path/to/repo/crates/kreuzberg-ffi
+pkg-config --libs kreuzberg-ffi    # Returns -L/path/to/repo/target/release -lkreuzberg_ffi
+```
+
+The installation variant is used in release artifacts for third-party use.
+
+### Installing from Release Artifacts
+
+Pre-built binaries are available for Linux, macOS, and Windows (MinGW) from the [releases page](https://github.com/kreuzberg-dev/kreuzberg/releases).
+
+Each `go-ffi-{platform}.tar.gz` archive contains:
+- `lib/`: Shared libraries (kreuzberg-ffi, pdfium, onnxruntime)
+- `include/`: C header file (kreuzberg.h)
+- `share/pkgconfig/`: pkg-config file for library discovery
+- `README.md`: Installation instructions
+
+Installation:
+
+```bash
+# Download and extract
+tar -xzf go-ffi-linux-x86_64.tar.gz
+cd kreuzberg-ffi
+
+# System-wide installation (requires sudo)
+sudo cp -r lib/* /usr/local/lib/
+sudo cp -r include/* /usr/local/include/
+sudo cp -r share/* /usr/local/share/
+sudo ldconfig  # Linux only
+
+# Verify
+pkg-config --modversion kreuzberg-ffi
+```
+
+For user-local installation or custom prefix, see the README.md included in the archive.
+
+## Quick Start: C Example
+
+### Basic Extraction
+
+```c
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+// Include the auto-generated Kreuzberg FFI header
+#include "kreuzberg_ffi.h"
+
+int main() {
+    // Extract text from a PDF file
+    const char* file_path = "document.pdf";
+    const char* mime_type = NULL;  // Auto-detect
+    const char* config_json = "{}";  // Empty config uses defaults
+
+    // Perform extraction
+    ExtractionResult result = kreuzberg_extract_file(
+        file_path,
+        mime_type,
+        config_json
+    );
+
+    // Check for errors
+    if (result.error != NULL) {
+        fprintf(stderr, "Extraction failed: %s\n", result.error);
+        kreuzberg_free_string(result.error);
+        return 1;
+    }
+
+    // Process the result
+    printf("Extracted content:\n%s\n", result.content);
+    printf("MIME Type: %s\n", result.mime_type);
+
+    // Free resources
+    kreuzberg_free_extraction_result(result);
+
+    return 0;
+}
+```
+
+### Compilation
+
+```bash
+# Link against the dynamic library
+gcc -o extract_example extract.c \
+    -L/path/to/kreuzberg/target/release \
+    -lkreuzberg_ffi
+```
+
+## API Reference
+
+### Core Extraction Functions
+
+All functions return results via out-parameters or result structs. Error messages are accessible via `kreuzberg_get_last_error()`.
+
+#### `ExtractionResult kreuzberg_extract_file(const char *path, const char *mime_type, const char *config_json)`
+
+Extract text and metadata from a file.
+
+**Parameters:**
+- `path`: Absolute or relative file path (must exist)
+- `mime_type`: Optional MIME type hint (e.g., "application/pdf"). Pass NULL for auto-detection.
+- `config_json`: JSON string with extraction configuration
+
+**Returns:**
+- `ExtractionResult`: Contains `content` (extracted text), `mime_type`, `metadata` (JSON), and optional `error`
+
+**Example:**
+
+```c
+const char* config = "{\"use_cache\": true, \"enable_quality_processing\": true}";
+ExtractionResult result = kreuzberg_extract_file("document.pdf", NULL, config);
+
+if (result.error) {
+    printf("Error: %s\n", result.error);
+} else {
+    printf("Content: %s\n", result.content);
+    printf("MIME: %s\n", result.mime_type);
+}
+
+kreuzberg_free_extraction_result(result);
+```
+
+#### `ExtractionResult kreuzberg_extract_bytes(const char *data, size_t data_len, const char *mime_type, const char *config_json)`
+
+Extract from a byte buffer (e.g., file in memory).
+
+**Parameters:**
+- `data`: Pointer to byte buffer
+- `data_len`: Buffer length in bytes
+- `mime_type`: Optional MIME type hint (required if auto-detection not desired)
+- `config_json`: JSON extraction configuration
+
+**Returns:**
+- `ExtractionResult`: Same as `extract_file`
+
+**Example:**
+
+```c
+// Read file into buffer
+FILE* f = fopen("document.pdf", "rb");
+fseek(f, 0, SEEK_END);
+size_t size = ftell(f);
+fseek(f, 0, SEEK_SET);
+unsigned char* buffer = malloc(size);
+fread(buffer, 1, size, f);
+fclose(f);
+
+// Extract from buffer
+const char* config = "{}";
+ExtractionResult result = kreuzberg_extract_bytes(
+    (const char*)buffer,
+    size,
+    "application/pdf",
+    config
+);
+
+if (!result.error) {
+    printf("Content: %s\n", result.content);
+}
+
+kreuzberg_free_extraction_result(result);
+free(buffer);
+```
+
+#### `BatchExtractionResult kreuzberg_batch_extract_files(const char **paths, size_t paths_count, const char *config_json)`
+
+Process multiple files in parallel.
+
+**Parameters:**
+- `paths`: Array of file path strings
+- `paths_count`: Number of paths in array
+- `config_json`: JSON extraction configuration (applied to all files)
+
+**Returns:**
+- `BatchExtractionResult`: Contains array of `ExtractionResult` and error status
+
+**Example:**
+
+```c
+const char* files[] = {"doc1.pdf", "doc2.docx", "doc3.xlsx"};
+size_t count = 3;
+
+BatchExtractionResult batch_result = kreuzberg_batch_extract_files(
+    files,
+    count,
+    "{}"
+);
+
+if (batch_result.error) {
+    printf("Batch error: %s\n", batch_result.error);
+} else {
+    for (size_t i = 0; i < batch_result.count; i++) {
+        printf("File %zu: %s\n", i, batch_result.results[i].mime_type);
+    }
+}
+
+kreuzberg_free_batch_extraction_result(batch_result);
+```
+
+#### `char* kreuzberg_detect_mime_type(const char *path, bool use_cache)`
+
+Identify the MIME type of a file using magic bytes and file signatures.
+
+**Parameters:**
+- `path`: File path to analyze
+- `use_cache`: Whether to use cached results for the same path
+
+**Returns:**
+- `char*`: MIME type string (must be freed with `kreuzberg_free_string`)
+
+**Example:**
+
+```c
+char* mime = kreuzberg_detect_mime_type("unknown-file", true);
+if (mime) {
+    printf("MIME Type: %s\n", mime);
+    kreuzberg_free_string(mime);
+}
+```
+
+### Configuration Management
+
+#### `char* kreuzberg_get_default_config_json()`
+
+Get default extraction configuration as JSON.
+
+**Returns:**
+- `char*`: JSON string (must be freed with `kreuzberg_free_string`)
+
+**Example:**
+
+```c
+char* default_config = kreuzberg_get_default_config_json();
+printf("Default config: %s\n", default_config);
+kreuzberg_free_string(default_config);
+```
+
+#### Configuration JSON Schema
+
+```json
+{
+  "use_cache": true,
+  "enable_quality_processing": false,
+  "force_ocr": false,
+  "ocr": {
+    "backend": "tesseract",
+    "language": "eng",
+    "tesseract_config": {
+      "enable_table_detection": true,
+      "psm": 6,
+      "min_confidence": 50.0
+    }
+  },
+  "chunking": {
+    "max_chars": 1000,
+    "max_overlap": 200
+  },
+  "language_detection": {
+    "enabled": false,
+    "min_confidence": 0.8,
+    "detect_multiple": false
+  }
+}
+```
+
+### Plugin System
+
+#### `void kreuzberg_register_ocr_backend(const char *backend_name, OcrBackendCallback callback)`
+
+Register a custom OCR backend implemented in the calling language.
+
+**Parameters:**
+- `backend_name`: Unique identifier for the backend (e.g., "custom-ocr")
+- `callback`: Function pointer to OCR processing implementation
+
+**Example (C):**
+
+```c
+// Define the callback function
+OcrBackendResult custom_ocr_callback(
+    const char* image_data,
+    size_t image_size,
+    const char* language,
+    void* user_data
+) {
+    // Call out to Python, Go, C#, etc. to perform OCR
+    // Implementation depends on the host language
+
+    OcrBackendResult result = {0};
+    result.content = "Extracted text from OCR";
+    result.metadata = "{}";
+    return result;
+}
+
+// Register the backend
+kreuzberg_register_ocr_backend("easyocr", custom_ocr_callback);
+
+// Now use it in extraction config:
+const char* config = "{\"ocr\": {\"backend\": \"easyocr\", \"language\": \"eng\"}}";
+ExtractionResult result = kreuzberg_extract_file("scanned.pdf", NULL, config);
+```
+
+### Error Handling
+
+#### `char* kreuzberg_get_last_error()`
+
+Retrieve the last error message.
+
+**Returns:**
+- `char*`: Error message string (NULL if no error)
+
+Note: Error messages are thread-local and persist until the next Kreuzberg function call.
+
+**Example:**
+
+```c
+ExtractionResult result = kreuzberg_extract_file("invalid.pdf", NULL, "{}");
+
+if (result.error == NULL) {
+    // Also check global error state
+    char* error = kreuzberg_get_last_error();
+    if (error) {
+        printf("Error: %s\n", error);
+    }
+}
+```
+
+### Memory Management
+
+#### `void kreuzberg_free_string(char *ptr)`
+
+Free a string allocated by FFI functions.
+
+**Note:** Only use for strings returned by FFI functions, not for caller-allocated strings.
+
+**Example:**
+
+```c
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+kreuzberg_free_string(mime);
+
+char* error = kreuzberg_get_last_error();
+if (error) {
+    kreuzberg_free_string(error);
+}
+```
+
+#### `void kreuzberg_free_extraction_result(ExtractionResult result)`
+
+Free memory associated with an extraction result.
+
+**Example:**
+
+```c
+ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
+// ... use result ...
+kreuzberg_free_extraction_result(result);
+```
+
+#### `void kreuzberg_free_batch_extraction_result(BatchExtractionResult result)`
+
+Free memory associated with a batch extraction result.
+
+**Example:**
+
+```c
+BatchExtractionResult batch = kreuzberg_batch_extract_files(files, 3, "{}");
+// ... use batch.results ...
+kreuzberg_free_batch_extraction_result(batch);
+```
+
+## Type Definitions
+
+### ExtractionResult
+
+```c
+typedef struct {
+    char* content;           // Extracted text
+    char* mime_type;         // MIME type of document
+    char* metadata;          // JSON metadata string
+    char* error;             // Error message (NULL if success)
+    // ... tables, images, chunks as needed
+} ExtractionResult;
+```
+
+### BatchExtractionResult
+
+```c
+typedef struct {
+    ExtractionResult* results;  // Array of extraction results
+    size_t count;               // Number of results
+    char* error;                // Batch-level error (NULL if success)
+} BatchExtractionResult;
+```
+
+### OcrBackendCallback
+
+```c
+typedef OcrBackendResult (*OcrBackendCallback)(
+    const char* image_data,
+    size_t image_size,
+    const char* language,
+    void* user_data
+);
+```
+
+## FFI Safety Guidelines
+
+### Thread Safety
+
+All FFI functions are **thread-safe**. The Kreuzberg core uses Arc, Mutex, and RwLock for safe concurrent access:
+
+```c
+// Safe to call from multiple threads
+for (int i = 0; i < 10; i++) {
+    pthread_t thread;
+    pthread_create(&thread, NULL, extract_worker, (void*)(intptr_t)i);
+}
+```
+
+### Pointer Validation
+
+The FFI layer validates all input pointers:
+
+```c
+// Safe: NULL pointers are handled gracefully
+ExtractionResult result = kreuzberg_extract_file(NULL, NULL, "{}");
+// Returns error in result.error
+
+// Safe: Invalid paths return errors
+ExtractionResult result = kreuzberg_extract_file("/nonexistent", NULL, "{}");
+// Returns error in result.error
+```
+
+### Memory Lifetime
+
+- **Returned strings** must be freed with `kreuzberg_free_string()`
+- **Result structs** must be freed with appropriate cleanup functions
+- **Input parameters** are copied internally; caller retains ownership
+
+```c
+// Correct: Free returned values
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+kreuzberg_free_string(mime);
+
+// Correct: Input paths can be freed immediately after call
+{
+    char path[256];
+    snprintf(path, sizeof(path), "document.pdf");
+    ExtractionResult result = kreuzberg_extract_file(path, NULL, "{}");
+    // path can be freed here; FFI has already copied it
+}
+
+// Correct: Free result structures
+ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
+// ... use result ...
+kreuzberg_free_extraction_result(result);
+```
+
+### Error Handling Pattern
+
+```c
+ExtractionResult result = kreuzberg_extract_file(path, NULL, config);
+
+// Check return value first
+if (result.error != NULL) {
+    fprintf(stderr, "FFI Error: %s\n", result.error);
+    kreuzberg_free_extraction_result(result);
+    return;
+}
+
+// Process successful result
+printf("Content: %s\n", result.content);
+printf("MIME: %s\n", result.mime_type);
+
+// Always cleanup
+kreuzberg_free_extraction_result(result);
+```
+
+## Building from C
+
+### Static Linking
+
+```bash
+# Build the library
+cargo build --release -p kreuzberg-ffi
+
+# Create your C program
+gcc -c -o myapp.o myapp.c
+
+# Link statically
+gcc -o myapp myapp.o \
+    -L/path/to/kreuzberg/target/release \
+    -l:libkreuzberg_ffi.a \
+    -pthread -ldl  # May need additional system libs
+
+# Run
+./myapp
+```
+
+### Dynamic Linking
+
+```bash
+# Build the library
+cargo build --release -p kreuzberg-ffi
+
+# Create your C program
+gcc -c -o myapp.o myapp.c
+
+# Link dynamically
+gcc -o myapp myapp.o \
+    -L/path/to/kreuzberg/target/release \
+    -lkreuzberg_ffi
+
+# Set library path and run
+export LD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$LD_LIBRARY_PATH
+./myapp
+```
+
+### macOS Considerations
+
+```bash
+# Build
+cargo build --release -p kreuzberg-ffi
+
+# Link
+gcc -o myapp myapp.o \
+    -L/path/to/kreuzberg/target/release \
+    -lkreuzberg_ffi
+
+# Set runtime path
+export DYLD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$DYLD_LIBRARY_PATH
+./myapp
+```
+
+## Language Binding Integration
+
+### Java Integration (Panama FFI)
+
+The FFI is wrapped in Java 25's Foreign Function & Memory API:
+
+```java
+// Java code that calls FFI
+Arena arena = Arena.ofConfined();
+MemorySegment path = arena.allocateUtf8String("document.pdf");
+MemorySegment config = arena.allocateUtf8String("{}");
+
+ExtractionResult result = KreuzbergFFI.extract_file(path, MemorySegment.NULL, config);
+```
+
+### Go Integration (cgo)
+
+The FFI is exposed through cgo bindings:
+
+```go
+// Go code that calls FFI
+C.kreuzberg_extract_file(C.CString("document.pdf"), nil, C.CString("{}"))
+```
+
+### C# Integration (P/Invoke)
+
+The FFI is declared in C# through P/Invoke:
+
+```csharp
+[DllImport("kreuzberg_ffi", CharSet = CharSet.Ansi)]
+private static extern IntPtr kreuzberg_extract_file(
+    string path,
+    string mimeType,
+    string configJson
+);
+```
+
+## Supported Features
+
+### Default Features
+- `html`: HTML to Markdown conversion support
+- `embeddings`: Text embedding extraction via fastembed-rs (requires ONNX Runtime - must be installed separately)
+
+### System Requirements for Embeddings
+
+If using the `embeddings` feature, ONNX Runtime must be installed on the system:
+
+```bash
+# macOS
+brew install onnxruntime
+
+# Ubuntu/Debian
+sudo apt install libonnxruntime libonnxruntime-dev
+
+# Windows (MSVC)
+scoop install onnxruntime
+# OR download from https://github.com/microsoft/onnxruntime/releases
+```
+
+Without ONNX Runtime, embeddings functionality will raise errors at runtime.
+
+### Core Feature (Windows MinGW Compatibility)
+- `core`: Minimal feature set for cross-platform compatibility
+  - Includes: `html` (HTML to Markdown conversion)
+  - Excludes: `embeddings` (ONNX Runtime not available on MinGW)
+  - Use case: Windows Go bindings with MinGW toolchain
+
+### Platform-Specific Build Requirements
+
+**Windows MinGW (Go bindings):**
+
+The Windows ONNX Runtime library only provides MSVC-compatible .lib files. MinGW cannot link against these, requiring the core feature:
+
+```bash
+# Windows MinGW - Use core feature
+cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-gnu --no-default-features --features core
+
+# Windows MSVC - Full features available
+cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-msvc
+
+# Unix (Linux/macOS) - Full features available
+cargo build --release -p kreuzberg-ffi
+```
+
+**Why MinGW Requires core Feature:**
+- ONNX Runtime distributes Windows binaries compiled with MSVC toolchain
+- MSVC .lib files use different name mangling and linking conventions than MinGW
+- MinGW's GNU toolchain cannot consume MSVC import libraries
+- The `core` feature excludes the `embeddings` dependency, which depends on ort-sys (ONNX Runtime)
+- HTML support (via html-to-markdown-rs) is pure Rust and works on all platforms
+
+### Building with Features
+
+```bash
+# Build with HTML and embeddings support (default)
+cargo build --release -p kreuzberg-ffi
+
+# Build with core feature only (Windows MinGW compatibility)
+cargo build --release -p kreuzberg-ffi --no-default-features --features core
+
+# Build without any features (minimal FFI)
+cargo build --release -p kreuzberg-ffi --no-default-features
+```
+
+## Performance Characteristics
+
+- **Single extraction**: 10-100ms (varies by file size and format)
+- **Batch processing**: Near-linear scaling with CPU cores
+- **OCR processing**: 100-500ms per page
+- **Memory overhead**: ~2-5MB per extraction operation
+- **Thread safety**: Zero synchronization overhead on single-threaded extraction
+
+## Key Files
+
+- `src/lib.rs`: FFI function implementations and plugin system
+- `Cargo.toml`: Dependencies and features
+- Generated header: Auto-created by cbindgen during build
+
+## Building
+
+### Development Build
+
+```bash
+cargo build -p kreuzberg-ffi
+```
+
+### Release Build
+
+```bash
+cargo build --release -p kreuzberg-ffi
+```
+
+### With All Features
+
+```bash
+cargo build --release -p kreuzberg-ffi --features html,embeddings
+```
+
+## Testing
+
+```bash
+# Run FFI tests
+cargo test -p kreuzberg-ffi
+
+# With logging
+RUST_LOG=debug cargo test -p kreuzberg-ffi -- --nocapture
+```
+
+## Troubleshooting
+
+### Library Not Found
+
+Ensure the built library is in the linker search path:
+
+```bash
+# Check for built libraries
+ls -la target/release/libkreuzberg_ffi*
+
+# Add to library path
+export LD_LIBRARY_PATH=target/release:$LD_LIBRARY_PATH
+export DYLD_LIBRARY_PATH=target/release:$DYLD_LIBRARY_PATH  # macOS
+```
+
+### Undefined Reference Errors
+
+Ensure you're linking against the FFI library, not the core library:
+
+```bash
+# Correct
+gcc -o app app.c -lkreuzberg_ffi
+
+# Incorrect
+gcc -o app app.c -lkreuzberg  # Wrong library
+```
+
+### Memory Leaks
+
+Always free returned strings and result structures:
+
+```c
+// Problem: Memory leak
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+printf("%s\n", mime);
+// mime not freed!
+
+// Solution: Free the string
+char* mime = kreuzberg_detect_mime_type("file.pdf", true);
+printf("%s\n", mime);
+kreuzberg_free_string(mime);
+```
+
+### Thread-Local Error Messages
+
+Each thread has its own error message storage. Check both return values and `kreuzberg_get_last_error()`:
+
+```c
+// Safe across threads
+#pragma omp parallel for
+for (int i = 0; i < 10; i++) {
+    ExtractionResult result = kreuzberg_extract_file(files[i], NULL, "{}");
+    if (result.error) {
+        printf("Thread %d error: %s\n", i, result.error);
+    }
+}
+```
+
+## References
+
+- **Kreuzberg Core**: `../kreuzberg/`
+- **C FFI Standards**: https://en.cppreference.com/w/c
+- **cbindgen Documentation**: https://rust-lang.github.io/cbindgen/
+- **Project Homepage**: https://kreuzberg.dev
+- **GitHub Repository**: https://github.com/kreuzberg-dev/kreuzberg
+
+## Contributing
+
+We welcome contributions! Please see the main Kreuzberg repository for contribution guidelines.
+
+## License
+
+MIT