kreuzberg 4.0.0.pre.rc.13 → 4.0.0.pre.rc.15

data/vendor/kreuzberg-ffi/README.md

@@ -1,851 +0,0 @@
-# kreuzberg-ffi
-
-C Foreign Function Interface (FFI) bindings for the Kreuzberg document intelligence library.
-
-## Overview
-
-This crate provides a C-compatible API that bridges the high-performance Rust core of Kreuzberg with multiple programming languages and FFI systems. It is the foundation for language bindings in Java (Panama FFI), Go (cgo), C# (P/Invoke), and other languages with C interoperability.
-
-The FFI exposes extraction functions, configuration management, plugin registration, and error handling through a stable C interface with thread-safe callbacks.
-
-## Architecture
-
-### FFI Bridge Layers
-
-```
-Language-Specific Bindings
-    ↓
-Kreuzberg FFI C Library (crates/kreuzberg-ffi) ← This crate
-    ↓
-Rust Core Library (crates/kreuzberg)
-    ↓
-Document Extraction Engines
-```
-
-### Binding Support
-
-This FFI layer is consumed by:
-
-- **Java** (packages/java): Using Java 25 Foreign Function & Memory API (Panama FFI)
-- **Go** (packages/go): Using cgo wrapper bindings
-- **C#** (packages/csharp): Using P/Invoke interop
-- **Zig** and other C-compatible languages
-
-### Key Components
-
-- **Core Extraction** (`extract_file`, `extract_bytes`): Document text and data extraction
-- **Batch Operations**: Parallel processing of multiple documents
-- **MIME Detection**: File format identification
-- **Configuration Management**: Loading and applying extraction settings
-- **Plugin System**: OCR backend registration and callbacks
-- **Error Handling**: Thread-local error message storage
-- **Memory Management**: Safe pointer handling and FFI boundaries
-
-## Installation
-
-### Build from Source
-
-```bash
-cargo build --release -p kreuzberg-ffi
-```
-
-### Output Artifacts
-
-After building, you will have:
-
-- **Dynamic Library**: `target/release/libkreuzberg_ffi.{so,dylib,dll}`
-  - For loading at runtime
-  - Platform-specific extensions (`.so` Linux, `.dylib` macOS, `.dll` Windows)
-
-- **Static Library**: `target/release/libkreuzberg_ffi.{a,lib}`
-  - For static linking into applications
-  - Platform-specific extensions (`.a` Unix, `.lib` Windows)
-
-- **Header File**: Auto-generated via `cbindgen` during build
-
-### Header Generation
-
-The C header file is automatically generated during the build process via `cbindgen`:
-
-```bash
-cargo build --release -p kreuzberg-ffi
-# Header is generated at build time based on #[no_mangle] functions
-```
-
-For manual header generation:
-
-```bash
-cargo build --features html,embeddings -p kreuzberg-ffi
-```
-
-### pkg-config File Generation
-
-The build process automatically generates pkg-config files for library discovery:
-
-```bash
-cargo build --release -p kreuzberg-ffi
-```
-
-This creates two variants in `crates/kreuzberg-ffi/`:
-- **kreuzberg-ffi.pc**: Development version (prefix points to repository)
-- **kreuzberg-ffi-install.pc**: Installation version (prefix=/usr/local)
-
-The development variant enables monorepo developers to use pkg-config:
-
-```bash
-export PKG_CONFIG_PATH="$PWD/crates/kreuzberg-ffi:$PKG_CONFIG_PATH"
-pkg-config --cflags kreuzberg-ffi  # Returns -I/path/to/repo/crates/kreuzberg-ffi
-pkg-config --libs kreuzberg-ffi    # Returns -L/path/to/repo/target/release -lkreuzberg_ffi
-```
-
-The installation variant is used in release artifacts for third-party use.
-
-### Installing from Release Artifacts
-
-Pre-built binaries are available for Linux, macOS, and Windows (MinGW) from the [releases page](https://github.com/kreuzberg-dev/kreuzberg/releases).
-
-Each `go-ffi-{platform}.tar.gz` archive contains:
-- `lib/`: Shared libraries (kreuzberg-ffi, pdfium, onnxruntime)
-- `include/`: C header file (kreuzberg.h)
-- `share/pkgconfig/`: pkg-config file for library discovery
-- `README.md`: Installation instructions
-
-Installation:
-
-```bash
-# Download and extract
-tar -xzf go-ffi-linux-x86_64.tar.gz
-cd kreuzberg-ffi
-
-# System-wide installation (requires sudo)
-sudo cp -r lib/* /usr/local/lib/
-sudo cp -r include/* /usr/local/include/
-sudo cp -r share/* /usr/local/share/
-sudo ldconfig  # Linux only
-
-# Verify
-pkg-config --modversion kreuzberg-ffi
-```
-
-For user-local installation or custom prefix, see the README.md included in the archive.
-
-## Quick Start: C Example
-
-### Basic Extraction
-
-```c
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-// Include the auto-generated Kreuzberg FFI header
-#include "kreuzberg_ffi.h"
-
-int main() {
-    // Extract text from a PDF file
-    const char* file_path = "document.pdf";
-    const char* mime_type = NULL;  // Auto-detect
-    const char* config_json = "{}";  // Empty config uses defaults
-
-    // Perform extraction
-    ExtractionResult result = kreuzberg_extract_file(
-        file_path,
-        mime_type,
-        config_json
-    );
-
-    // Check for errors
-    if (result.error != NULL) {
-        fprintf(stderr, "Extraction failed: %s\n", result.error);
-        kreuzberg_free_string(result.error);
-        return 1;
-    }
-
-    // Process the result
-    printf("Extracted content:\n%s\n", result.content);
-    printf("MIME Type: %s\n", result.mime_type);
-
-    // Free resources
-    kreuzberg_free_extraction_result(result);
-
-    return 0;
-}
-```
-
-### Compilation
-
-```bash
-# Link against the dynamic library
-gcc -o extract_example extract.c \
-    -L/path/to/kreuzberg/target/release \
-    -lkreuzberg_ffi
-```
-
-## API Reference
-
-### Core Extraction Functions
-
-All functions return results via out-parameters or result structs. Error messages are accessible via `kreuzberg_get_last_error()`.
-
-#### `ExtractionResult kreuzberg_extract_file(const char *path, const char *mime_type, const char *config_json)`
-
-Extract text and metadata from a file.
-
-**Parameters:**
-- `path`: Absolute or relative file path (must exist)
-- `mime_type`: Optional MIME type hint (e.g., "application/pdf"). Pass NULL for auto-detection.
-- `config_json`: JSON string with extraction configuration
-
-**Returns:**
-- `ExtractionResult`: Contains `content` (extracted text), `mime_type`, `metadata` (JSON), and optional `error`
-
-**Example:**
-
-```c
-const char* config = "{\"use_cache\": true, \"enable_quality_processing\": true}";
-ExtractionResult result = kreuzberg_extract_file("document.pdf", NULL, config);
-
-if (result.error) {
-    printf("Error: %s\n", result.error);
-} else {
-    printf("Content: %s\n", result.content);
-    printf("MIME: %s\n", result.mime_type);
-}
-
-kreuzberg_free_extraction_result(result);
-```
-
-#### `ExtractionResult kreuzberg_extract_bytes(const char *data, size_t data_len, const char *mime_type, const char *config_json)`
-
-Extract from a byte buffer (e.g., file in memory).
-
-**Parameters:**
-- `data`: Pointer to byte buffer
-- `data_len`: Buffer length in bytes
-- `mime_type`: Optional MIME type hint (required if auto-detection not desired)
-- `config_json`: JSON extraction configuration
-
-**Returns:**
-- `ExtractionResult`: Same as `extract_file`
-
-**Example:**
-
-```c
-// Read file into buffer
-FILE* f = fopen("document.pdf", "rb");
-fseek(f, 0, SEEK_END);
-size_t size = ftell(f);
-fseek(f, 0, SEEK_SET);
-unsigned char* buffer = malloc(size);
-fread(buffer, 1, size, f);
-fclose(f);
-
-// Extract from buffer
-const char* config = "{}";
-ExtractionResult result = kreuzberg_extract_bytes(
-    (const char*)buffer,
-    size,
-    "application/pdf",
-    config
-);
-
-if (!result.error) {
-    printf("Content: %s\n", result.content);
-}
-
-kreuzberg_free_extraction_result(result);
-free(buffer);
-```
-
-#### `BatchExtractionResult kreuzberg_batch_extract_files(const char **paths, size_t paths_count, const char *config_json)`
-
-Process multiple files in parallel.
-
-**Parameters:**
-- `paths`: Array of file path strings
-- `paths_count`: Number of paths in array
-- `config_json`: JSON extraction configuration (applied to all files)
-
-**Returns:**
-- `BatchExtractionResult`: Contains array of `ExtractionResult` and error status
-
-**Example:**
-
-```c
-const char* files[] = {"doc1.pdf", "doc2.docx", "doc3.xlsx"};
-size_t count = 3;
-
-BatchExtractionResult batch_result = kreuzberg_batch_extract_files(
-    files,
-    count,
-    "{}"
-);
-
-if (batch_result.error) {
-    printf("Batch error: %s\n", batch_result.error);
-} else {
-    for (size_t i = 0; i < batch_result.count; i++) {
-        printf("File %zu: %s\n", i, batch_result.results[i].mime_type);
-    }
-}
-
-kreuzberg_free_batch_extraction_result(batch_result);
-```
-
-#### `char* kreuzberg_detect_mime_type(const char *path, bool use_cache)`
-
-Identify the MIME type of a file using magic bytes and file signatures.
-
-**Parameters:**
-- `path`: File path to analyze
-- `use_cache`: Whether to use cached results for the same path
-
-**Returns:**
-- `char*`: MIME type string (must be freed with `kreuzberg_free_string`)
-
-**Example:**
-
-```c
-char* mime = kreuzberg_detect_mime_type("unknown-file", true);
-if (mime) {
-    printf("MIME Type: %s\n", mime);
-    kreuzberg_free_string(mime);
-}
-```
-
-### Configuration Management
-
-#### `char* kreuzberg_get_default_config_json()`
-
-Get default extraction configuration as JSON.
-
-**Returns:**
-- `char*`: JSON string (must be freed with `kreuzberg_free_string`)
-
-**Example:**
-
-```c
-char* default_config = kreuzberg_get_default_config_json();
-printf("Default config: %s\n", default_config);
-kreuzberg_free_string(default_config);
-```
-
-#### Configuration JSON Schema
-
-```json
-{
-  "use_cache": true,
-  "enable_quality_processing": false,
-  "force_ocr": false,
-  "ocr": {
-    "backend": "tesseract",
-    "language": "eng",
-    "tesseract_config": {
-      "enable_table_detection": true,
-      "psm": 6,
-      "min_confidence": 50.0
-    }
-  },
-  "chunking": {
-    "max_chars": 1000,
-    "max_overlap": 200
-  },
-  "language_detection": {
-    "enabled": false,
-    "min_confidence": 0.8,
-    "detect_multiple": false
-  }
-}
-```
-
-### Plugin System
-
-#### `void kreuzberg_register_ocr_backend(const char *backend_name, OcrBackendCallback callback)`
-
-Register a custom OCR backend implemented in the calling language.
-
-**Parameters:**
-- `backend_name`: Unique identifier for the backend (e.g., "custom-ocr")
-- `callback`: Function pointer to OCR processing implementation
-
-**Example (C):**
-
-```c
-// Define the callback function
-OcrBackendResult custom_ocr_callback(
-    const char* image_data,
-    size_t image_size,
-    const char* language,
-    void* user_data
-) {
-    // Call out to Python, Go, C#, etc. to perform OCR
-    // Implementation depends on the host language
-
-    OcrBackendResult result = {0};
-    result.content = "Extracted text from OCR";
-    result.metadata = "{}";
-    return result;
-}
-
-// Register the backend
-kreuzberg_register_ocr_backend("easyocr", custom_ocr_callback);
-
-// Now use it in extraction config:
-const char* config = "{\"ocr\": {\"backend\": \"easyocr\", \"language\": \"eng\"}}";
-ExtractionResult result = kreuzberg_extract_file("scanned.pdf", NULL, config);
-```
-
-### Error Handling
-
-#### `char* kreuzberg_get_last_error()`
-
-Retrieve the last error message.
-
-**Returns:**
-- `char*`: Error message string (NULL if no error)
-
-Note: Error messages are thread-local and persist until the next Kreuzberg function call.
-
-**Example:**
-
-```c
-ExtractionResult result = kreuzberg_extract_file("invalid.pdf", NULL, "{}");
-
-if (result.error == NULL) {
-    // Also check global error state
-    char* error = kreuzberg_get_last_error();
-    if (error) {
-        printf("Error: %s\n", error);
-    }
-}
-```
-
-### Memory Management
-
-#### `void kreuzberg_free_string(char *ptr)`
-
-Free a string allocated by FFI functions.
-
-**Note:** Only use for strings returned by FFI functions, not for caller-allocated strings.
-
-**Example:**
-
-```c
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-kreuzberg_free_string(mime);
-
-char* error = kreuzberg_get_last_error();
-if (error) {
-    kreuzberg_free_string(error);
-}
-```
-
-#### `void kreuzberg_free_extraction_result(ExtractionResult result)`
-
-Free memory associated with an extraction result.
-
-**Example:**
-
-```c
-ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
-// ... use result ...
-kreuzberg_free_extraction_result(result);
-```
-
-#### `void kreuzberg_free_batch_extraction_result(BatchExtractionResult result)`
-
-Free memory associated with a batch extraction result.
-
-**Example:**
-
-```c
-BatchExtractionResult batch = kreuzberg_batch_extract_files(files, 3, "{}");
-// ... use batch.results ...
-kreuzberg_free_batch_extraction_result(batch);
-```
-
-## Type Definitions
-
-### ExtractionResult
-
-```c
-typedef struct {
-    char* content;           // Extracted text
-    char* mime_type;         // MIME type of document
-    char* metadata;          // JSON metadata string
-    char* error;             // Error message (NULL if success)
-    // ... tables, images, chunks as needed
-} ExtractionResult;
-```
-
-### BatchExtractionResult
-
-```c
-typedef struct {
-    ExtractionResult* results;  // Array of extraction results
-    size_t count;               // Number of results
-    char* error;                // Batch-level error (NULL if success)
-} BatchExtractionResult;
-```
-
-### OcrBackendCallback
-
-```c
-typedef OcrBackendResult (*OcrBackendCallback)(
-    const char* image_data,
-    size_t image_size,
-    const char* language,
-    void* user_data
-);
-```
-
-## FFI Safety Guidelines
-
-### Thread Safety
-
-All FFI functions are **thread-safe**. The Kreuzberg core uses Arc, Mutex, and RwLock for safe concurrent access:
-
-```c
-// Safe to call from multiple threads
-for (int i = 0; i < 10; i++) {
-    pthread_t thread;
-    pthread_create(&thread, NULL, extract_worker, (void*)(intptr_t)i);
-}
-```
-
-### Pointer Validation
-
-The FFI layer validates all input pointers:
-
-```c
-// Safe: NULL pointers are handled gracefully
-ExtractionResult result = kreuzberg_extract_file(NULL, NULL, "{}");
-// Returns error in result.error
-
-// Safe: Invalid paths return errors
-ExtractionResult result = kreuzberg_extract_file("/nonexistent", NULL, "{}");
-// Returns error in result.error
-```
-
-### Memory Lifetime
-
-- **Returned strings** must be freed with `kreuzberg_free_string()`
-- **Result structs** must be freed with appropriate cleanup functions
-- **Input parameters** are copied internally; caller retains ownership
-
-```c
-// Correct: Free returned values
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-kreuzberg_free_string(mime);
-
-// Correct: Input paths can be freed immediately after call
-{
-    char path[256];
-    snprintf(path, sizeof(path), "document.pdf");
-    ExtractionResult result = kreuzberg_extract_file(path, NULL, "{}");
-    // path can be freed here; FFI has already copied it
-}
-
-// Correct: Free result structures
-ExtractionResult result = kreuzberg_extract_file("doc.pdf", NULL, "{}");
-// ... use result ...
-kreuzberg_free_extraction_result(result);
-```
-
-### Error Handling Pattern
-
-```c
-ExtractionResult result = kreuzberg_extract_file(path, NULL, config);
-
-// Check return value first
-if (result.error != NULL) {
-    fprintf(stderr, "FFI Error: %s\n", result.error);
-    kreuzberg_free_extraction_result(result);
-    return;
-}
-
-// Process successful result
-printf("Content: %s\n", result.content);
-printf("MIME: %s\n", result.mime_type);
-
-// Always cleanup
-kreuzberg_free_extraction_result(result);
-```
-
-## Building from C
-
-### Static Linking
-
-```bash
-# Build the library
-cargo build --release -p kreuzberg-ffi
-
-# Create your C program
-gcc -c -o myapp.o myapp.c
-
-# Link statically
-gcc -o myapp myapp.o \
-    -L/path/to/kreuzberg/target/release \
-    -l:libkreuzberg_ffi.a \
-    -pthread -ldl  # May need additional system libs
-
-# Run
-./myapp
-```
-
-### Dynamic Linking
-
-```bash
-# Build the library
-cargo build --release -p kreuzberg-ffi
-
-# Create your C program
-gcc -c -o myapp.o myapp.c
-
-# Link dynamically
-gcc -o myapp myapp.o \
-    -L/path/to/kreuzberg/target/release \
-    -lkreuzberg_ffi
-
-# Set library path and run
-export LD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$LD_LIBRARY_PATH
-./myapp
-```
-
-### macOS Considerations
-
-```bash
-# Build
-cargo build --release -p kreuzberg-ffi
-
-# Link
-gcc -o myapp myapp.o \
-    -L/path/to/kreuzberg/target/release \
-    -lkreuzberg_ffi
-
-# Set runtime path
-export DYLD_LIBRARY_PATH=/path/to/kreuzberg/target/release:$DYLD_LIBRARY_PATH
-./myapp
-```
-
-## Language Binding Integration
-
-### Java Integration (Panama FFI)
-
-The FFI is wrapped in Java 25's Foreign Function & Memory API:
-
-```java
-// Java code that calls FFI
-Arena arena = Arena.ofConfined();
-MemorySegment path = arena.allocateUtf8String("document.pdf");
-MemorySegment config = arena.allocateUtf8String("{}");
-
-ExtractionResult result = KreuzbergFFI.extract_file(path, MemorySegment.NULL, config);
-```
-
-### Go Integration (cgo)
-
-The FFI is exposed through cgo bindings:
-
-```go
-// Go code that calls FFI
-C.kreuzberg_extract_file(C.CString("document.pdf"), nil, C.CString("{}"))
-```
-
-### C# Integration (P/Invoke)
-
-The FFI is declared in C# through P/Invoke:
-
-```csharp
-[DllImport("kreuzberg_ffi", CharSet = CharSet.Ansi)]
-private static extern IntPtr kreuzberg_extract_file(
-    string path,
-    string mimeType,
-    string configJson
-);
-```
-
-## Supported Features
-
-### Default Features
-- `html`: HTML to Markdown conversion support
-- `embeddings`: Text embedding extraction via fastembed-rs (requires ONNX Runtime - must be installed separately)
-
-### System Requirements for Embeddings
-
-If using the `embeddings` feature, ONNX Runtime must be installed on the system:
-
-```bash
-# macOS
-brew install onnxruntime
-
-# Ubuntu/Debian
-sudo apt install libonnxruntime libonnxruntime-dev
-
-# Windows (MSVC)
-scoop install onnxruntime
-# OR download from https://github.com/microsoft/onnxruntime/releases
-```
-
-Without ONNX Runtime, embeddings functionality will raise errors at runtime.
-
-### Core Feature (Windows MinGW Compatibility)
-- `core`: Minimal feature set for cross-platform compatibility
-  - Includes: `html` (HTML to Markdown conversion)
-  - Excludes: `embeddings` (ONNX Runtime not available on MinGW)
-  - Use case: Windows Go bindings with MinGW toolchain
-
-### Platform-Specific Build Requirements
-
-**Windows MinGW (Go bindings):**
-
-The Windows ONNX Runtime library only provides MSVC-compatible .lib files. MinGW cannot link against these, requiring the core feature:
-
-```bash
-# Windows MinGW - Use core feature
-cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-gnu --no-default-features --features core
-
-# Windows MSVC - Full features available
-cargo build --release -p kreuzberg-ffi --target x86_64-pc-windows-msvc
-
-# Unix (Linux/macOS) - Full features available
-cargo build --release -p kreuzberg-ffi
-```
-
-**Why MinGW Requires core Feature:**
-- ONNX Runtime distributes Windows binaries compiled with MSVC toolchain
-- MSVC .lib files use different name mangling and linking conventions than MinGW
-- MinGW's GNU toolchain cannot consume MSVC import libraries
-- The `core` feature excludes the `embeddings` dependency, which depends on ort-sys (ONNX Runtime)
-- HTML support (via html-to-markdown-rs) is pure Rust and works on all platforms
-
-### Building with Features
-
-```bash
-# Build with HTML and embeddings support (default)
-cargo build --release -p kreuzberg-ffi
-
-# Build with core feature only (Windows MinGW compatibility)
-cargo build --release -p kreuzberg-ffi --no-default-features --features core
-
-# Build without any features (minimal FFI)
-cargo build --release -p kreuzberg-ffi --no-default-features
-```
-
-## Performance Characteristics
-
-- **Single extraction**: 10-100ms (varies by file size and format)
-- **Batch processing**: Near-linear scaling with CPU cores
-- **OCR processing**: 100-500ms per page
-- **Memory overhead**: ~2-5MB per extraction operation
-- **Thread safety**: Zero synchronization overhead on single-threaded extraction
-
-## Key Files
-
-- `src/lib.rs`: FFI function implementations and plugin system
-- `Cargo.toml`: Dependencies and features
-- Generated header: Auto-created by cbindgen during build
-
-## Building
-
-### Development Build
-
-```bash
-cargo build -p kreuzberg-ffi
-```
-
-### Release Build
-
-```bash
-cargo build --release -p kreuzberg-ffi
-```
-
-### With All Features
-
-```bash
-cargo build --release -p kreuzberg-ffi --features html,embeddings
-```
-
-## Testing
-
-```bash
-# Run FFI tests
-cargo test -p kreuzberg-ffi
-
-# With logging
-RUST_LOG=debug cargo test -p kreuzberg-ffi -- --nocapture
-```
-
-## Troubleshooting
-
-### Library Not Found
-
-Ensure the built library is in the linker search path:
-
-```bash
-# Check for built libraries
-ls -la target/release/libkreuzberg_ffi*
-
-# Add to library path
-export LD_LIBRARY_PATH=target/release:$LD_LIBRARY_PATH
-export DYLD_LIBRARY_PATH=target/release:$DYLD_LIBRARY_PATH  # macOS
-```
-
-### Undefined Reference Errors
-
-Ensure you're linking against the FFI library, not the core library:
-
-```bash
-# Correct
-gcc -o app app.c -lkreuzberg_ffi
-
-# Incorrect
-gcc -o app app.c -lkreuzberg  # Wrong library
-```
-
-### Memory Leaks
-
-Always free returned strings and result structures:
-
-```c
-// Problem: Memory leak
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-printf("%s\n", mime);
-// mime not freed!
-
-// Solution: Free the string
-char* mime = kreuzberg_detect_mime_type("file.pdf", true);
-printf("%s\n", mime);
-kreuzberg_free_string(mime);
-```
-
-### Thread-Local Error Messages
-
-Each thread has its own error message storage. Check both return values and `kreuzberg_get_last_error()`:
-
-```c
-// Safe across threads
-#pragma omp parallel for
-for (int i = 0; i < 10; i++) {
-    ExtractionResult result = kreuzberg_extract_file(files[i], NULL, "{}");
-    if (result.error) {
-        printf("Thread %d error: %s\n", i, result.error);
-    }
-}
-```
-
-## References
-
-- **Kreuzberg Core**: `../kreuzberg/`
-- **C FFI Standards**: https://en.cppreference.com/w/c
-- **cbindgen Documentation**: https://rust-lang.github.io/cbindgen/
-- **Project Homepage**: https://kreuzberg.dev
-- **GitHub Repository**: https://github.com/kreuzberg-dev/kreuzberg
-
-## Contributing
-
-We welcome contributions! Please see the main Kreuzberg repository for contribution guidelines.
-
-## License
-
-MIT