@xberg-io/opencode-xberg 1.3.6 → 1.3.8

package/.opencode/skills/xberg/references/rust-api.md

@@ -0,0 +1,412 @@
+<!--
+AI-RULEZ :: GENERATED FILE — DO NOT EDIT
+Content-Hash: blake3:a6ececeeda3031d9d84eb780fcb5ecb41b3f4297fa1e6b86f492e0d755b084a9
+Source-Hash: blake3:3fca38eb564d0ed9c9194d070fd8dec96de727edc92e82954cb4b9af52cddbaf
+Schema-Version: v1
+-->
+
+# Xberg Rust API Reference
+
+Complete API reference for the Xberg document extraction library in Rust. Xberg is the Rust core that powers every language binding.
+
+## Setup
+
+```bash
+cargo add xberg
+```
+
+The crate version is `1.1.0`. Add it to `Cargo.toml`:
+
+```toml
+[dependencies]
+xberg = { version = "1.1.0", features = ["full"] }
+tokio = { version = "1", features = ["full"] }
+```
+
+`extract` / `extract_batch` are async and need a Tokio runtime, which you bring yourself (`#[tokio::main]` or an existing runtime).
+
+### Feature flags
+
+The `tokio-runtime` feature is on by default. Capabilities are gated behind features:
+
+| Feature              | What it enables                                            |
+| -------------------- | --------------------------------------------------------- |
+| `tokio-runtime`      | Async runtime support (default).                          |
+| `formats`            | All document formats (no OCR/ML/api/mcp/chunking).        |
+| `full`               | `formats` + ocr + layout + embeddings + tree-sitter + LLM. |
+| `pdf`                | PDF extraction.                                           |
+| `ocr`                | Tesseract-based OCR.                                      |
+| `chunking`           | Text chunking for RAG pipelines.                         |
+| `embeddings`         | ONNX embedding generation.                                |
+| `language-detection` | Detect document language(s).                              |
+| `keywords`           | YAKE + RAKE keyword extraction.                          |
+| `api`                | HTTP API server (Axum).                                  |
+| `mcp`                | Model Context Protocol server.                            |
+
+---
+
+## Core Extraction Functions
+
+There are exactly two entry points. Both are `async`, take an [`ExtractInput`](#extractinput) and a `&ExtractionConfig`, and return an [`ExtractionResult`](#extractionresult-envelope) **envelope**.
+
+### `extract`
+
+```rust
+pub async fn extract(input: ExtractInput, config: &ExtractionConfig) -> Result<ExtractionResult>
+```
+
+```rust
+use xberg::{extract, ExtractInput, ExtractionConfig};
+
+#[tokio::main]
+async fn main() -> xberg::Result<()> {
+    let config = ExtractionConfig {
+        use_cache: true,
+        enable_quality_processing: true,
+        ..Default::default()
+    };
+
+    let output = extract(ExtractInput::from_uri("document.pdf"), &config).await?;
+    println!("{}", output.results[0].content);
+    println!("MIME type: {}", output.results[0].mime_type);
+    Ok(())
+}
+```
+
+### `extract_batch`
+
+```rust
+pub async fn extract_batch(inputs: Vec<ExtractInput>, config: &ExtractionConfig) -> Result<ExtractionResult>
+```
+
+Extract multiple inputs in parallel. The returned envelope's `results` holds one `ExtractedDocument` per input (in input order), with non-fatal failures in `errors`.
+
+```rust
+use xberg::{extract_batch, ExtractInput, ExtractionConfig};
+
+#[tokio::main]
+async fn main() -> xberg::Result<()> {
+    let config = ExtractionConfig::default();
+    let inputs = vec![
+        ExtractInput::from_uri("document.pdf"),
+        ExtractInput::from_bytes(
+            b"Hello from memory".to_vec(),
+            "text/plain",
+            Some("note.txt".to_string()),
+        ),
+    ];
+
+    let output = extract_batch(inputs, &config).await?;
+    for document in &output.results {
+        println!("{}", document.content);
+    }
+    Ok(())
+}
+```
+
+---
+
+## ExtractInput
+
+Unified input for both entry points.
+
+```rust
+pub struct ExtractInput {
+    pub kind: ExtractInputKind,            // Bytes | Uri
+    pub bytes: Option<Vec<u8>>,            // for kind = Bytes
+    pub uri: Option<String>,               // path, file:// URI, or HTTP(S) URL
+    pub mime_type: Option<String>,         // MIME type hint
+    pub filename: Option<String>,          // filename hint
+    pub config: Option<FileExtractionConfig>, // per-input overrides
+}
+```
+
+Constructors:
+
+```rust
+// From a path, file:// URI, or URL
+let input = ExtractInput::from_uri("document.pdf");
+
+// From raw bytes (mime_type required, filename optional)
+let input = ExtractInput::from_bytes(bytes.to_vec(), "application/pdf", Some("document.pdf".to_string()));
+```
+
+---
+
+## Configuration
+
+### `ExtractionConfig`
+
+Construct with a struct literal and `..Default::default()`. Common fields:
+
+```rust
+pub struct ExtractionConfig {
+    pub use_cache: bool,                   // default: true
+    pub enable_quality_processing: bool,   // default: true
+    pub ocr: Option<OcrConfig>,            // None = OCR disabled
+    pub force_ocr: bool,                   // default: false
+    pub chunking: Option<ChunkingConfig>,  // None = disabled
+    pub images: Option<ImageExtractionConfig>,
+    pub pdf_options: Option<PdfConfig>,
+    pub token_reduction: Option<TokenReductionOptions>,
+    pub language_detection: Option<LanguageDetectionConfig>,
+    pub pages: Option<PageConfig>,
+    pub keywords: Option<KeywordConfig>,
+    pub postprocessor: Option<PostProcessorConfig>,
+    pub max_concurrent_extractions: Option<usize>,
+    pub result_format: ResultFormat,       // Unified | ElementBased
+    pub output_format: OutputFormat,       // Plain | Markdown | Djot | Html | Json | DocTags | Custom
+    pub security_limits: Option<SecurityLimits>,
+    // ... additional optional fields
+}
+```
+
+```rust
+use xberg::{extract, ChunkingConfig, ExtractionConfig, ExtractInput, OcrConfig, TesseractConfig};
+
+#[tokio::main]
+async fn main() -> xberg::Result<()> {
+    let config = ExtractionConfig {
+        use_cache: true,
+        ocr: Some(OcrConfig {
+            backend: "tesseract".to_string(),
+            language: vec!["eng".to_string(), "deu".to_string()],
+            tesseract_config: Some(TesseractConfig { psm: 6, ..Default::default() }),
+            ..Default::default()
+        }),
+        chunking: Some(ChunkingConfig {
+            max_characters: 1000,
+            overlap: 200,
+            ..Default::default()
+        }),
+        ..Default::default()
+    };
+
+    let output = extract(ExtractInput::from_uri("document.pdf"), &config).await?;
+    println!("Content length: {}", output.results[0].content.len());
+    Ok(())
+}
+```
+
+### `OcrConfig`
+
+```rust
+pub struct OcrConfig {
+    pub enabled: bool,                       // default: true
+    pub backend: String,                     // "tesseract", "paddleocr", "vlm"
+    pub language: Vec<String>,               // ["eng"], ["eng", "fra"], ...
+    pub tesseract_config: Option<TesseractConfig>,
+    // ...
+}
+```
+
+### `ChunkingConfig`
+
+```rust
+pub struct ChunkingConfig {
+    pub max_characters: usize,    // default: 1000
+    pub overlap: usize,           // default: 200
+    pub trim: bool,               // default: true
+    pub chunker_type: ChunkerType,  // Text | Markdown | Semantic
+    pub embedding: Option<EmbeddingConfig>,
+    pub preset: Option<String>,
+    // ...
+}
+```
+
+The chunk-size fields are `max_characters` and `overlap`.
+
+```rust
+use xberg::{extract, ChunkingConfig, ChunkerType, ExtractInput, ExtractionConfig};
+
+let config = ExtractionConfig {
+    chunking: Some(ChunkingConfig {
+        max_characters: 512,
+        overlap: 50,
+        chunker_type: ChunkerType::Markdown,
+        ..Default::default()
+    }),
+    ..Default::default()
+};
+```
+
+### Output formats
+
+`ExtractionConfig::output_format` controls the `content` text format:
+
+```rust
+pub enum OutputFormat { Plain, Markdown, Djot, Html, Json, DocTags, Custom(String) }
+```
+
+`ExtractionConfig::result_format` controls the result structure:
+
+```rust
+pub enum ResultFormat { Unified, ElementBased }
+```
+
+```rust
+use xberg::{ExtractionConfig, OutputFormat};
+
+let config = ExtractionConfig {
+    output_format: OutputFormat::Markdown,
+    ..Default::default()
+};
+```
+
+---
+
+## ExtractionResult (envelope)
+
+`extract` and `extract_batch` return this envelope. Per-document data lives in each `ExtractedDocument` in `results`.
+
+```rust
+pub struct ExtractionResult {
+    pub results: Vec<ExtractedDocument>,       // one per input, in order
+    pub errors: Vec<ExtractionErrorItem>,      // non-fatal per-input errors
+    pub summary: ExtractionSummary,            // aggregate counts
+    pub crawl_final_urls: Vec<String>,
+    pub crawl_redirect_count: usize,
+    pub crawl_unique_normalized_urls: Vec<String>,
+}
+
+pub struct ExtractionSummary {
+    pub inputs: usize,
+    pub results: usize,
+    pub errors: usize,
+    pub remote_urls: usize,
+    pub pages_crawled: usize,
+    pub documents_downloaded: usize,
+}
+
+pub struct ExtractionErrorItem {
+    pub index: usize,
+    pub code: u32,
+    pub error_type: String,
+    pub source: String,
+    pub message: String,
+}
+```
+
+### `ExtractedDocument`
+
+```rust
+pub struct ExtractedDocument {
+    pub content: String,
+    pub mime_type: Cow<'static, str>,
+    pub metadata: Metadata,
+    pub extraction_method: Option<ExtractionMethod>,
+    pub tables: Vec<Table>,
+    pub detected_languages: Option<Vec<String>>,
+    pub chunks: Option<Vec<Chunk>>,
+    pub images: Option<Vec<ExtractedImage>>,
+    pub pages: Option<Vec<PageContent>>,
+    pub elements: Option<Vec<Element>>,
+    pub extracted_keywords: Option<Vec<Keyword>>,
+    pub quality_score: Option<f64>,
+    pub processing_warnings: Vec<ProcessingWarning>,
+    // ... additional optional fields
+}
+```
+
+Note that `detected_languages`, `chunks`, `images`, `pages`, `elements`, and `extracted_keywords` are `Option<Vec<...>>` — match or `if let Some(...)` before iterating.
+
+```rust
+use xberg::{extract, ChunkingConfig, ExtractInput, ExtractionConfig};
+
+#[tokio::main]
+async fn main() -> xberg::Result<()> {
+    let config = ExtractionConfig {
+        chunking: Some(ChunkingConfig { max_characters: 512, overlap: 50, ..Default::default() }),
+        ..Default::default()
+    };
+
+    let output = extract(ExtractInput::from_uri("document.pdf"), &config).await?;
+    let document = &output.results[0];
+
+    println!("Content: {}", document.content);
+    println!("MIME: {}", document.mime_type);
+
+    if let Some(langs) = &document.detected_languages {
+        println!("Languages: {langs:?}");
+    }
+    if let Some(chunks) = &document.chunks {
+        println!("Chunks: {}", chunks.len());
+        for chunk in chunks {
+            println!("  - {}", &chunk.content[..50.min(chunk.content.len())]);
+        }
+    }
+    Ok(())
+}
+```
+
+### `Chunk`
+
+```rust
+pub struct Chunk {
+    pub content: String,
+    pub chunk_type: ChunkType,
+    pub embedding: Option<Vec<f32>>,
+    pub metadata: ChunkMetadata,
+}
+```
+
+---
+
+## Error Handling
+
+### `XbergError`
+
+```rust
+pub enum XbergError {
+    Io(std::io::Error),
+    Parsing { message: String, source: Option<Box<dyn std::error::Error + Send + Sync>> },
+    Ocr { message: String, source: Option<Box<dyn std::error::Error + Send + Sync>> },
+    Validation { message: String, source: Option<Box<dyn std::error::Error + Send + Sync>> },
+    Cache { message: String, source: Option<Box<dyn std::error::Error + Send + Sync>> },
+    ImageProcessing { message: String, source: Option<Box<dyn std::error::Error + Send + Sync>> },
+    Serialization { message: String, source: Option<Box<dyn std::error::Error + Send + Sync>> },
+    MissingDependency(String),
+    Plugin { message: String, plugin_name: String },
+    LockPoisoned(String),
+    UnsupportedFormat(String),
+    Other(String),
+}
+```
+
+```rust
+use xberg::{extract, ExtractInput, ExtractionConfig, XbergError, Result};
+
+async fn extract_text(bytes: &[u8], mime_type: &str) -> Result<String> {
+    let config = ExtractionConfig::default();
+    let output = extract(
+        ExtractInput::from_bytes(bytes.to_vec(), mime_type, Some("document.pdf".to_string())),
+        &config,
+    )
+    .await?;
+
+    Ok(output.results.first().map(|d| d.content.clone()).unwrap_or_default())
+}
+
+#[tokio::main]
+async fn main() {
+    let bytes = std::fs::read("document.pdf").unwrap_or_default();
+    match extract_text(&bytes, "application/pdf").await {
+        Ok(text) => println!("Extracted {} chars", text.len()),
+        Err(XbergError::UnsupportedFormat(mime)) => eprintln!("Format not supported: {mime}"),
+        Err(XbergError::Ocr { message, .. }) => eprintln!("OCR failed: {message}"),
+        Err(e) => eprintln!("Error: {e}"),
+    }
+}
+```
+
+### Result type alias
+
+```rust
+pub type Result<T> = std::result::Result<T, XbergError>;
+```
+
+---
+
+## Version
+
+This reference targets Xberg `1.1.0`.

package/.opencode/skills/xberg/references/supported-formats.md

@@ -0,0 +1,252 @@
+<!--
+AI-RULEZ :: GENERATED FILE — DO NOT EDIT
+Content-Hash: blake3:e3c1dcb34a4784efc4f7fd868468061c3bd49d41e7e39792086633dcfe029d63
+Source-Hash: blake3:3fca38eb564d0ed9c9194d070fd8dec96de727edc92e82954cb4b9af52cddbaf
+Schema-Version: v1
+-->
+
+# Supported Formats Reference
+
+Xberg supports 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases. The tables below summarize the current extension families; `xberg formats` and the [generated format reference](https://docs.xberg.io/reference/formats/) are authoritative for individual MIME mappings and feature-gated availability.
+
+## Office Documents
+
+### Word Processing
+
+| Format             | Extensions               | MIME Type                                                                 | Capabilities                                                    |
+| ------------------ | ------------------------ | ------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| Microsoft Word     | `.docx`, `.doc`          | OOXML and legacy Word MIME types                                          | Full text extraction, tables, embedded images, metadata, styles |
+| Word Macro-Enabled | `.docm`                  | `application/vnd.ms-word.document.macroEnabled.12`                        | Macro-enabled document extraction, metadata                     |
+| Word Template      | `.dotx`, `.dotm`, `.dot` | Various Word template MIME types                                          | Template document extraction, metadata                          |
+| OpenDocument Text  | `.odt`                   | `application/vnd.oasis.opendocument.text`                                 | Full text extraction, tables, embedded images, metadata, styles |
+| Apple Pages        | `.pages`                 | `application/x-iwork-pages-sffpages`                                      | Text, tables, images, and metadata                               |
+| WordPerfect        | `.wpd`, `.wp`, `.wp5`, `.wp6` | `application/vnd.wordperfect`                                        | WordPerfect 4.2 through X-series documents                      |
+
+### Spreadsheets
+
+| Format                   | Extensions | MIME Type                                                              | Capabilities                                             |
+| ------------------------ | ---------- | ---------------------------------------------------------------------- | -------------------------------------------------------- |
+| Excel Workbook           | `.xlsx`    | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`    | Sheet data, cell values, formulas, cell metadata, charts |
+| Excel Macro-Enabled      | `.xlsm`    | `application/vnd.ms-excel.sheet.macroEnabled.12`                       | Sheet data, formulas, macros (text only), metadata       |
+| Excel Binary             | `.xlsb`    | `application/vnd.ms-excel.sheet.binary.macroEnabled.12`                | Binary sheet data extraction, metadata                   |
+| Excel Legacy             | `.xls`     | `application/vnd.ms-excel`                                             | Legacy sheet data extraction, metadata                   |
+| Excel Add-in             | `.xla`     | `application/vnd.ms-excel.template.macroEnabled.12`                    | Add-in data extraction                                   |
+| Excel Macro Add-in       | `.xlam`    | `application/vnd.ms-excel.addin.macroEnabled.12`                       | Macro add-in metadata                                    |
+| Excel Template (XML)     | `.xltx`, `.xltm` | OOXML spreadsheet template MIME types                              | XML template data and metadata                           |
+| Excel Template (Legacy)  | `.xlt`     | `application/vnd.ms-excel`                                             | Legacy template data extraction                          |
+| OpenDocument Spreadsheet | `.ods`     | `application/vnd.oasis.opendocument.spreadsheet`                       | Sheet data, cell values, formulas, metadata              |
+| Apple Numbers            | `.numbers` | `application/x-iwork-numbers-sffnumbers`                               | Sheet data, formulas, and metadata                       |
+
+### Presentations
+
+| Format                  | Extensions               | MIME Type                                                                   | Capabilities                                         |
+| ----------------------- | ------------------------ | --------------------------------------------------------------------------- | ---------------------------------------------------- |
+| PowerPoint Presentation | `.pptx`, `.pptm`         | OOXML presentation MIME types                                             | Slide text, speaker notes, embedded images, metadata |
+| PowerPoint Legacy       | `.ppt`, `.pps`           | `application/vnd.ms-powerpoint`                                             | Legacy slide text extraction, metadata               |
+| PowerPoint Slideshow    | `.ppsx`                  | `application/vnd.openxmlformats-officedocument.presentationml.slideshow`    | Slideshow content, speaker notes, metadata           |
+| PowerPoint Template     | `.potx`, `.potm`, `.pot` | Various PowerPoint template MIME types                                      | Template slide extraction, metadata                  |
+| OpenDocument Presentation | `.odp`                 | `application/vnd.oasis.opendocument.presentation`                           | Slides, notes, images, and metadata                   |
+| Apple Keynote           | `.key`                   | `application/x-iwork-keynote-sffkey`                                        | Slides, notes, images, and metadata                   |
+
+### PDF
+
+| Format                   | Extensions | MIME Type         | Capabilities                                                                                       |
+| ------------------------ | ---------- | ----------------- | -------------------------------------------------------------------------------------------------- |
+| Portable Document Format | `.pdf`     | `application/pdf` | Text extraction, tables, embedded images, metadata, OCR (when needed), password protection support |
+
+### eBooks
+
+| Format      | Extensions | MIME Type                       | Capabilities                                           |
+| ----------- | ---------- | ------------------------------- | ------------------------------------------------------ |
+| EPUB        | `.epub`    | `application/epub+zip`          | Chapter text, metadata, embedded resources, navigation |
+| FictionBook | `.fb2`     | `application/x-fictionbook+xml` | Book content, metadata, chapter structure              |
+
+### Database
+
+| Format | Extensions | MIME Type           | Capabilities                                          |
+| ------ | ---------- | ------------------- | ----------------------------------------------------- |
+| dBASE  | `.dbf`     | `application/vnd.dbf` | Table data extraction as markdown, field type support |
+| SQLite | `.sqlite`, `.sqlite3`, `.db` | `application/vnd.sqlite3` | Bounded table extraction and schema metadata |
+| GeoPackage | `.gpkg`, `.gpkx` | `application/geopackage+sqlite3` | SQLite table extraction with application-ID detection |
+
+### Hangul
+
+| Format                | Extensions      | MIME Type                                       | Capabilities                            |
+| --------------------- | --------------- | ----------------------------------------------- | --------------------------------------- |
+| Hangul Word Processor | `.hwp`, `.hwpx` | `application/x-hwp`, `application/haansofthwpx` | Korean document format, text extraction |
+
+## Images (OCR-Enabled)
+
+### Raster Images
+
+| Format | Extensions      | MIME Type    | Capabilities                                                                 |
+| ------ | --------------- | ------------ | ---------------------------------------------------------------------------- |
+| PNG    | `.png`          | `image/png`  | OCR text extraction, table detection, EXIF metadata, dimensions, color space |
+| JPEG   | `.jpg`, `.jpeg` | `image/jpeg` | OCR text extraction, table detection, EXIF metadata, color profile           |
+| GIF    | `.gif`          | `image/gif`  | OCR text extraction, animation metadata, dimensions                          |
+| WebP   | `.webp`         | `image/webp` | OCR text extraction, metadata, lossy/lossless detection                      |
+| Bitmap | `.bmp`          | `image/bmp`  | OCR text extraction, dimensions, color depth                                 |
+| TIFF   | `.tiff`, `.tif` | `image/tiff` | OCR text extraction, multi-page support, EXIF metadata, compression info     |
+| HEIC family | `.heic`, `.heics`, `.heif`, `.heifs`, `.hif`, `.avif`, `.avcs` | HEIF/AVIF MIME types | Metadata extraction and optional pixel decoding |
+
+### Advanced Image Formats
+
+| Format             | Extensions                     | MIME Type                 | Capabilities                                                                     |
+| ------------------ | ------------------------------ | ------------------------- | -------------------------------------------------------------------------------- |
+| JPEG 2000 container | `.jp2`, `.jpg2`               | `image/jp2`               | OCR via pure Rust decoder (hayro-jpeg2000), table detection, resolution metadata |
+| JPEG 2000 codestream | `.j2c`, `.j2k`, `.jpc`        | `image/j2c`               | OCR for raw JPEG 2000 codestreams                                                |
+| JBIG2              | `.jbig2`, `.jb2`               | `image/x-jbig2`           | Bi-level image OCR, high compression, technical documents                        |
+| Portable PixMap    | `.pnm`, `.pbm`, `.pgm`, `.ppm` | `.pnm`=`image/x-portable-anymap`, `.pbm`=`image/x-portable-bitmap`, `.pgm`=`image/x-portable-graymap`, `.ppm`=`image/x-portable-pixmap` | OCR for plain image formats, raw pixel data                                      |
+
+### Vector Graphics
+
+| Format                   | Extensions | MIME Type       | Capabilities                                                              |
+| ------------------------ | ---------- | --------------- | ------------------------------------------------------------------------- |
+| Scalable Vector Graphics | `.svg`     | `image/svg+xml` | DOM parsing, embedded text extraction, graphics metadata, vector elements |
+
+## Web & Data
+
+### Markup & Structured Text
+
+| Format           | Extensions      | MIME Type               | Capabilities                                                                       |
+| ---------------- | --------------- | ----------------------- | ---------------------------------------------------------------------------------- |
+| HyperText Markup | `.html`, `.htm` | `text/html` | DOM parsing, text extraction, metadata (Open Graph, Twitter Card), link extraction |
+| XHTML | `.xhtml`, `.xht` | `application/xhtml+xml` | XML-serialized HTML extraction and metadata |
+| XML              | `.xml`          | `application/xml`       | DOM parsing, namespace handling, text extraction, structure analysis               |
+| KML              | `.kml`          | `application/vnd.google-earth.kml+xml` | Geographic features through structural XML extraction                |
+
+### Structured Data Formats
+
+| Format | Extensions      | MIME Type                   | Capabilities                                               |
+| ------ | --------------- | --------------------------- | ---------------------------------------------------------- |
+| JSON   | `.json`         | `application/json`          | Schema detection, nested structure parsing, validation     |
+| GeoJSON | `.geojson`     | `application/geo+json`      | Geographic objects and coordinates through structured JSON extraction |
+| JSON Lines | `.jsonl`, `.ndjson` | `application/x-ndjson` | Newline-delimited JSON record extraction          |
+| YAML   | `.yaml`, `.yml` | `application/yaml`          | Hierarchical data parsing, custom tags, nested structures  |
+| TOML   | `.toml`         | `application/toml`          | Configuration parsing, table structures, type preservation |
+| CSV    | `.csv`          | `text/csv`                  | Delimiter detection, header inference, type detection      |
+| TSV    | `.tsv`          | `text/tab-separated-values` | Tab-separated value parsing, header detection              |
+
+### Text & Markup Languages
+
+| Format           | Extensions         | MIME Type         | Capabilities                                      |
+| ---------------- | ------------------ | ----------------- | ------------------------------------------------- |
+| Plain Text       | `.txt`             | `text/plain`      | Raw text extraction, encoding detection           |
+| AsciiDoc         | `.adoc`, `.asciidoc` | `text/asciidoc` | AsciiDoc structure and text extraction             |
+| WebVTT           | `.vtt`             | `text/vtt`        | Timed cue and transcript extraction                |
+| Markdown         | `.md`, `.markdown` | `text/markdown`   | CommonMark parsing, GFM extensions, front matter  |
+| CommonMark       | `.commonmark`      | `text/x-commonmark` | Standard CommonMark parsing                      |
+| Quarto           | `.qmd`             | `text/x-quarto`   | Quarto Markdown extraction                          |
+| R Markdown       | `.rmd`             | `text/x-r-markdown` | R Markdown extraction                            |
+| Djot             | `.djot`, `.dj`     | `text/x-djot`     | Djot format parsing, semantic structure           |
+| Docling DocTags  | `.doctags`         | `text/vnd.docling.doctags` | DocTags representation extraction          |
+| reStructuredText | `.rst`             | `text/prs.fallenstein.rst` | RST parsing, directive handling, role extraction  |
+| Org Mode         | `.org`             | `text/org`        | Org mode structure, outline parsing, metadata     |
+| Rich Text Format | `.rtf`             | `application/rtf` | Text with formatting extraction, font information |
+| MDX              | `.mdx`             | `text/mdx`        | Markdown and embedded JSX content                  |
+
+## Audio & Video
+
+| Category | Extensions | Capabilities |
+| -------- | ---------- | ------------ |
+| Audio | `.mp3`, `.mpga`, `.m4a`, `.wav`, `.webm` | Whisper transcription |
+| MP4 audio track | `.mp4`, `.mpg4`, `.mp4v`, `.m4v` | Audio-track transcription |
+| MPEG audio track | `.mpeg`, `.mpg`, `.mpe`, `.m1v`, `.m2v` | Audio-track transcription |
+| WebM audio track | `.webm` | Audio-track transcription |
+
+## Email & Archives
+
+### Email Formats
+
+| Format            | Extensions | MIME Type                    | Capabilities                                                                           |
+| ----------------- | ---------- | ---------------------------- | -------------------------------------------------------------------------------------- |
+| Email Message     | `.eml`     | `message/rfc822`             | Headers (from, to, subject, date), body (HTML/plain text), attachments, threading info |
+| Microsoft Outlook | `.msg`     | `application/vnd.ms-outlook` | Outlook headers, body content, attachments, recipient metadata                         |
+| Outlook Data File | `.pst`     | `application/vnd.ms-outlook-pst` | Folder hierarchy, messages, attachments, and metadata                              |
+
+### Archive Formats
+
+| Format      | Extensions | MIME Type                     | Capabilities                                               |
+| ----------- | ---------- | ----------------------------- | ---------------------------------------------------------- |
+| ZIP Archive | `.zip`     | `application/zip`             | File listing, nested archive support, compression metadata |
+| Tar Archive | `.tar`     | `application/x-tar`           | File listing, permission metadata, nested archives         |
+| Gzip Tar    | `.tgz`     | `application/gzip`            | Compressed archive listing, metadata                       |
+| Gzip        | `.gz`      | `application/gzip`            | Compressed file metadata                                   |
+| 7-Zip       | `.7z`      | `application/x-7z-compressed` | File listing, compression info, nested archives            |
+
+## Academic & Scientific
+
+### Citation Formats
+
+| Format                  | Extensions  | MIME Type                                | Capabilities                                      |
+| ----------------------- | ----------- | ---------------------------------------- | ------------------------------------------------- |
+| BibTeX                  | `.bib`      | `application/x-bibtex`                    | Structured parsing, entry types, field extraction |
+| BibLaTeX                | `.bib`      | `application/x-biblatex`                  | Extended BibTeX format, advanced field support    |
+| RIS                     | `.ris`      | `application/x-research-info-systems`    | Structured RIS format parsing, type detection     |
+| NIH RIS                 | `.nbib`     | `application/x-pubmed`                    | NIH/PubMed format, structured citation data       |
+| EndNote                 | `.enw`      | `application/x-endnote+xml`              | EndNote XML format, citation metadata             |
+| Citation Style Language | —           | `application/csl+json`                   | CSL JSON parsing, style definitions               |
+
+### Scientific & Technical Formats
+
+| Format           | Extensions       | MIME Type                  | Capabilities                                                |
+| ---------------- | ---------------- | -------------------------- | ----------------------------------------------------------- |
+| LaTeX            | `.tex`, `.latex` | `application/x-latex`      | LaTeX source parsing, commands, document structure          |
+| Typst            | `.typ`, `.typst` | `text/vnd.typst`           | Typst markup parsing, document structure                    |
+| JATS XML         | `.jats`, `.nxml` | `application/x-jats+xml`   | PubMed JATS parsing, article structure, metadata            |
+| Jupyter Notebook | `.ipynb`         | `application/x-ipynb+json` | Cell extraction (code + markdown), output parsing, metadata |
+| DocBook          | `.docbook`, `.dbk`, `.docbook4`, `.docbook5` | `application/docbook+xml` | DocBook XML parsing, semantic structure |
+
+### Documentation Formats
+
+| Format      | Extensions | MIME Type                | Capabilities                                    |
+| ----------- | ---------- | ------------------------ | ----------------------------------------------- |
+| OPML        | `.opml`    | `application/xml+opml`   | Outline parsing, hierarchy extraction, metadata |
+
+## Format Capabilities Summary
+
+### Text Extraction
+
+Supported formats expose text, metadata, or both according to the active feature set. OCR and transcription extend text extraction to images, scans, audio, and video.
+
+### Metadata Support
+
+Comprehensive metadata extraction includes:
+
+- Document properties (title, author, subject, creation date, modification date)
+- Format-specific metadata (page count, dimensions, encoding, language)
+- EXIF data (for images)
+- Document statistics (word count, character count)
+
+### OCR (Optical Character Recognition)
+
+OCR is available for image formats:
+
+- **Raster Images**: PNG, JPEG, GIF, WebP, BMP, TIFF
+- **Advanced Formats**: JPEG 2000, JBIG2, PNM/PBM/PGM/PPM
+- **Configurable Backends**: Tesseract (default), PaddleOCR (`paddleocr`/`paddle-ocr`), VLM (Candle-based: TrOCR, PaddleOCR-VL, GLM-OCR, DeepSeek-OCR)
+
+### Table Detection
+
+Smart table detection and reconstruction available for:
+
+- PDF documents (native tables and scanned content with OCR)
+- Office documents (Excel, Word)
+- Images (via OCR backends)
+- HTML/XML (from markup structure)
+
+### Archive & Nested Document Support
+
+Archives and nested formats support file listing and sequential extraction:
+
+- ZIP, TAR, TGZ, 7Z archives
+- Email attachments
+- Nested archives within archives
+
+## Getting Started
+
+For language-specific examples and detailed API documentation, see the [API Reference](https://docs.xberg.io/reference/api-python/).
+
+For OCR configuration and backend selection, see the [OCR Backends Guide](https://docs.xberg.io/guides/ocr/).
+
+For comprehensive format details and format detection, see the [Complete Format Reference](https://docs.xberg.io/reference/formats/).

package/.pi/prompts/xberg-extract.md

@@ -0,0 +1,14 @@
+---
+description: Extract a document locally with Xberg
+---
+
+<!--
+AI-RULEZ :: GENERATED FILE — DO NOT EDIT
+Content-Hash: blake3:3d695d0525269a1935328e6568418a562cac9ba7a64c6e8121d5136dc93b36e3
+Source-Hash: blake3:3fca38eb564d0ed9c9194d070fd8dec96de727edc92e82954cb4b9af52cddbaf
+Schema-Version: v1
+-->
+
+Extract this document: $ARGUMENTS
+
+Read the xberg and picking-a-format skills. Use `xberg extract <file> --format json` and choose --content-format for the consumer. Preserve input paths as separate arguments. Report extraction errors and content omissions.