@xberg-io/opencode-xberg 1.3.8 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.opencode/ai-rulez-bundle.json +17 -16
- package/.opencode/ai-rulez-content.js +3 -3
- package/.opencode/commands/xberg-extract.md +3 -3
- package/.opencode/commands/xberg-ocr.md +3 -3
- package/.opencode/plugins/xberg.js +1 -1
- package/.opencode/skills/xberg/SKILL.md +20 -25
- package/.opencode/skills/xberg/index.md +10 -0
- package/.opencode/skills/xberg/references/advanced-features.md +1 -1
- package/.opencode/skills/xberg/references/cli-reference.md +6 -4
- package/.opencode/skills/xberg/references/configuration.md +6 -6
- package/.opencode/skills/xberg/references/nodejs-api.md +1 -1
- package/.opencode/skills/xberg/references/other-bindings.md +1 -1
- package/.opencode/skills/xberg/references/python-api.md +6 -6
- package/.opencode/skills/xberg/references/rust-api.md +5 -5
- package/.opencode/skills/xberg/references/supported-formats.md +1 -1
- package/.opencode/skills/{batch-extraction → xberg-batch-extraction}/SKILL.md +11 -9
- package/.opencode/skills/xberg-batch-extraction/index.md +10 -0
- package/.opencode/skills/xberg-batch-extraction/references/advanced-features.md +628 -0
- package/.opencode/skills/xberg-batch-extraction/references/cli-reference.md +464 -0
- package/.opencode/skills/xberg-batch-extraction/references/configuration.md +587 -0
- package/.opencode/skills/xberg-batch-extraction/references/nodejs-api.md +485 -0
- package/.opencode/skills/xberg-batch-extraction/references/other-bindings.md +345 -0
- package/.opencode/skills/xberg-batch-extraction/references/python-api.md +657 -0
- package/.opencode/skills/xberg-batch-extraction/references/rust-api.md +412 -0
- package/.opencode/skills/xberg-batch-extraction/references/supported-formats.md +252 -0
- package/{.pi/skills/chunking → .opencode/skills/xberg-chunking}/SKILL.md +5 -5
- package/.opencode/skills/xberg-chunking/index.md +10 -0
- package/.opencode/skills/xberg-chunking/references/advanced-features.md +628 -0
- package/.opencode/skills/xberg-chunking/references/cli-reference.md +464 -0
- package/.opencode/skills/xberg-chunking/references/configuration.md +587 -0
- package/.opencode/skills/xberg-chunking/references/nodejs-api.md +485 -0
- package/.opencode/skills/xberg-chunking/references/other-bindings.md +345 -0
- package/.opencode/skills/xberg-chunking/references/python-api.md +657 -0
- package/.opencode/skills/xberg-chunking/references/rust-api.md +412 -0
- package/.opencode/skills/xberg-chunking/references/supported-formats.md +252 -0
- package/.opencode/skills/{extracting-keywords → xberg-extracting-keywords}/SKILL.md +7 -8
- package/.opencode/skills/xberg-extracting-keywords/index.md +10 -0
- package/.opencode/skills/xberg-extracting-keywords/references/advanced-features.md +628 -0
- package/.opencode/skills/xberg-extracting-keywords/references/cli-reference.md +464 -0
- package/.opencode/skills/xberg-extracting-keywords/references/configuration.md +587 -0
- package/.opencode/skills/xberg-extracting-keywords/references/nodejs-api.md +485 -0
- package/.opencode/skills/xberg-extracting-keywords/references/other-bindings.md +345 -0
- package/.opencode/skills/xberg-extracting-keywords/references/python-api.md +657 -0
- package/.opencode/skills/xberg-extracting-keywords/references/rust-api.md +412 -0
- package/.opencode/skills/xberg-extracting-keywords/references/supported-formats.md +252 -0
- package/{.pi/skills/extracting-tables → .opencode/skills/xberg-extracting-tables}/SKILL.md +4 -5
- package/.opencode/skills/xberg-extracting-tables/index.md +10 -0
- package/.opencode/skills/xberg-extracting-tables/references/advanced-features.md +628 -0
- package/.opencode/skills/xberg-extracting-tables/references/cli-reference.md +464 -0
- package/.opencode/skills/xberg-extracting-tables/references/configuration.md +587 -0
- package/.opencode/skills/xberg-extracting-tables/references/nodejs-api.md +485 -0
- package/.opencode/skills/xberg-extracting-tables/references/other-bindings.md +345 -0
- package/.opencode/skills/xberg-extracting-tables/references/python-api.md +657 -0
- package/.opencode/skills/xberg-extracting-tables/references/rust-api.md +412 -0
- package/.opencode/skills/xberg-extracting-tables/references/supported-formats.md +252 -0
- package/{.pi/skills/extracting-with-ocr → .opencode/skills/xberg-extracting-with-ocr}/SKILL.md +5 -6
- package/.opencode/skills/xberg-extracting-with-ocr/index.md +10 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/advanced-features.md +628 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/cli-reference.md +464 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/configuration.md +587 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/nodejs-api.md +485 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/other-bindings.md +345 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/python-api.md +657 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/rust-api.md +412 -0
- package/.opencode/skills/xberg-extracting-with-ocr/references/supported-formats.md +252 -0
- package/.opencode/skills/{picking-a-format → xberg-picking-a-format}/SKILL.md +4 -4
- package/.opencode/skills/xberg-picking-a-format/index.md +10 -0
- package/.opencode/skills/xberg-picking-a-format/references/advanced-features.md +628 -0
- package/.opencode/skills/xberg-picking-a-format/references/cli-reference.md +464 -0
- package/.opencode/skills/xberg-picking-a-format/references/configuration.md +587 -0
- package/.opencode/skills/xberg-picking-a-format/references/nodejs-api.md +485 -0
- package/.opencode/skills/xberg-picking-a-format/references/other-bindings.md +345 -0
- package/.opencode/skills/xberg-picking-a-format/references/python-api.md +657 -0
- package/.opencode/skills/xberg-picking-a-format/references/rust-api.md +412 -0
- package/.opencode/skills/xberg-picking-a-format/references/supported-formats.md +252 -0
- package/.pi/prompts/xberg-extract.md +1 -1
- package/.pi/prompts/xberg-ocr.md +1 -1
- package/.pi/skills/xberg/SKILL.md +20 -25
- package/.pi/skills/xberg/index.md +10 -0
- package/.pi/skills/xberg/references/advanced-features.md +1 -1
- package/.pi/skills/xberg/references/cli-reference.md +6 -4
- package/.pi/skills/xberg/references/configuration.md +6 -6
- package/.pi/skills/xberg/references/nodejs-api.md +1 -1
- package/.pi/skills/xberg/references/other-bindings.md +1 -1
- package/.pi/skills/xberg/references/python-api.md +6 -6
- package/.pi/skills/xberg/references/rust-api.md +5 -5
- package/.pi/skills/xberg/references/supported-formats.md +1 -1
- package/.pi/skills/{batch-extraction → xberg-batch-extraction}/SKILL.md +11 -9
- package/.pi/skills/xberg-batch-extraction/index.md +10 -0
- package/.pi/skills/xberg-batch-extraction/references/advanced-features.md +628 -0
- package/.pi/skills/xberg-batch-extraction/references/cli-reference.md +464 -0
- package/.pi/skills/xberg-batch-extraction/references/configuration.md +587 -0
- package/.pi/skills/xberg-batch-extraction/references/nodejs-api.md +485 -0
- package/.pi/skills/xberg-batch-extraction/references/other-bindings.md +345 -0
- package/.pi/skills/xberg-batch-extraction/references/python-api.md +657 -0
- package/.pi/skills/xberg-batch-extraction/references/rust-api.md +412 -0
- package/.pi/skills/xberg-batch-extraction/references/supported-formats.md +252 -0
- package/{.opencode/skills/chunking → .pi/skills/xberg-chunking}/SKILL.md +5 -5
- package/.pi/skills/xberg-chunking/index.md +10 -0
- package/.pi/skills/xberg-chunking/references/advanced-features.md +628 -0
- package/.pi/skills/xberg-chunking/references/cli-reference.md +464 -0
- package/.pi/skills/xberg-chunking/references/configuration.md +587 -0
- package/.pi/skills/xberg-chunking/references/nodejs-api.md +485 -0
- package/.pi/skills/xberg-chunking/references/other-bindings.md +345 -0
- package/.pi/skills/xberg-chunking/references/python-api.md +657 -0
- package/.pi/skills/xberg-chunking/references/rust-api.md +412 -0
- package/.pi/skills/xberg-chunking/references/supported-formats.md +252 -0
- package/.pi/skills/{extracting-keywords → xberg-extracting-keywords}/SKILL.md +7 -8
- package/.pi/skills/xberg-extracting-keywords/index.md +10 -0
- package/.pi/skills/xberg-extracting-keywords/references/advanced-features.md +628 -0
- package/.pi/skills/xberg-extracting-keywords/references/cli-reference.md +464 -0
- package/.pi/skills/xberg-extracting-keywords/references/configuration.md +587 -0
- package/.pi/skills/xberg-extracting-keywords/references/nodejs-api.md +485 -0
- package/.pi/skills/xberg-extracting-keywords/references/other-bindings.md +345 -0
- package/.pi/skills/xberg-extracting-keywords/references/python-api.md +657 -0
- package/.pi/skills/xberg-extracting-keywords/references/rust-api.md +412 -0
- package/.pi/skills/xberg-extracting-keywords/references/supported-formats.md +252 -0
- package/{.opencode/skills/extracting-tables → .pi/skills/xberg-extracting-tables}/SKILL.md +4 -5
- package/.pi/skills/xberg-extracting-tables/index.md +10 -0
- package/.pi/skills/xberg-extracting-tables/references/advanced-features.md +628 -0
- package/.pi/skills/xberg-extracting-tables/references/cli-reference.md +464 -0
- package/.pi/skills/xberg-extracting-tables/references/configuration.md +587 -0
- package/.pi/skills/xberg-extracting-tables/references/nodejs-api.md +485 -0
- package/.pi/skills/xberg-extracting-tables/references/other-bindings.md +345 -0
- package/.pi/skills/xberg-extracting-tables/references/python-api.md +657 -0
- package/.pi/skills/xberg-extracting-tables/references/rust-api.md +412 -0
- package/.pi/skills/xberg-extracting-tables/references/supported-formats.md +252 -0
- package/{.opencode/skills/extracting-with-ocr → .pi/skills/xberg-extracting-with-ocr}/SKILL.md +5 -6
- package/.pi/skills/xberg-extracting-with-ocr/index.md +10 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/advanced-features.md +628 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/cli-reference.md +464 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/configuration.md +587 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/nodejs-api.md +485 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/other-bindings.md +345 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/python-api.md +657 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/rust-api.md +412 -0
- package/.pi/skills/xberg-extracting-with-ocr/references/supported-formats.md +252 -0
- package/.pi/skills/{picking-a-format → xberg-picking-a-format}/SKILL.md +4 -4
- package/.pi/skills/xberg-picking-a-format/index.md +10 -0
- package/.pi/skills/xberg-picking-a-format/references/advanced-features.md +628 -0
- package/.pi/skills/xberg-picking-a-format/references/cli-reference.md +464 -0
- package/.pi/skills/xberg-picking-a-format/references/configuration.md +587 -0
- package/.pi/skills/xberg-picking-a-format/references/nodejs-api.md +485 -0
- package/.pi/skills/xberg-picking-a-format/references/other-bindings.md +345 -0
- package/.pi/skills/xberg-picking-a-format/references/python-api.md +657 -0
- package/.pi/skills/xberg-picking-a-format/references/rust-api.md +412 -0
- package/.pi/skills/xberg-picking-a-format/references/supported-formats.md +252 -0
- package/README.md +75 -9
- package/assets/icon.svg +16 -9
- package/assets/logo.png +0 -0
- package/package.json +136 -21
- package/scripts/mcp-launch.sh +6 -5
package/.pi/prompts/xberg-ocr.md
CHANGED
|
@@ -5,7 +5,7 @@ description: Extract scanned documents with OCR
|
|
|
5
5
|
<!--
|
|
6
6
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
7
7
|
Content-Hash: blake3:1715ebd3d000212eb2e4dbfa814d6d5990bcc0c636bae67734a503f2c3ae8be4
|
|
8
|
-
Source-Hash: blake3:
|
|
8
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
9
9
|
Schema-Version: v1
|
|
10
10
|
-->
|
|
11
11
|
|
|
@@ -1,22 +1,17 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
description: >-
|
|
4
|
-
Extract text, tables, metadata, and images from 107 document formats
|
|
5
|
-
(PDF, Office, images, HTML, email, archives, academic) using Xberg.
|
|
6
|
-
Use when writing code that calls Xberg APIs in Python, Node.js/TypeScript,
|
|
7
|
-
Rust, or CLI. Covers installation, extraction (sync/async), configuration
|
|
8
|
-
(OCR, chunking, output format), batch processing, error handling, and plugins.
|
|
2
|
+
description: Extract text, tables, metadata, and images from 107 document formats (PDF, Office, images, HTML, email, archives, academic) using Xberg. Use when writing code that calls Xberg APIs in Python, Node.js/TypeScript, Rust, or CLI. Covers installation, extraction (sync/async), configuration (OCR, chunking, output format), batch processing, error handling, and plugins.
|
|
9
3
|
license: Elastic-2.0
|
|
10
4
|
metadata:
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
5
|
+
author: xberg-io
|
|
6
|
+
version: "0.1.0"
|
|
7
|
+
repository: https://github.com/xberg-io/xberg
|
|
8
|
+
name: xberg
|
|
14
9
|
---
|
|
15
10
|
|
|
16
11
|
<!--
|
|
17
12
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
18
|
-
Content-Hash: blake3:
|
|
19
|
-
Source-Hash: blake3:
|
|
13
|
+
Content-Hash: blake3:bdb659ec3959943d7e4b8da0cc3b8fbe51dc3bfec07e27dbbd8f84e190a63e1a
|
|
14
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
20
15
|
Schema-Version: v1
|
|
21
16
|
-->
|
|
22
17
|
|
|
@@ -68,8 +63,8 @@ tokio = { version = "1", features = ["full"] }
|
|
|
68
63
|
```bash
|
|
69
64
|
brew install xberg-io/tap/xberg
|
|
70
65
|
# or run without a persistent install (the CLI proxy package self-installs the binary):
|
|
71
|
-
npx @xberg-io/xberg-cli --help
|
|
72
|
-
uvx --from xberg-cli xberg --help
|
|
66
|
+
npx @xberg-io/xberg-cli@1.3.10 --help
|
|
67
|
+
uvx --from xberg-cli==1.3.8 xberg --help
|
|
73
68
|
# or download a prebuilt binary from the latest GitHub release:
|
|
74
69
|
# https://github.com/xberg-io/xberg/releases/latest
|
|
75
70
|
# or build from source:
|
|
@@ -220,7 +215,7 @@ xberg extract doc.pdf --config-json '{"ocr":{"backend":"tesseract","language":"d
|
|
|
220
215
|
|
|
221
216
|
## Batch Processing
|
|
222
217
|
|
|
223
|
-
`extract_batch` takes a list of `ExtractInput`s and returns one envelope whose `results` array holds
|
|
218
|
+
`extract_batch` takes a list of `ExtractInput`s and returns one envelope whose `results` array holds documents in discovery order; one input may yield multiple documents. Per-input failures are reported in `result.errors` with an input index.
|
|
224
219
|
|
|
225
220
|
### Python
|
|
226
221
|
|
|
@@ -319,16 +314,16 @@ config = ExtractionConfig(force_ocr=True) # OCR even if text is extractable
|
|
|
319
314
|
|
|
320
315
|
### Python
|
|
321
316
|
|
|
322
|
-
`extract` / `extract_batch` raise
|
|
317
|
+
`extract` / `extract_batch` raise an exported `XbergError` subclass, such as `ParsingError`, `SecurityError`, or `ValidationError`, including calls with `on_progress`. Catch `XbergError` for extraction failures. Per-input failures during `extract_batch` are reported non-fatally in `result.errors`.
|
|
323
318
|
|
|
324
319
|
```python
|
|
325
|
-
from xberg import ExtractInput, extract, ExtractionConfig
|
|
320
|
+
from xberg import ExtractInput, extract, ExtractionConfig, XbergError
|
|
326
321
|
|
|
327
322
|
try:
|
|
328
323
|
result = await extract(ExtractInput(uri="file.pdf"), ExtractionConfig())
|
|
329
324
|
for err in result.errors:
|
|
330
325
|
print(f"Per-input error: {err}")
|
|
331
|
-
except
|
|
326
|
+
except XbergError as e:
|
|
332
327
|
print(f"Extraction failed: {e}")
|
|
333
328
|
```
|
|
334
329
|
|
|
@@ -372,7 +367,7 @@ match extract(ExtractInput::from_uri("file.pdf"), &config).await {
|
|
|
372
367
|
2. **Async-only**: Python and Node have no sync variants — always `await extract(...)`. Rust `extract` is async; use `#[tokio::main]` or an async context.
|
|
373
368
|
3. **Build the input**: pass an `ExtractInput`, not a bare path. Use `ExtractInput(uri=...)` / `ExtractInput::from_uri(...)` (Python/Rust) or `{ kind: "uri", uri: "..." }` (Node); for bytes use `kind="bytes"` with `bytes`/`mime_type`.
|
|
374
369
|
4. **Python ChunkingConfig fields**: construct with `max_characters` and `overlap` (defaults 1000 / 200); these are also the readable attributes. When passing config as a dict/JSON, the `max_chars` / `max_overlap` aliases are also accepted. Node uses `maxCharacters` / `overlap`; Rust struct fields are `max_characters` / `overlap`.
|
|
375
|
-
5. **Python errors**:
|
|
370
|
+
5. **Python errors**: catch `XbergError` or a specific exported subclass for `extract` / `extract_batch` failures, including calls with `on_progress`. Node throws plain `Error` (no typed error subclasses).
|
|
376
371
|
6. **Rust extract signature**: `extract(input, &config)` — the config is a reference. Use `&ExtractionConfig::default()` for defaults.
|
|
377
372
|
7. **CLI --format vs --content-format**: `--format` controls CLI output (`text`, `json`, or `toon`). `--content-format` controls content rendering (`plain`, `markdown`, `djot`, `html`, `json`, or `doctags`).
|
|
378
373
|
8. **Config file field names**: Use snake_case in TOML/YAML/JSON config files — `[chunking]` fields are `max_characters` and `overlap`; other fields use names like `output_format`, `pdf_options`.
|
|
@@ -416,12 +411,12 @@ Detailed reference files for specific topics:
|
|
|
416
411
|
|
|
417
412
|
Task-focused sibling skills go deeper than this overview:
|
|
418
413
|
|
|
419
|
-
- **extracting-with-ocr** — OCR backends, language packs, force-OCR, tuning.
|
|
420
|
-
- **extracting-tables** — layout-aware table detection and table models.
|
|
421
|
-
- **chunking** — chunk size/overlap, markdown/yaml/semantic chunkers, the `chunk` command.
|
|
422
|
-
- **extracting-keywords** — YAKE/RAKE keywords, language detection, the `embed` command.
|
|
423
|
-
- **batch-extraction** — the `batch` command, `--file-configs`, parallelism, error recovery.
|
|
424
|
-
- **picking-a-format** — choosing `--format` / `--content-format` per consumer.
|
|
414
|
+
- **xberg-extracting-with-ocr** — OCR backends, language packs, force-OCR, tuning.
|
|
415
|
+
- **xberg-extracting-tables** — layout-aware table detection and table models.
|
|
416
|
+
- **xberg-chunking** — chunk size/overlap, markdown/yaml/semantic chunkers, the `chunk` command.
|
|
417
|
+
- **xberg-extracting-keywords** — YAKE/RAKE keywords, language detection, the `embed` command.
|
|
418
|
+
- **xberg-batch-extraction** — the `batch` command, `--file-configs`, parallelism, error recovery.
|
|
419
|
+
- **xberg-picking-a-format** — choosing `--format` / `--content-format` per consumer.
|
|
425
420
|
|
|
426
421
|
Full documentation: <https://docs.xberg.io>
|
|
427
422
|
GitHub: <https://github.com/xberg-io/xberg>
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
|
+
Content-Hash: blake3:03f23e46ea452f0a341ed6552d505e85c4d31333aea0862be9ff0774b399c82d
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
|
+
Schema-Version: v1
|
|
6
|
+
-->
|
|
7
|
+
|
|
8
|
+
# Concepts
|
|
9
|
+
|
|
10
|
+
- [Xberg](SKILL.md) - Extract text, tables, metadata, and images from 107 document formats (PDF, Office, images, HTML, email, archives, academic) using Xberg. Use when writing code that calls Xberg APIs in Python, Node.js/TypeScript, Rust, or CLI. Covers installation, extraction (sync/async), configuration (OCR, chunking, output format), batch processing, error handling, and plugins.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
3
|
Content-Hash: blake3:353a7a05cbe84a2af1d053c8173d2154f418972a2bbfc2846dbb2c17a8a46197
|
|
4
|
-
Source-Hash: blake3:
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
|
-
Content-Hash: blake3:
|
|
4
|
-
Source-Hash: blake3:
|
|
3
|
+
Content-Hash: blake3:70bcb43e649e885bb822d41efb7221b365fd6f1870d1e6a5bb4ab3ec10be6acc
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -11,11 +11,13 @@ Comprehensive command-line interface for the Xberg document intelligence library
|
|
|
11
11
|
|
|
12
12
|
## Installation
|
|
13
13
|
|
|
14
|
+
The pinned command below follows this source release (1.4.0); use it once that release is published.
|
|
15
|
+
|
|
14
16
|
```bash
|
|
15
17
|
brew install xberg-io/tap/xberg
|
|
16
18
|
# or run without a persistent install (self-installs the binary):
|
|
17
19
|
npx @xberg-io/xberg-cli --help
|
|
18
|
-
uvx --from xberg-cli xberg --help
|
|
20
|
+
uvx --from xberg-cli==1.4.0 xberg --help
|
|
19
21
|
```
|
|
20
22
|
|
|
21
23
|
Or download a pre-built binary from the [latest GitHub Release](https://github.com/xberg-io/xberg/releases/latest), or build from source:
|
|
@@ -422,7 +424,7 @@ xberg batch *.pdf --config extraction-config.toml
|
|
|
422
424
|
|
|
423
425
|
```bash
|
|
424
426
|
# Extract to JSON for downstream processing
|
|
425
|
-
xberg extract file.pdf --format json | jq '.content'
|
|
427
|
+
xberg extract file.pdf --format json | jq '.result.content'
|
|
426
428
|
|
|
427
429
|
# Batch process with error handling
|
|
428
430
|
xberg batch docs/*.pdf --format json || exit 1
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
|
-
Content-Hash: blake3:
|
|
4
|
-
Source-Hash: blake3:
|
|
3
|
+
Content-Hash: blake3:d9846749cb2b57685af6f077cfe2d16d5d7e58768437f14ee12f3c78a260106e
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -106,7 +106,7 @@ max_archive_depth = 3
|
|
|
106
106
|
| `cache_ttl_secs` | int \| null | `null` | Per-request cache TTL in seconds. `0` skips the cache entirely |
|
|
107
107
|
| `qr_codes` | boolean \| null | `null` | Enable QR-code detection in extracted images |
|
|
108
108
|
|
|
109
|
-
Nested sections (each `None`/unset by default unless noted): `ocr`, `chunking`, `content_filter`, `images`, `pdf_options`, `token_reduction`, `language_detection`, `pages`, `keywords`, `postprocessor`, `html_output`, `security_limits`, `layout`, `transcription`, `acceleration`, `email`, `tree_sitter`, `structured_extraction`, `ner`, `redaction`, `summarization`, `translation`, `page_classification`, `captioning`, and `url`. The `url` section always has a default (it is not optional). The sections documented below are the ones most commonly set in config files.
|
|
109
|
+
Nested sections (each `None`/unset by default unless noted): `ocr`, `xberg-chunking`, `content_filter`, `images`, `pdf_options`, `token_reduction`, `language_detection`, `pages`, `keywords`, `postprocessor`, `html_output`, `security_limits`, `layout`, `transcription`, `acceleration`, `email`, `tree_sitter`, `structured_extraction`, `ner`, `redaction`, `summarization`, `translation`, `page_classification`, `captioning`, and `url`. The `url` section always has a default (it is not optional). The sections documented below are the ones most commonly set in config files.
|
|
110
110
|
|
|
111
111
|
### OCR Configuration
|
|
112
112
|
|
|
@@ -115,7 +115,7 @@ Nested sections (each `None`/unset by default unless noted): `ocr`, `chunking`,
|
|
|
115
115
|
enabled = true
|
|
116
116
|
backend = "tesseract"
|
|
117
117
|
language = ["eng"]
|
|
118
|
-
auto_rotate =
|
|
118
|
+
auto_rotate = true
|
|
119
119
|
```
|
|
120
120
|
|
|
121
121
|
| Option | Type | Default | Description |
|
|
@@ -123,7 +123,7 @@ auto_rotate = false
|
|
|
123
123
|
| `enabled` | boolean | `true` | Whether OCR is enabled. Setting `false` is equivalent to `disable_ocr` |
|
|
124
124
|
| `backend` | string | `"tesseract"` | OCR backend: `tesseract`, `paddleocr`/`paddle-ocr`, or `vlm` |
|
|
125
125
|
| `language` | string or array | `["eng"]` | Language code(s). Accepts `"eng"` or `["eng", "deu"]`. Tesseract joins with `+` |
|
|
126
|
-
| `auto_rotate` | boolean
|
|
126
|
+
| `auto_rotate` | boolean \| null | `null` | `true` always rotates, `false` never rotates or retries, unset keeps the orientation retry |
|
|
127
127
|
| `vlm_fallback` | string | `"disabled"` | VLM fallback policy |
|
|
128
128
|
| `output_format`| string \| null | `null` | Optional OCR output format override |
|
|
129
129
|
|
|
@@ -449,7 +449,7 @@ result = await extract_batch(
|
|
|
449
449
|
)
|
|
450
450
|
```
|
|
451
451
|
|
|
452
|
-
**Overridable fields:** `enable_quality_processing`, `ocr`, `force_ocr`, `force_ocr_pages`, `disable_ocr`, `chunking`, `content_filter`, `images`, `pdf_options`, `token_reduction`, `language_detection`, `pages`, `keywords`, `postprocessor`, `html_output` (and `html_options`), `result_format`, `output_format`, `include_document_structure`, `layout`, `transcription`, `timeout_secs`, `tree_sitter`, `structured_extraction`, `url`, `ner`, `redaction`, `summarization`, `translation`, `page_classification`, `captioning`, `qr_codes`.
|
|
452
|
+
**Overridable fields:** `enable_quality_processing`, `ocr`, `force_ocr`, `force_ocr_pages`, `disable_ocr`, `xberg-chunking`, `content_filter`, `images`, `pdf_options`, `token_reduction`, `language_detection`, `pages`, `keywords`, `postprocessor`, `html_output` (and `html_options`), `result_format`, `output_format`, `include_document_structure`, `layout`, `transcription`, `timeout_secs`, `tree_sitter`, `structured_extraction`, `url`, `ner`, `redaction`, `summarization`, `translation`, `page_classification`, `captioning`, `qr_codes`.
|
|
453
453
|
|
|
454
454
|
**Batch-level only (not overridable per input):** `max_concurrent_extractions`, `use_cache`, `acceleration`, `security_limits`.
|
|
455
455
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
3
|
Content-Hash: blake3:5dd1dce4a7814ed37c2ea273dda3dd60d7caa18b08b0aff933ce3a8f7717d1a6
|
|
4
|
-
Source-Hash: blake3:
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
3
|
Content-Hash: blake3:dacb4d7def21681b8b237603d2c7b89beaf18b8ee4110175b6ea5593022f3229
|
|
4
|
-
Source-Hash: blake3:
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
|
-
Content-Hash: blake3:
|
|
4
|
-
Source-Hash: blake3:
|
|
3
|
+
Content-Hash: blake3:9250a30997ec851cbb228ddcfd1f538e81e843f81e488e5f3d927ba94af93e4f
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -128,7 +128,7 @@ Main extraction configuration. All fields are optional and use sensible defaults
|
|
|
128
128
|
| `enable_quality_processing` | bool | True | Quality post-processing to clean and normalize extracted text. |
|
|
129
129
|
| `ocr` | OcrConfig \| None | None | OCR configuration. None = OCR disabled. |
|
|
130
130
|
| `force_ocr` | bool | False | Force OCR even for searchable PDFs that contain extractable text. |
|
|
131
|
-
| `chunking` | ChunkingConfig \| None | None | Text chunking configuration. None = disabled. |
|
|
131
|
+
| `xberg-chunking` | ChunkingConfig \| None | None | Text chunking configuration. None = disabled. |
|
|
132
132
|
| `images` | ImageExtractionConfig \| None | None | Extract images FROM documents. None = no image extraction. |
|
|
133
133
|
| `pdf_options` | PdfConfig \| None | None | PDF-specific options. |
|
|
134
134
|
| `token_reduction` | TokenReductionOptions \| None | None | Reduce token count in extracted content. None = no reduction. |
|
|
@@ -485,7 +485,7 @@ for table in result.results[0].tables:
|
|
|
485
485
|
|
|
486
486
|
## Error Classes
|
|
487
487
|
|
|
488
|
-
|
|
488
|
+
`extract()` and `extract_batch()` raise the exported exception subclass for the underlying Rust error kind, including calls with `on_progress`. Catch `XbergError` for extraction failures, or a specific subclass such as `ParsingError`, `SecurityError`, or `ValidationError`.
|
|
489
489
|
|
|
490
490
|
Xberg also defines a family of typed exception classes re-exported from `xberg`. The `XbergError`-rooted ones inherit from `XbergError` (base): `ParsingError`, `OcrError`, `ValidationError`, `MissingDependencyError`, `UnsupportedFormatError`, `CacheError`, `ImageProcessingError`, `SerializationError`, `PluginError`, `SecurityError`, `TranscriptionError`, `EmbeddingError`, `RerankingError`, `XbergTimeoutError`, `IoError`, and others. A few classes sit in separate hierarchies rooted at `Exception` rather than `XbergError` — e.g. `ConfigError`/`PdfAnalysisError` and `ParseError`/`SchemaValidationError`.
|
|
491
491
|
|
|
@@ -493,12 +493,12 @@ Xberg also defines a family of typed exception classes re-exported from `xberg`.
|
|
|
493
493
|
|
|
494
494
|
```python
|
|
495
495
|
import asyncio
|
|
496
|
-
from xberg import ExtractInput, extract, ExtractionConfig
|
|
496
|
+
from xberg import ExtractInput, extract, ExtractionConfig, XbergError
|
|
497
497
|
|
|
498
498
|
async def main() -> None:
|
|
499
499
|
try:
|
|
500
500
|
result = await extract(ExtractInput(uri="document.pdf"), ExtractionConfig())
|
|
501
|
-
except
|
|
501
|
+
except XbergError as e:
|
|
502
502
|
print(f"Extraction failed: {e}")
|
|
503
503
|
|
|
504
504
|
asyncio.run(main())
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
|
-
Content-Hash: blake3:
|
|
4
|
-
Source-Hash: blake3:
|
|
3
|
+
Content-Hash: blake3:a257c85add0ba02dd9ab63eecac13348152038919a09ba99964261256acf4ee4
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -36,7 +36,7 @@ The `tokio-runtime` feature is on by default. Capabilities are gated behind feat
|
|
|
36
36
|
| `full` | `formats` + ocr + layout + embeddings + tree-sitter + LLM. |
|
|
37
37
|
| `pdf` | PDF extraction. |
|
|
38
38
|
| `ocr` | Tesseract-based OCR. |
|
|
39
|
-
| `chunking` | Text chunking for RAG pipelines. |
|
|
39
|
+
| `xberg-chunking` | Text chunking for RAG pipelines. |
|
|
40
40
|
| `embeddings` | ONNX embedding generation. |
|
|
41
41
|
| `language-detection` | Detect document language(s). |
|
|
42
42
|
| `keywords` | YAKE + RAKE keyword extraction. |
|
|
@@ -79,7 +79,7 @@ async fn main() -> xberg::Result<()> {
|
|
|
79
79
|
pub async fn extract_batch(inputs: Vec<ExtractInput>, config: &ExtractionConfig) -> Result<ExtractionResult>
|
|
80
80
|
```
|
|
81
81
|
|
|
82
|
-
Extract multiple inputs in parallel. The returned envelope's `results` holds
|
|
82
|
+
Extract multiple inputs in parallel. The returned envelope's `results` holds `ExtractedDocument` values in discovery order; one input may yield multiple documents. Non-fatal failures in `errors` carry the original input index.
|
|
83
83
|
|
|
84
84
|
```rust
|
|
85
85
|
use xberg::{extract_batch, ExtractInput, ExtractionConfig};
|
|
@@ -261,7 +261,7 @@ let config = ExtractionConfig {
|
|
|
261
261
|
|
|
262
262
|
```rust
|
|
263
263
|
pub struct ExtractionResult {
|
|
264
|
-
pub results: Vec<ExtractedDocument>, //
|
|
264
|
+
pub results: Vec<ExtractedDocument>, // documents in discovery order; multiple per input are possible
|
|
265
265
|
pub errors: Vec<ExtractionErrorItem>, // non-fatal per-input errors
|
|
266
266
|
pub summary: ExtractionSummary, // aggregate counts
|
|
267
267
|
pub crawl_final_urls: Vec<String>,
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<!--
|
|
2
2
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
3
|
Content-Hash: blake3:e3c1dcb34a4784efc4f7fd868468061c3bd49d41e7e39792086633dcfe029d63
|
|
4
|
-
Source-Hash: blake3:
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
5
|
Schema-Version: v1
|
|
6
6
|
-->
|
|
7
7
|
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: batch-extraction
|
|
3
2
|
description: Use when extracting from many files at once with shared config, bounded parallelism, per-file overrides, and error recovery. Covers the `batch` command, `--file-configs`, `--max-concurrent`, and output layout.
|
|
3
|
+
name: xberg-batch-extraction
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
<!--
|
|
7
7
|
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
8
|
-
Content-Hash: blake3:
|
|
9
|
-
Source-Hash: blake3:
|
|
8
|
+
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
|
|
9
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
10
10
|
Schema-Version: v1
|
|
11
11
|
-->
|
|
12
12
|
|
|
@@ -14,13 +14,13 @@ Schema-Version: v1
|
|
|
14
14
|
|
|
15
15
|
Use this when processing a directory or glob of documents in one pass.
|
|
16
16
|
`xberg batch` shares one extraction config across every file, runs
|
|
17
|
-
extractions concurrently, and returns
|
|
17
|
+
extractions concurrently, and returns a JSON envelope — failures on
|
|
18
18
|
individual files do not abort the run.
|
|
19
19
|
|
|
20
20
|
## Basic usage
|
|
21
21
|
|
|
22
22
|
```bash
|
|
23
|
-
# Glob expands to many paths; results come back
|
|
23
|
+
# Glob expands to many paths; results come back in a JSON envelope (default)
|
|
24
24
|
xberg batch *.pdf
|
|
25
25
|
|
|
26
26
|
# Mixed formats, markdown content for LLM ingestion
|
|
@@ -31,12 +31,14 @@ xberg batch $(find ./corpus -name '*.pdf')
|
|
|
31
31
|
```
|
|
32
32
|
|
|
33
33
|
`batch` defaults to `--format json` (vs `--format text` for single
|
|
34
|
-
`extract`).
|
|
35
|
-
|
|
34
|
+
`extract`). The envelope contains `results`, `total_ms`, `per_file_ms` and, when failures
|
|
35
|
+
occur, `errors`. Each `results` entry is a full extracted document. One input may produce
|
|
36
|
+
multiple documents, so do not pair results with inputs by position. `per_file_ms` is
|
|
37
|
+
aligned with inputs; inspect `errors` for per-input failures.
|
|
36
38
|
|
|
37
39
|
```bash
|
|
38
40
|
xberg batch reports/*.pdf \
|
|
39
|
-
| jq '.[] | {chars: (.content | length), mime: .mime_type}'
|
|
41
|
+
| jq '.results[] | {chars: (.content | length), mime: .mime_type}'
|
|
40
42
|
```
|
|
41
43
|
|
|
42
44
|
## Parallelism
|
|
@@ -146,7 +148,7 @@ for doc in output.results:
|
|
|
146
148
|
Per-input overrides go on `ExtractInput.config` (a `FileExtractionConfig`).
|
|
147
149
|
Node.js mirrors this with `extractBatch`; Rust uses `extract_batch(inputs, &config)`.
|
|
148
150
|
See `references/python-api.md`, `references/nodejs-api.md`, and
|
|
149
|
-
`references/rust-api.md
|
|
151
|
+
`references/rust-api.md`.
|
|
150
152
|
|
|
151
153
|
## MCP
|
|
152
154
|
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
3
|
+
Content-Hash: blake3:3aac352725eacf5cb70a1f994f0a1d4b51eb2f4f30872b388f66d7d47e18c5f2
|
|
4
|
+
Source-Hash: blake3:ab2bcd453f9c2ded67752f6a9800186d42b6921f4be16f6ab3a773254a9fba00
|
|
5
|
+
Schema-Version: v1
|
|
6
|
+
-->
|
|
7
|
+
|
|
8
|
+
# Concepts
|
|
9
|
+
|
|
10
|
+
- [Batch Extraction](SKILL.md) - Use when extracting from many files at once with shared config, bounded parallelism, per-file overrides, and error recovery. Covers the `batch` command, `--file-configs`, `--max-concurrent`, and output layout.
|