@xberg-io/opencode-xberg 1.3.7 → 1.3.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.opencode/ai-rulez-bundle.json +51 -0
- package/.opencode/ai-rulez-content.js +167 -0
- package/.opencode/commands/xberg-extract.md +14 -0
- package/.opencode/commands/xberg-ocr.md +14 -0
- package/.opencode/plugins/xberg.js +75 -62
- package/.opencode/skills/batch-extraction/SKILL.md +168 -0
- package/.opencode/skills/chunking/SKILL.md +163 -0
- package/.opencode/skills/extracting-keywords/SKILL.md +171 -0
- package/.opencode/skills/extracting-tables/SKILL.md +156 -0
- package/.opencode/skills/extracting-with-ocr/SKILL.md +130 -0
- package/.opencode/skills/picking-a-format/SKILL.md +104 -0
- package/.opencode/skills/xberg/SKILL.md +427 -0
- package/.opencode/skills/xberg/references/advanced-features.md +628 -0
- package/.opencode/skills/xberg/references/cli-reference.md +462 -0
- package/.opencode/skills/xberg/references/configuration.md +587 -0
- package/.opencode/skills/xberg/references/nodejs-api.md +485 -0
- package/.opencode/skills/xberg/references/other-bindings.md +345 -0
- package/.opencode/skills/xberg/references/python-api.md +657 -0
- package/.opencode/skills/xberg/references/rust-api.md +412 -0
- package/.opencode/skills/xberg/references/supported-formats.md +252 -0
- package/.pi/prompts/xberg-extract.md +14 -0
- package/.pi/prompts/xberg-ocr.md +14 -0
- package/.pi/skills/batch-extraction/SKILL.md +168 -0
- package/.pi/skills/chunking/SKILL.md +163 -0
- package/.pi/skills/extracting-keywords/SKILL.md +171 -0
- package/.pi/skills/extracting-tables/SKILL.md +156 -0
- package/.pi/skills/extracting-with-ocr/SKILL.md +130 -0
- package/.pi/skills/picking-a-format/SKILL.md +104 -0
- package/.pi/skills/xberg/SKILL.md +427 -0
- package/.pi/skills/xberg/references/advanced-features.md +628 -0
- package/.pi/skills/xberg/references/cli-reference.md +462 -0
- package/.pi/skills/xberg/references/configuration.md +587 -0
- package/.pi/skills/xberg/references/nodejs-api.md +485 -0
- package/.pi/skills/xberg/references/other-bindings.md +345 -0
- package/.pi/skills/xberg/references/python-api.md +657 -0
- package/.pi/skills/xberg/references/rust-api.md +412 -0
- package/.pi/skills/xberg/references/supported-formats.md +252 -0
- package/README.md +40 -0
- package/package.json +69 -16
- package/scripts/mcp-launch.sh +188 -0
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: chunking
|
|
3
|
+
description: Use when splitting extracted text into chunks for LLM context windows or RAG ingestion. Covers chunk size, overlap, markdown/yaml/semantic chunkers, tokenizer-based sizing, and the standalone `chunk` command.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
<!--
|
|
7
|
+
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
8
|
+
Content-Hash: blake3:0b5cd4bec9d2a8f3452e07139ef56d6b87f9f7990714d15483a03435e751af59
|
|
9
|
+
Source-Hash: blake3:3fca38eb564d0ed9c9194d070fd8dec96de727edc92e82954cb4b9af52cddbaf
|
|
10
|
+
Schema-Version: v1
|
|
11
|
+
-->
|
|
12
|
+
|
|
13
|
+
# Chunking
|
|
14
|
+
|
|
15
|
+
Use this when feeding documents into an LLM context window or a vector
|
|
16
|
+
store. Xberg chunks two ways: inline during extraction (chunks land on
|
|
17
|
+
each document's `chunks` field), or standalone via the `chunk` command for text you
|
|
18
|
+
already have. Sizing is character-based by default, or token-based when a
|
|
19
|
+
tokenizer model is supplied.
|
|
20
|
+
|
|
21
|
+
## Inline during extraction
|
|
22
|
+
|
|
23
|
+
Turn on chunking with `--chunk` and the chunks appear on the structured
|
|
24
|
+
result under `chunks`:
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
# 1000-char chunks, 200-char overlap (defaults when --chunk is on)
|
|
28
|
+
xberg extract report.pdf --chunk --format json | jq '.chunks | length'
|
|
29
|
+
|
|
30
|
+
# Explicit size + overlap
|
|
31
|
+
xberg extract report.pdf --chunk --chunk-size 1500 --chunk-overlap 300 --format json
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Overlap must be smaller than chunk size — the CLI rejects
|
|
35
|
+
`--chunk-overlap >= --chunk-size`. When you set only `--chunk-overlap`
|
|
36
|
+
against an existing config, an overlap that exceeds the size is clamped to
|
|
37
|
+
`chunk_size / 4`.
|
|
38
|
+
|
|
39
|
+
## Standalone `chunk` command
|
|
40
|
+
|
|
41
|
+
Chunk text you already have, from `--text` or stdin. Output defaults to
|
|
42
|
+
JSON:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
# From a flag
|
|
46
|
+
xberg chunk --text "long document text ..." --chunk-size 800 --chunk-overlap 100
|
|
47
|
+
|
|
48
|
+
# From stdin (pipe extracted content straight in)
|
|
49
|
+
xberg extract notes.md | xberg chunk --chunk-size 500 --format json
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
JSON output carries `chunks` (array of strings), `chunk_count`, the
|
|
53
|
+
resolved `config` (`max_characters`, `overlap`, `chunker_type`), and
|
|
54
|
+
`input_size_bytes`. Use `--format text` for a human-readable dump with
|
|
55
|
+
`--- chunk N ---` separators.
|
|
56
|
+
|
|
57
|
+
> Note: in the JSON output, `chunker_type` is rendered capitalized (`"Text"`,
|
|
58
|
+
> `"Markdown"`, `"Yaml"`, `"Semantic"`) because it is emitted via Rust's Debug
|
|
59
|
+
> formatting, whereas the `--chunker-type` input flag is lowercase
|
|
60
|
+
> (`text`, `markdown`, `yaml`, `semantic`). Lowercase the value before
|
|
61
|
+
> comparing if you parse it back.
|
|
62
|
+
|
|
63
|
+
## Chunker types
|
|
64
|
+
|
|
65
|
+
`--chunker-type` selects the splitting strategy (standalone `chunk`
|
|
66
|
+
command):
|
|
67
|
+
|
|
68
|
+
| Type | Behavior |
|
|
69
|
+
| ---------- | ------------------------------------------------------------------- |
|
|
70
|
+
| `text` | **Default.** Plain character-window splitting with overlap. |
|
|
71
|
+
| `markdown` | Markdown-aware — splits on structure (headings, blocks) where possible. |
|
|
72
|
+
| `yaml` | YAML-aware splitting for structured config/data documents. |
|
|
73
|
+
| `semantic` | Topic-boundary splitting driven by `--topic-threshold` (0.0–1.0, default 0.75). |
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
# Markdown-aware chunking keeps headings and blocks intact
|
|
77
|
+
xberg chunk --text "$(cat README.md)" --chunker-type markdown
|
|
78
|
+
|
|
79
|
+
# Semantic chunking — lower threshold = more, smaller topic chunks
|
|
80
|
+
xberg chunk --text "$(cat transcript.txt)" --chunker-type semantic --topic-threshold 0.6
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
## Token-based sizing
|
|
84
|
+
|
|
85
|
+
By default `--chunk-size` counts characters. To size chunks by tokens for
|
|
86
|
+
a specific model, pass `--chunking-tokenizer` with a HuggingFace tokenizer
|
|
87
|
+
id. On the `extract` command this implicitly enables chunking. Requires the
|
|
88
|
+
`chunking-tokenizers` feature (present in the default CLI build).
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
# Size chunks by GPT-4o tokens during extraction
|
|
92
|
+
xberg extract report.pdf --chunking-tokenizer Xenova/gpt-4o --format json
|
|
93
|
+
|
|
94
|
+
# Or on the standalone command
|
|
95
|
+
xberg chunk --text "$(cat doc.txt)" --chunking-tokenizer Xenova/gpt-4o --chunk-size 512
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
With a tokenizer set, `--chunk-size` is interpreted in tokens, not
|
|
99
|
+
characters.
|
|
100
|
+
|
|
101
|
+
## Config file alternative
|
|
102
|
+
|
|
103
|
+
Field names in config files are snake_case under `[chunking]`:
|
|
104
|
+
|
|
105
|
+
```toml
|
|
106
|
+
[chunking]
|
|
107
|
+
max_characters = 1000
|
|
108
|
+
overlap = 200
|
|
109
|
+
chunker_type = "markdown"
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
xberg extract report.pdf --config xberg.toml --format json
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
> CLI flags map to config fields as `--chunk-size` → `max_characters` and
|
|
117
|
+
> `--chunk-overlap` → `overlap`. In config files use the snake_case names.
|
|
118
|
+
|
|
119
|
+
## Programmatic access
|
|
120
|
+
|
|
121
|
+
From Python, enable chunking on the config and read the chunks off the
|
|
122
|
+
document in the result envelope (`result.results[0].chunks`):
|
|
123
|
+
|
|
124
|
+
```python
|
|
125
|
+
from xberg import ExtractInput, extract, ExtractionConfig, ChunkingConfig
|
|
126
|
+
|
|
127
|
+
config = ExtractionConfig(
|
|
128
|
+
chunking=ChunkingConfig(max_characters=1000, overlap=200),
|
|
129
|
+
)
|
|
130
|
+
result = await extract(ExtractInput(uri="report.pdf"), config)
|
|
131
|
+
for chunk in result.results[0].chunks or []:
|
|
132
|
+
print(len(chunk.content))
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
> The public Python `ChunkingConfig` (a dataclass) uses constructor kwargs
|
|
136
|
+
> `max_characters` / `overlap`; the Rust core struct fields are also
|
|
137
|
+
> `max_characters` / `overlap`. TOML/JSON config keys are `max_chars` /
|
|
138
|
+
> `max_overlap` (with `max_characters` / `overlap` accepted as serde aliases),
|
|
139
|
+
> and dict-form config passed to `ExtractionConfig` likewise accepts the
|
|
140
|
+
> `max_chars` / `max_overlap` aliases; Node's `ChunkingConfig` interface uses
|
|
141
|
+
> `maxCharacters` / `overlap`. See `references/python-api.md` and
|
|
142
|
+
> `references/rust-api.md` in the sibling `xberg` skill.
|
|
143
|
+
|
|
144
|
+
## Picking parameters
|
|
145
|
+
|
|
146
|
+
- **RAG / vector store** — 500–1000 chars (or 256–512 tokens) with
|
|
147
|
+
10–20% overlap. Use `markdown` chunking for docs to keep sections whole.
|
|
148
|
+
- **LLM summarization** — larger chunks (1500–4000 chars) with small
|
|
149
|
+
overlap; size by tokens to stay under the model window.
|
|
150
|
+
- **Topic segmentation** — `semantic` chunker; tune `--topic-threshold`
|
|
151
|
+
down for finer splits, up for coarser ones.
|
|
152
|
+
|
|
153
|
+
## Common pitfalls
|
|
154
|
+
|
|
155
|
+
- **Overlap ≥ size** — rejected on `extract`; clamped to `size / 4` when
|
|
156
|
+
only overlap is changed against an existing config.
|
|
157
|
+
- **Tokenizer without the feature** — `--chunking-tokenizer` errors if the
|
|
158
|
+
CLI was built without `chunking-tokenizers`. The default build includes it.
|
|
159
|
+
- **Empty input** — the standalone `chunk` command bails on empty text;
|
|
160
|
+
provide `--text` or pipe non-empty stdin.
|
|
161
|
+
|
|
162
|
+
See `references/configuration.md` for the full `[chunking]` schema and
|
|
163
|
+
`references/cli-reference.md` for every chunk flag.
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: extracting-keywords
|
|
3
|
+
description: Use when extracting keywords (YAKE/RAKE) from documents — and, secondarily, when detecting document language or generating embeddings for RAG and search. Covers the keyword config (and its feature gating), `--detect-language`, and the standalone `embed` command with real flags.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
<!--
|
|
7
|
+
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
8
|
+
Content-Hash: blake3:0da13ce50f1fef8c192e912e7c2b0d039209a7b676829abe13f4fc86b5ea20bc
|
|
9
|
+
Source-Hash: blake3:3fca38eb564d0ed9c9194d070fd8dec96de727edc92e82954cb4b9af52cddbaf
|
|
10
|
+
Schema-Version: v1
|
|
11
|
+
-->
|
|
12
|
+
|
|
13
|
+
# Extracting keywords, language, and embeddings
|
|
14
|
+
|
|
15
|
+
Use this for the enrichment surface around extraction: statistical keyword
|
|
16
|
+
extraction, language detection, and vector embeddings. Keywords and
|
|
17
|
+
language detection ride along with extraction and land on the result;
|
|
18
|
+
embeddings are produced by a dedicated `embed` command.
|
|
19
|
+
|
|
20
|
+
## Keywords (YAKE / RAKE)
|
|
21
|
+
|
|
22
|
+
Keyword extraction is configured via the `[keywords]` config block (or
|
|
23
|
+
inline JSON) — there is no single `--keywords` CLI flag. When enabled,
|
|
24
|
+
extracted keywords appear on `result.extracted_keywords` (`extractedKeywords`
|
|
25
|
+
in Node.js; the CLI JSON field is `extracted_keywords`). Two algorithms are
|
|
26
|
+
available:
|
|
27
|
+
|
|
28
|
+
- **YAKE** (`"yake"`) — statistical, unsupervised single-document
|
|
29
|
+
extraction. Good general default.
|
|
30
|
+
- **RAKE** (`"rake"`) — co-occurrence / phrase-based. Favors multi-word
|
|
31
|
+
key phrases.
|
|
32
|
+
|
|
33
|
+
> Feature-gated: keyword extraction requires the CLI to be built with the
|
|
34
|
+
> `keywords-yake` and/or `keywords-rake` Cargo features (both are in the
|
|
35
|
+
> default/`full` build). If the CLI was built without them, the `[keywords]`
|
|
36
|
+
> config block is silently ignored — `result.extracted_keywords` simply stays empty
|
|
37
|
+
> rather than erroring. The `"yake"` algorithm needs `keywords-yake`; `"rake"`
|
|
38
|
+
> needs `keywords-rake`.
|
|
39
|
+
|
|
40
|
+
Enable via inline JSON on the CLI:
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
xberg extract paper.pdf --format json \
|
|
44
|
+
--config-json '{"keywords":{"algorithm":"yake","max_keywords":15,"language":"en"}}' \
|
|
45
|
+
| jq '.extracted_keywords'
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Or in a config file:
|
|
49
|
+
|
|
50
|
+
```toml
|
|
51
|
+
[keywords]
|
|
52
|
+
algorithm = "rake" # "yake" or "rake"
|
|
53
|
+
max_keywords = 10 # default 10
|
|
54
|
+
min_score = 0.0 # filter below this score (normalized 0.0-1.0 for both algorithms)
|
|
55
|
+
ngram_range = [1, 3] # unigrams..trigrams (default); config-file only
|
|
56
|
+
language = "en" # stopword language; omit to skip stopword filtering
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
xberg extract report.pdf --config xberg.toml --format json | jq '.extracted_keywords'
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Field notes:
|
|
64
|
+
|
|
65
|
+
- `max_keywords` caps how many keywords are returned (default 10).
|
|
66
|
+
- `min_score` filters low-scoring keywords. Both YAKE and RAKE normalize
|
|
67
|
+
their scores to the `0.0`-`1.0` range with *higher-is-better*, so
|
|
68
|
+
`min_score` retains keywords with `score >= min_score` identically for
|
|
69
|
+
either algorithm.
|
|
70
|
+
- `ngram_range` is `[min, max]`: `[1,1]` unigrams only, `[1,2]` adds
|
|
71
|
+
bigrams, `[1,3]` (default) adds trigrams. Config-file only — it is not a
|
|
72
|
+
field on the language bindings' `KeywordConfig`.
|
|
73
|
+
- `language` enables stopword filtering for that language; omit it to
|
|
74
|
+
disable stopword filtering entirely.
|
|
75
|
+
|
|
76
|
+
## Language detection
|
|
77
|
+
|
|
78
|
+
Language detection is a real CLI flag: `--detect-language`. Detected
|
|
79
|
+
languages appear on `result.detected_languages`:
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
xberg extract multilingual.pdf --detect-language true --format json \
|
|
83
|
+
| jq '.detected_languages'
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
In a config file it lives under `[language_detection]`:
|
|
87
|
+
|
|
88
|
+
```toml
|
|
89
|
+
[language_detection]
|
|
90
|
+
enabled = true
|
|
91
|
+
min_confidence = 0.8
|
|
92
|
+
detect_multiple = false
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The CLI flag enables detection with `min_confidence = 0.8` and
|
|
96
|
+
single-language mode; use the config block to detect multiple languages or
|
|
97
|
+
tune confidence.
|
|
98
|
+
|
|
99
|
+
## Embeddings (`embed` command)
|
|
100
|
+
|
|
101
|
+
The standalone `embed` command produces vector embeddings for text from
|
|
102
|
+
`--text` (repeatable) or stdin. It does not run extraction — pipe
|
|
103
|
+
extracted content in if you want document embeddings.
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
# Local ONNX preset model (default provider)
|
|
107
|
+
xberg embed --text "first passage" --text "second passage" --preset balanced
|
|
108
|
+
|
|
109
|
+
# Embed extracted document text
|
|
110
|
+
xberg extract report.pdf | xberg embed --preset quality
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Presets for the local provider: `fast`, `balanced` (default), `quality`,
|
|
114
|
+
`multilingual`. Output defaults to JSON (`--format json`).
|
|
115
|
+
|
|
116
|
+
`--provider` selects the embedding source:
|
|
117
|
+
|
|
118
|
+
| Provider | Flag | Notes |
|
|
119
|
+
| -------- | ------------------------------------- | --------------------------------------------- |
|
|
120
|
+
| `local` | `--preset <fast\|balanced\|quality\|multilingual>` | **Default.** ONNX model, no API key. |
|
|
121
|
+
| `llm` | `--model <id>` `--api-key <key>` | liter-llm routing, e.g. `openai/text-embedding-3-small`. |
|
|
122
|
+
| `plugin` | `--plugin <name>` | A backend pre-registered in-process via the plugin API. |
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
# Provider-hosted embeddings via an LLM
|
|
126
|
+
xberg embed --text "query text" \
|
|
127
|
+
--provider llm --model openai/text-embedding-3-small --api-key "$OPENAI_API_KEY"
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Local embedding presets must be downloaded first if not cached. Pre-warm
|
|
131
|
+
them with the cache command:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
xberg cache warm --embedding-model balanced # one preset
|
|
135
|
+
xberg cache warm --all-embeddings # all available presets (currently 8)
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
## Programmatic access
|
|
139
|
+
|
|
140
|
+
Keywords and detected languages live on the document in the result envelope:
|
|
141
|
+
|
|
142
|
+
```python
|
|
143
|
+
from xberg import ExtractInput, extract, ExtractionConfig, KeywordConfig, KeywordAlgorithm
|
|
144
|
+
|
|
145
|
+
config = ExtractionConfig(
|
|
146
|
+
keywords=KeywordConfig(algorithm=KeywordAlgorithm.YAKE, max_keywords=15, language="en"),
|
|
147
|
+
)
|
|
148
|
+
result = await extract(ExtractInput(uri="paper.pdf"), config)
|
|
149
|
+
doc = result.results[0]
|
|
150
|
+
print(doc.extracted_keywords) # extracted keywords (when enabled)
|
|
151
|
+
print(doc.detected_languages) # detected languages (when enabled)
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
See `references/python-api.md` and `references/configuration.md` in the
|
|
155
|
+
sibling `xberg` skill for the keyword / language-detection config
|
|
156
|
+
classes and the embedding presets.
|
|
157
|
+
|
|
158
|
+
## Common pitfalls
|
|
159
|
+
|
|
160
|
+
- **No `--keywords` flag** — keyword extraction is config-only. Use
|
|
161
|
+
`--config-json '{"keywords":{...}}'` or a `[keywords]` config block.
|
|
162
|
+
- **`min_score` direction** — scores are normalized to `0.0`-`1.0` with
|
|
163
|
+
higher-is-better for both YAKE and RAKE, so the same threshold behaves
|
|
164
|
+
identically for either algorithm.
|
|
165
|
+
- **Embeddings ≠ extraction** — `embed` only takes raw text. Pipe
|
|
166
|
+
`xberg extract` output into it for document vectors.
|
|
167
|
+
- **Cold embedding models** — first local run downloads the preset; run
|
|
168
|
+
`xberg cache warm --all-embeddings` to pre-populate.
|
|
169
|
+
|
|
170
|
+
See `references/advanced-features.md` for the embeddings pipeline and
|
|
171
|
+
`references/cli-reference.md` for the `embed` and `cache warm` flag sets.
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: extracting-tables
|
|
3
|
+
description: Use when extracting tabular data from PDFs, spreadsheets, or images. Covers layout-aware table detection, table model selection, output formats (markdown / JSON cells), and known limits.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
<!--
|
|
7
|
+
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
8
|
+
Content-Hash: blake3:7667a52a8674605a45cc61b67e7879a0104d5e86c0d82b4bde5ced9e6e3463a8
|
|
9
|
+
Source-Hash: blake3:3fca38eb564d0ed9c9194d070fd8dec96de727edc92e82954cb4b9af52cddbaf
|
|
10
|
+
Schema-Version: v1
|
|
11
|
+
-->
|
|
12
|
+
|
|
13
|
+
# Extracting tables
|
|
14
|
+
|
|
15
|
+
Use this when the user wants structured tabular data — financial
|
|
16
|
+
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
|
|
17
|
+
detects tables via a layout model (RT-DETR v2) and reconstructs cell
|
|
18
|
+
structure with a configurable table model.
|
|
19
|
+
|
|
20
|
+
## Basic usage
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
# Markdown tables embedded in the content stream
|
|
24
|
+
xberg extract report.pdf --layout --content-format markdown
|
|
25
|
+
|
|
26
|
+
# Structured JSON output, tables appear under result.tables
|
|
27
|
+
xberg extract report.pdf --layout --format json
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
`--layout` turns on layout-aware extraction; without it, tables fall back
|
|
31
|
+
to plain text reflow and you lose cell boundaries.
|
|
32
|
+
|
|
33
|
+
## Output shapes
|
|
34
|
+
|
|
35
|
+
Two surfaces, picked via `--format` (CLI shape) and `--content-format`
|
|
36
|
+
(content rendering):
|
|
37
|
+
|
|
38
|
+
- **Markdown tables in `content`** — `--content-format markdown`. Tables
|
|
39
|
+
appear inline as `| col | col |` blocks. Good for LLM ingestion.
|
|
40
|
+
- **Structured `tables` array** — `--format json`. Each entry has
|
|
41
|
+
`cells[][]` (rows × cols), `markdown` (pre-rendered), `page_number`,
|
|
42
|
+
`bounding_box`. Use this when downstream code needs exact cell access.
|
|
43
|
+
(`bounding_box` is omitted when no position data is available.)
|
|
44
|
+
|
|
45
|
+
Both are populated at once when `--layout` is on. The `tables` array is
|
|
46
|
+
always structured; the `content` stream switches representation.
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
xberg extract financials.pdf --layout --format json \
|
|
50
|
+
| jq '.result.tables[] | {page: .page_number, rows: (.cells | length)}'
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Table models
|
|
54
|
+
|
|
55
|
+
`--layout-table-model` picks the reconstruction backend:
|
|
56
|
+
|
|
57
|
+
| Model | Best for | Notes |
|
|
58
|
+
| ------------------ | ----------------------------------------------------- | ------------------------------------------- |
|
|
59
|
+
| `tatr` | dense complex tables (academic, financial) | **Default.** Heaviest, highest accuracy. |
|
|
60
|
+
| `slanet_auto` | dispatches per-table to wired/wireless | Good when table styles are mixed. |
|
|
61
|
+
| `slanet_wired` | tables with visible borders | Faster than tatr. |
|
|
62
|
+
| `slanet_wireless` | tables without borders (whitespace-separated) | For invoices, simple grids. |
|
|
63
|
+
| `slanet_plus` | hybrid wired / wireless | Lighter than `slanet_auto`. |
|
|
64
|
+
| `disabled` | layout detection only, no table structure | Use to skip table model cost. |
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
xberg extract bank-statement.pdf \
|
|
68
|
+
--layout --layout-table-model tatr --content-format markdown
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Drop `--layout-confidence` when the layout model misses tables (default
|
|
72
|
+
threshold ~0.5):
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
xberg extract noisy-scan.pdf --layout --layout-confidence 0.3
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Spreadsheets
|
|
79
|
+
|
|
80
|
+
`.xlsx`, `.ods`, `.csv`, `.tsv` are extracted by dedicated parsers — no
|
|
81
|
+
layout model needed. Each sheet becomes a markdown table (or structured
|
|
82
|
+
table) automatically:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
xberg extract workbook.xlsx --content-format markdown
|
|
86
|
+
xberg extract data.csv --format json
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Pass `--no-cache=true` only when iterating on the same file with different
|
|
90
|
+
configs.
|
|
91
|
+
|
|
92
|
+
## Config file alternative
|
|
93
|
+
|
|
94
|
+
```toml
|
|
95
|
+
# `output_format` in config files equals `--content-format` on the CLI.
|
|
96
|
+
output_format = "markdown"
|
|
97
|
+
|
|
98
|
+
[layout]
|
|
99
|
+
confidence_threshold = 0.5
|
|
100
|
+
table_model = "tatr"
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Then:
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
xberg extract report.pdf --format json
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
## Programmatic access
|
|
110
|
+
|
|
111
|
+
From Python, structured tables live on the document in the result envelope
|
|
112
|
+
(`result.results[0].tables`):
|
|
113
|
+
|
|
114
|
+
```python
|
|
115
|
+
from xberg import ExtractInput, extract, ExtractionConfig, LayoutDetectionConfig
|
|
116
|
+
|
|
117
|
+
config = ExtractionConfig(
|
|
118
|
+
layout=LayoutDetectionConfig(table_model="tatr"),
|
|
119
|
+
output_format="markdown",
|
|
120
|
+
)
|
|
121
|
+
result = await extract(ExtractInput(uri="report.pdf"), config)
|
|
122
|
+
for table in result.results[0].tables:
|
|
123
|
+
print(table.markdown) # rendered markdown
|
|
124
|
+
print(table.cells[0][0]) # cell access
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
Node.js mirrors this (`extract`, `output.results[0].tables`, camelCase fields).
|
|
128
|
+
See `references/python-api.md` and `references/nodejs-api.md` in the
|
|
129
|
+
sibling `xberg` skill for full type signatures.
|
|
130
|
+
|
|
131
|
+
## Known limitations
|
|
132
|
+
|
|
133
|
+
- **Merged cells** — reconstructed as repeated values across the spanned
|
|
134
|
+
region; the merge is not preserved as metadata.
|
|
135
|
+
- **Rotated tables** — enable `--ocr-auto-rotate true` for image-based
|
|
136
|
+
PDFs before extraction.
|
|
137
|
+
- **Nested tables** — flattened. Detection succeeds; structural nesting is
|
|
138
|
+
lost.
|
|
139
|
+
- **Multi-page tables** — each page yields a separate `tables[]` entry.
|
|
140
|
+
Stitch by matching column headers if needed.
|
|
141
|
+
- **ONNX Runtime required** — layout and table models are unavailable in
|
|
142
|
+
WASM builds and on the Android x86_64 emulator; native targets ship
|
|
143
|
+
full support.
|
|
144
|
+
|
|
145
|
+
## Common failure modes
|
|
146
|
+
|
|
147
|
+
- **Empty `tables` with `--layout` on** — confidence threshold too high or
|
|
148
|
+
table model mismatched. Drop `--layout-confidence` to 0.3, try
|
|
149
|
+
`--layout-table-model tatr`.
|
|
150
|
+
- **Markdown tables look ragged** — switch `--layout-table-model` to
|
|
151
|
+
`slanet_wired` for bordered grids or `slanet_wireless` for invoices.
|
|
152
|
+
- **Slow extraction** — `tatr` is heavy. Use `slanet_auto` or
|
|
153
|
+
`slanet_plus` as a default; reach for `tatr` only when accuracy matters.
|
|
154
|
+
|
|
155
|
+
See `references/cli-reference.md` for the full layout flag set and
|
|
156
|
+
`references/advanced-features.md` for the layout pipeline internals.
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: extracting-with-ocr
|
|
3
|
+
description: Use when extracting text from scanned PDFs, photographed pages, or images that have no embedded text layer. Covers OCR backends, language packs, force-OCR, and performance tuning.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
<!--
|
|
7
|
+
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
|
|
8
|
+
Content-Hash: blake3:3ec8b7cf60f56cbe5cc15a5a0b29d0c2f8e3cf4c3823503eb1128fb7f9ee11db
|
|
9
|
+
Source-Hash: blake3:3fca38eb564d0ed9c9194d070fd8dec96de727edc92e82954cb4b9af52cddbaf
|
|
10
|
+
Schema-Version: v1
|
|
11
|
+
-->
|
|
12
|
+
|
|
13
|
+
# Extracting with OCR
|
|
14
|
+
|
|
15
|
+
Use this when a document is image-based: scanned PDFs, photographed pages,
|
|
16
|
+
screenshots, JPEG/PNG/TIFF with text. Xberg auto-OCRs raster images and
|
|
17
|
+
auto-detects PDFs that lack a text layer. Force it on when extraction
|
|
18
|
+
returned empty/garbled text from a PDF that "looks" textual.
|
|
19
|
+
|
|
20
|
+
## When to force OCR
|
|
21
|
+
|
|
22
|
+
- Extraction returned an empty `content` field, but the file opens visually.
|
|
23
|
+
- The PDF text layer is junk (copy-paste from a viewer produces gibberish).
|
|
24
|
+
- You want consistent output across mixed scanned + digital PDFs.
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
xberg extract scan.pdf --force-ocr=true
|
|
28
|
+
xberg extract scan.pdf --ocr=true --ocr-language eng
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
If a page has an unreliable text layer, `--force-ocr=true` re-rasterizes
|
|
32
|
+
and runs OCR on every page.
|
|
33
|
+
|
|
34
|
+
## Backends
|
|
35
|
+
|
|
36
|
+
Tesseract is the default and ships with the CLI — no extra install. Other
|
|
37
|
+
backends are opt-in:
|
|
38
|
+
|
|
39
|
+
| Backend | Flag | Install | Notes |
|
|
40
|
+
| ------------- | ------------------------------------- | ------------------------------------------------ | -------------------------------------------------------------- |
|
|
41
|
+
| Tesseract | `--ocr-backend tesseract` (default) | bundled | Best general-purpose, 100+ languages via tessdata. |
|
|
42
|
+
| PaddleOCR | `--ocr-backend paddle-ocr` | bundled (ONNX Runtime) | Strong on Asian scripts. Not available on WASM or Windows. |
|
|
43
|
+
| Candle VLM | `--ocr-backend candle-trocr` (and other `candle-*`) | bundled (Candle) | Local vision OCR models (`candle-trocr`, `candle-paddleocr-vl`, `candle-glm-ocr`, `candle-deepseek-ocr`). |
|
|
44
|
+
| VLM (hosted) | `--ocr-backend vlm` + `--vlm-model` | liter-llm provider (`--vlm-api-key`) | Multimodal LLM via liter-llm. Use when OCR fails on dense or handwritten layouts. |
|
|
45
|
+
|
|
46
|
+
Pick Tesseract first. Switch only when accuracy is unacceptable.
|
|
47
|
+
|
|
48
|
+
## Language packs
|
|
49
|
+
|
|
50
|
+
Tesseract uses ISO 639-2 codes. Default is `eng`. Combine with `+`:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
xberg extract menu.jpg --ocr=true --ocr-language "eng+deu"
|
|
54
|
+
xberg extract bilingual.pdf --ocr-language "eng+jpn"
|
|
55
|
+
xberg extract any.pdf --ocr-language all # all installed packs
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Install missing packs at the OS level:
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
# macOS
|
|
62
|
+
brew install tesseract-lang
|
|
63
|
+
|
|
64
|
+
# Debian/Ubuntu
|
|
65
|
+
sudo apt install tesseract-ocr-deu tesseract-ocr-jpn tesseract-ocr-fra
|
|
66
|
+
|
|
67
|
+
# Specific lang only
|
|
68
|
+
sudo apt install tesseract-ocr-<iso639-2>
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Xberg fails fast with a helpful error if you request a language pack
|
|
72
|
+
that is not installed. Read the error — it names the missing file.
|
|
73
|
+
|
|
74
|
+
## Useful flags
|
|
75
|
+
|
|
76
|
+
- `--ocr=true` — enable OCR (auto-enabled for images and scanned PDFs).
|
|
77
|
+
- `--force-ocr=true` — OCR every page even if a text layer exists.
|
|
78
|
+
- `--disable-ocr=true` — never OCR (extract embedded text only or fail).
|
|
79
|
+
- `--ocr-language <lang>` — single code or `+`-joined list, or `all`.
|
|
80
|
+
- `--ocr-backend <tesseract|paddle-ocr|vlm|candle-trocr|candle-paddleocr-vl|candle-glm-ocr|candle-deepseek-ocr>` — pick backend.
|
|
81
|
+
- `--ocr-auto-rotate=true` — pre-rotate via the auto-rotate model.
|
|
82
|
+
- `--acceleration <cpu|coreml|cuda|tensorrt|auto>` — ONNX accelerator for
|
|
83
|
+
paddle-ocr / auto-rotate / layout models.
|
|
84
|
+
|
|
85
|
+
## Performance tips
|
|
86
|
+
|
|
87
|
+
- Cache is on by default. Repeated extraction of the same file + config is
|
|
88
|
+
instant. Do not pass `--no-cache=true` unless you have a reason.
|
|
89
|
+
- For batch OCR, use `xberg batch *.pdf --ocr=true` — internal worker
|
|
90
|
+
pool parallelizes across CPU cores. Cap with `--max-concurrent N` if
|
|
91
|
+
memory is tight.
|
|
92
|
+
- Raise `--target-dpi` (default 300) only for low-resolution scans. Higher
|
|
93
|
+
DPI is slower; 200 is usually enough for printed text.
|
|
94
|
+
- Enable `--ocr-auto-rotate=true` only when pages may be rotated; the
|
|
95
|
+
classifier adds latency.
|
|
96
|
+
- On Apple Silicon, `--acceleration coreml` typically beats CPU for
|
|
97
|
+
paddle-ocr and layout detection.
|
|
98
|
+
|
|
99
|
+
## Config file alternative
|
|
100
|
+
|
|
101
|
+
Long flag chains belong in `xberg.toml` — auto-discovered from cwd
|
|
102
|
+
upward.
|
|
103
|
+
|
|
104
|
+
```toml
|
|
105
|
+
force_ocr = true
|
|
106
|
+
output_format = "markdown"
|
|
107
|
+
|
|
108
|
+
[ocr]
|
|
109
|
+
backend = "tesseract"
|
|
110
|
+
language = "eng+deu"
|
|
111
|
+
auto_rotate = true
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Then just run:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
xberg extract document.pdf
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## Common failure modes
|
|
121
|
+
|
|
122
|
+
- **"missing tessdata"** — install the language pack at OS level (see above).
|
|
123
|
+
- **Empty content on a scanned PDF without `--force-ocr`** — the file has a
|
|
124
|
+
bogus zero-width text layer. Re-run with `--force-ocr=true`.
|
|
125
|
+
- **OCR on a rotated page** — add `--ocr-auto-rotate=true` or pre-rotate.
|
|
126
|
+
- **Garbled CJK output** — ensure the right language pack is installed and
|
|
127
|
+
passed via `--ocr-language`; consider `paddle-ocr` for Chinese/Japanese.
|
|
128
|
+
|
|
129
|
+
See `references/cli-reference.md` and `references/configuration.md` in the
|
|
130
|
+
sibling `xberg` skill for the full flag and config schema.
|