@autorag/librarian 2.13.3 → 2.13.5

package/NOTICE

@@ -36,6 +36,9 @@ License: Apache-2.0
 fast-xml-parser@5.11.2
 License: MIT
 
+gpt-tokenizer@4.1.0
+License: MIT
+
 iconv-lite@0.7.3
 License: MIT
 
@@ -75,6 +78,9 @@ License: Apache-2.0
 typebox@1.3.37
 License: MIT
 
+xlsx@0.20.3
+License: Apache-2.0
+
 Bundled Windows binaries (vendor/everything, extracted only on Windows):
 Everything 1.4.1.1032 portable (voidtools, David Carpenter)
 License: MIT (closed source); includes PCRE (BSD-3-Clause)

package/README.md

@@ -187,11 +187,11 @@ AutoRAG Agent:
 
 Three principles drive every design decision in AutoRAG Agent:
 
-1. **Never migrate your data to search it.** Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data **in place**, querying CLI-native stores (`lazykatok`, `discrawl`, `slacrawl`, `mailcrawl`, `rclone`) where your data already lives. Results retain opaque, source-native identities (`/kakao/...`, `/slack/...`) that preserve local access control and privacy. *(See our [Competitive Landscape Study](docs/competitive-landscape-2026-09.md) on why in-place federation is the durable differentiator).*
+1. **Never migrate your data to search it.** Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data **in place**, querying CLI-native stores (`lazykatok`, `discrawl`, `slacrawl`, `mailcrawl`) where your data already lives. Results retain opaque, source-native identities (`/kakao/...`, `/slack/...`) that preserve local access control and privacy. *(See our [Competitive Landscape Study](docs/competitive-landscape-2026-09.md) on why in-place federation is the durable differentiator).*
 
-2. **Just works — no RAG degree required.** No pipeline tuning, no vector-DB maintenance, and no remote embedding keys required. AutoRAG Agent automatically manages [MinSync](docs/minsync-setup.md) for incremental Change Data Capture (CDC) chunking and provides a local [embedding backend](docs/minsync-setup.md#embedding-backends) out of the box with zero external telemetry. A [remote rerank model](docs/rerank.md) (the hosted Dazzi Models reranker when signed in, else OpenRouter `voyageai/rerank-3-lite`) is an opt-in extra.
+2. **Just works — no RAG degree required.** No pipeline tuning, no vector-DB maintenance, and no remote embedding keys required. AutoRAG Agent automatically manages [MinSync](docs/minsync-setup.md) for incremental Change Data Capture (CDC) chunking and provides a local [embedding backend](docs/minsync-setup.md#embedding-backends) out of the box with zero external telemetry. A [remote rerank model](docs/rerank.md) is an opt-in extra: it runs only when you name one in `rerank.model`, and otherwise the first answer is built from each source's own top 10.
 
-3. **Fast by design.** Rather than coordinating slow multi-agent hierarchies, a single configured model owns the entire retrieval, direct-read, and curation loop. Local CDC chunks (BM25, vector, and hybrid modes) deliver rapid, low-latency turnaround across multi-turn research queries.
+3. **Fast by design.** Rather than coordinating slow multi-agent hierarchies, a single configured model owns the entire retrieval, direct-read, and curation loop. Local CDC chunks (SPLADE sparse, vector, and hybrid modes) deliver rapid, low-latency turnaround across multi-turn research queries.
 
 ---
 
@@ -201,9 +201,9 @@ AutoRAG Agent orchestrates five integrated subsystems:
 
 1. **Retrieval Memory (`check_memory`):** After each answer, cited evidence Jev judges to genuinely support it is stored in `~/.autorag/memory.json`. Later searches receive this conversation's evidence, similar past questions, and long-term insights as **advisory reference** — it never reorders results or overrides current evidence. See [docs/retrieval-memory.md](docs/retrieval-memory.md).
 2. **Pluggable Multi-Method Retrieval:**
-   - **BM25 Lexical Search:** Fast keyword ranking via MinSync over parsed markdown mirrors. AutoRAG always uses MinSync's `multilingual` tokenizer, which keeps each original word and adds Korean morphemes (via the [Kiwi](https://github.com/bab2min/Kiwi) library), so both `연구목표` and `인증서를` → `인증서` match; without Kiwi AutoRAG warns and tokenizes on whitespace. See [docs/minsync-setup.md](docs/minsync-setup.md#bm25-tokenizer).
+   - **SPLADE Sparse Search:** Learned-sparse keyword ranking via MinSync's KURE-Sparse-nano encoder over parsed markdown mirrors; it replaced BM25 in MinSync 0.6 and matches Korean inflected forms (`인증서를` → `인증서`) with no tokenizer library. Exact identifiers among many look-alike files rank less reliably than with BM25. See [docs/minsync-setup.md](docs/minsync-setup.md#sparse-retrieval-splade).
    - **Semantic Vector Search:** Dense vector retrieval over CDC chunks via the shared MinSync embedder.
-   - **Hybrid Search:** Combines BM25 and vector ranking via Reciprocal Rank Fusion (RRF).
+   - **Hybrid Search:** MinSync fuses sparse and vector ranking over the same chunks.
    - **Jikji Find-First Discovery:** Local CLI-backed fast discovery answer packs.
    - **Everything File-Name Search (Windows):** The bundled [voidtools Everything](https://www.voidtools.com/) indexes file and folder names under your search roots so the agent finds files by name, extension, path, size, or date instantly (`everything_search`).
    - **FSearch File-Name Search (macOS/Linux):** [fsearch-cli](https://github.com/NomaDamas/fsearch-mac) (FSearch) keeps a live, per-workspace name index of your search roots so the agent finds files by name, extension, path, size, or date instantly (`fsearch_search`); without fsearch-cli installed it degrades to a slow filesystem walk.
@@ -260,7 +260,7 @@ bun add @autorag/librarian
 ### System Prerequisites
 
 - **No Java required:** Document parsing (HWP/HWPX/HWPML, PDF, DOCX, XLSX/XLS) runs in-process through [`kordoc`](https://github.com/chrisryugj/kordoc).
-- **Rust Toolchain (Optional):** Automatically compiles Jikji (`jikji-cli`) if installed.
+- **Rust Toolchain (Optional):** Automatically compiles Jikji (`jikji-cli`, pinned to 0.2.1) if installed. A `jikji` older than 0.2.0 on your PATH is skipped and replaced by the pinned build on the next refresh.
 - **MinSync:** Automatically downloaded and installed into `<workspace>/.autorag/bin` on first run.
 - **Everything (Windows only, bundled):** The package ships the portable Everything 1.4.1.1032 and its ES 1.1.0.38 CLI (x64 and ARM64, SHA-256 pinned). On Windows, AutoRAG extracts them into `<workspace>/.autorag/everything/` and runs a private, user-level instance that indexes only your configured search roots: no administrator rights, no Everything service, no whole-drive scan, no HTTP/ETP server, and no change to any Everything you already run. Set `"everything": false` in the config to turn it off. macOS and Linux are not affected.
 - **filee-cli (file conversion, automatic):** `autorag setup` and the TUI download the pinned filee-cli release (~85 MB, SHA-256 verified) into `<AUTORAG_HOME>/tools/filee-cli/<version>/`, so the agent can convert between 255 formats offline with `convert_files`. A `filee`/`filee-cli` already on `PATH` is used instead. Set `"filee": { "autoInstall": false }` to forbid the download or `"filee": { "enabled": false }` to remove the tools. Filee fetches its own engines (FFmpeg, Pandoc, LibreOffice, calibre, Ghostscript; hundreds of MB, only when a conversion needs one) on first use.
@@ -279,7 +279,7 @@ autorag init --search-paths ~/Documents/research
 # Korean + English is the default; set it explicitly for other corpora
 autorag init --search-paths ~/Documents/research --languages ja,en
 
-# Index documents (parses PDFs/Markdown, builds BM25 and MinSync vectors)
+# Index documents (parses PDFs/Markdown, builds MinSync sparse and dense vectors)
 autorag refresh
 
 # Check indexing health
@@ -314,7 +314,9 @@ falling back to `["ko", "en"]`. Supported tags: `ko`, `en`, `ja`, `zh-hans`,
 
 | Extension | Parser |
 |---|---|
-| `.hwp` `.hwpx` `.hml` `.hwpml` `.pdf` `.docx` `.xlsx` `.xls` | `kordoc` (nested tables, per-sheet workbooks, no Java) |
+| `.hwp` `.hwpx` `.hml` `.hwpml` `.pdf` `.docx` | `kordoc` (nested tables, no Java) |
+| `.xlsx` `.xlsm` `.xlsb` `.xls` `.ods` | built-in spreadsheet reader (SheetJS) |
+| `.csv` `.tsv` | built-in CSV reader (CP949/EUC-KR aware) |
 | `.pptx` | built-in PPTX reader |
 | `.eml` | built-in mail reader |
 | `.txt` `.text` `.md` `.markdown` | plain text (CP949/EUC-KR aware) |
@@ -322,6 +324,34 @@ falling back to `["ko", "en"]`. Supported tags: `ko`, `en`, `ja`, `zh-hans`,
 
 `.bmp` and `.tiff` are not parsed: kordoc rejects both as unsupported.
 
+The spreadsheet reader writes one line per row. Each line names the sheet, the
+cell range and the table title, and pairs every value with its column label, so
+a search hit always says what a number is:
+
+```text
+[예산총괄!A4:F4 · 2026년 연구개발비 편성표 (단위: 천원)] 구분: 직접비 | 비목: 인건비 | 편성액: 742,000 | 비율(%): 32.1%
+```
+
+Values are what the spreadsheet shows: cached formula results with their number
+formats (`742,000`, `32.1%`, dates). A format that scales the stored number keeps
+that number too (`742천원 (742000)`), and a formula saved without a result shows
+its formula (`=SUM(B2:B3)`). Titles above a table become its caption, merged
+group headers become `상반기 > 1월`, several tables on one sheet are kept apart,
+and a row longer than a search chunk is split into lines that repeat its first
+value. A first row counts as the header only with clear evidence (a merged group
+header, or at least three rows below it with a mostly numeric, date or formula
+column); otherwise rows are kept exactly as written, so a form's label/value
+cells are never relabelled. Sheets are read up to 100,000 rows each; a longer
+sheet is indexed partially with a `parser-warning` diagnostic.
+
+`.csv` and `.tsv` files produce the same lines, with the file name in place of
+the sheet name. Bytes decode as UTF-8 (with or without BOM) or CP949/EUC-KR,
+quoted fields keep separators, line breaks and doubled quotes, and values stay
+exactly as written (`00123` and `=1+2` are not converted). `.tsv` is
+tab-separated; a `.csv` follows Excel's `sep=;` first line when present and
+otherwise picks the separator (`,` `;` tab `|`) that splits its first lines
+most consistently.
+
 ### Making photos and scans searchable
 
 Image files and scanned PDF pages become searchable through one switch,
@@ -348,7 +378,7 @@ explicit `parserOptions.kordoc.ocr` overrides the language-based choice.
 
 ### Bounding and cancelling a parse
 
-`kordoc` (PDF, HWP/HWPX, DOCX, XLSX/XLS, images), `.pptx` and `.eml` parse in a worker
+`kordoc` (PDF, HWP/HWPX, DOCX, images), the spreadsheet reader, `.pptx` and `.eml` parse in a worker
 thread during a refresh, so one corrupt or hostile file cannot hang or crash the run. A
 `Promise.race` deadline cannot stop a synchronous parse; terminating the worker can.
 
@@ -480,7 +510,7 @@ behaves as before.
 | `autorag search "<query>"` | Run the librarian agent to curate structured answers |
 | `autorag status` | Inspect corpus freshness, indexing status, and vector readiness |
 | `autorag health` | Check model provider authentication, token validity, and API reachability |
-| `autorag models list` | List chat models the pi runtime can resolve (built-ins, `models.json`, custom/extension providers) with provider auth status; never prints credential values |
+| `autorag models list` | List chat models the pi runtime can resolve (built-ins, `models.json`, custom/extension providers) with provider auth status; `--decision` lists decision models (`jev.backend`/`jev.model` candidates) instead; never prints credential values |
 | `autorag update-check` | Compare the running `autorag` against the published npm version (also runs on `autorag tui` launch) |
 | `autorag tui` | Open Pi's interactive librarian TUI (`/login`, `/model`, `/resume`, …); the agent can stop and ask you a multiple-choice or open question through the [`ask_user` tool](docs/ask-user.md); `--telegram` / `--whatsapp` / `--discord` add the Telegram / WhatsApp / Discord remote-control bridges; `--permission-mode ask\|auto\|full` (Shift+Tab, `/permissions`) sets the [approval gate for file changes](docs/permissions.md) |
 | `autorag duplicates [DIR]` | Read-only scan for exact and near-duplicate document families with `dupey` |
@@ -653,7 +683,7 @@ AutoRAG Agent connects to external tools and communication platforms using dedic
 
 | Datasource | Skill Alias | Backend / Driver | Storage & Privacy Model | Search Capabilities |
 |---|---|---|---|---|
-| **Local Documents** | `local` | Native filesystem & MinSync | Workspace-local parsed mirrors (`.autorag/`) | BM25, Semantic, Hybrid |
+| **Local Documents** | `local` | Native filesystem & MinSync | Workspace-local parsed mirrors (`.autorag/`) | Sparse (SPLADE), Semantic, Hybrid |
 | **KakaoTalk** | `kakao` | [`lazykatok`](https://github.com/changeroa/lazykatok) CLI | Native KakaoTalk archive; zero direct DB access | BM25, Semantic, Hybrid |
 | **Discord** | `discord` | [`discrawl`](https://github.com/openclaw/discrawl) CLI | Native SQLite archive; token-free wiretap mode | BM25 + native semantic hybrid ranking by default via a [read-only sidecar](docs/datasource-skills.md#discord-bm25-ranking); native FTS supplements lexical recall |
 | **WhatsApp** | `whatsapp` | [`wacrawl`](https://github.com/openclaw/wacrawl) CLI | Local-first incremental archive + FTS5 | Lexical FTS5; BM25 + Semantic + Hybrid by default via [`sidecar`](docs/datasource-skills.md#hybrid-search-for-chat-crawlers-sidecar) |
@@ -661,13 +691,12 @@ AutoRAG Agent connects to external tools and communication platforms using dedic
 | **Slack** | `slack` | [`slacrawl`](https://github.com/openclaw/slacrawl) CLI | Local workspace/channel/thread archive + FTS5 | Lexical FTS5; BM25 + Semantic + Hybrid by default via [`sidecar`](docs/datasource-skills.md#hybrid-search-for-chat-crawlers-sidecar) |
 | **Notion** | `notion` | [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | Local page/database/block archive + FTS5 | Lexical FTS5 |
 | **Email Archives** | `mailcrawl` | [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | Local Gmail, IMAP, Maildir, and `.eml`/`.mbox` mailbox backups (`--source files`) | BM25, Semantic, Hybrid |
+| **Coding-Agent Sessions** | `wombat` | [`wombat`](https://github.com/NomaDamas/wombat) CLI (`cargo install wombat-sessions`) | Codex, Claude Code, OpenCode, pi, and omp conversations; thinking, tool output, and diffs are not indexed | Keyword (SPLADE), Semantic, Hybrid |
 | **Obsidian Vaults** | `obsidian` | Indexed with MinSync in its own workspace (no external CLI) | Markdown notes, folder-scopable | Lexical (BM25), Semantic, Scoped |
 | **GitHub** | `github` | GitHub REST API | In-memory fetched Issues and Pull Requests | Lexical, Scoped |
 | **GitHub Gists** | `github-gist` | GitHub REST API | Incremental local index of own-account public and permitted secret Gist content/metadata; token not stored | Lexical (BM25), Local Semantic, Scoped |
-| **Cloud Drives** | `cloud-drive` | [`rclone`](https://rclone.org) CLI | Google Drive (Tier-1), OneDrive, Dropbox, etc. | Incremental Mirror + BM25 |
 | **Photos & Shots** | `clawgallery` | `clawgallery` CLI | Local screenshot and photo store | Hybrid OCR/Visual Search |
 | **RSS / News** | `rss` | Native HTTP Poller | RSS 2.0 & Atom feeds (24h deduplication) | Lexical |
-| **macOS Spotlight** | `spotlight` | Native `mdfind` CLI | macOS system metadata and content index | System Native |
 | **Browser History** | `hister` | [Hister](https://github.com/asciimoo/hister) server (AGPL-3.0-or-later, separate program, HTTP API) | Pages visited in Chrome/Chromium/Firefox, captured by Hister's own extension and indexed by Hister (web pages only; its local-file indexing is never used) | Lexical full-text; optional Semantic via Hister's own embedding endpoint |
 
 For configuration syntax and connector details, see [docs/datasource-skills.md](docs/datasource-skills.md). Non-interactive `sync`/`index` steps get a one-hour per-connector budget (`connector.indexTimeoutMs`) because first-run imports routinely take minutes; interactive search keeps its 60-second default.
@@ -734,9 +763,9 @@ the whole librarian loop runs on the hosted account:
 
 `jev.backend: "dazzi"` posts Jev's native dialect to `<root>/v1/jev`
 (default model `jev-1.13`); `rerank.provider: "dazzi"` reranks against
-`<root>/v1/rerank` (default model `qwen/qwen3-reranker-8b`). Both use the stored
-`/login` credential, and an explicit `rerank.baseUrl` or `rerank.model` still overrides
-the defaults.
+`<root>/v1/rerank`. Both use the stored `/login` credential. Reranking is opt-in and
+has no default model: it runs only when `rerank.model` names one, and an explicit
+`rerank.baseUrl` overrides the endpoint.
 
 ---
 
@@ -750,7 +779,7 @@ Deep dive into AutoRAG Agent's architecture, security, and integration guides:
 - **[Telegram Remote Control](docs/telegram.md):** Ask the librarian from a Telegram chat through a running `autorag tui --telegram` session: setup, commands, and keeping it alive.
 - **[Datasource Skills Reference](docs/datasource-skills.md):** Full configuration contracts, connection aliases, and connector options.
 - **[Apple Calendar & Reminders via EventKit MCP](docs/eventkit-mcp.md):** macOS-only: read Calendar events and Reminders (Google/iCloud/Exchange accounts already in Calendar.app) by running `mcp-server-apple-events` beside `autorag-mcp`. Live, read-only; not indexed.
-- **[Jev Decisions](docs/jev-decisions.md):** Jev query pipeline, **on by default** via Dazzi Models when signed in (OpenRouter otherwise): it routes each question to local search (everything needing more information, including the public web when the question needs recent public facts), `extra` (a knowledge answer or any other action such as committing code; one turn with every tool, no retrieval), `organize` (move, rename, convert, or sort files and folders), or its own `config` branch, and keeps a weakly-favored non-local verdict on local search through a 0.75 confidence floor. It also splits multi-part questions into up to five parallel searches (`dazzi/openai/gpt-6-luna` when signed in, else `openrouter/qwen/qwen3.7-flash`), picks which registered datasources to search before the fast answer, and ends the run after the fast answer when that answer is complete. Also covers the `jev` judgment tool, backends (OpenRouter, TypeSafe, Vercel AI Gateway), and how to opt out.
+- **[Jev Decisions](docs/jev-decisions.md):** Jev, the **required** decision model, via Dazzi Models when signed in (OpenRouter otherwise), or any decision model in pi's catalog: `jev.backend` names the provider (`openrouter`, `openai`, `typesafe`, `vercel-ai-gateway`, `cloudflare-workers-ai`, `opencode`, or `dazzi`) and `jev.model` the model (e.g. `openai/gpt-6-luna-decisions` on OpenRouter, `gpt-6-luna` on OpenAI); `autorag models list --decision` lists them. The first question of every session checks that it answers; when it cannot, the session moves to an sLLM fallback (`jev.sllm.model`, the decomposition model by default, with a `decision-model-fallback` warning), and `jev.backend: "sllm"` makes that small chat LLM the decision model with no Jev call, and when that fails too it stops with `decision-model-unavailable` (exit 3 for `autorag search`). It routes each question to local search (everything needing more information, including the public web when the question needs recent public facts), `extra` (a knowledge answer or any other action such as committing code; one turn with every tool, no retrieval), `organize` (move, rename, convert, or sort files and folders), or its own `config` branch, and keeps a weakly-favored non-local verdict on local search through a 0.75 confidence floor. It also splits multi-part questions into up to five parallel searches (`dazzi/openai/gpt-6-luna` when signed in, else `openrouter/qwen/qwen3.7-flash`), picks which registered datasources to search before the fast answer, and ends the run after the fast answer when that answer is complete.
 - **[Manual QA & Datasource Test Harnesses](docs/manual-qa-datasources.md):** Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.
 - **[Supply Chain Security & License Audits](docs/supply-chain.md):** Software bill of materials (SBOM) and dependency gate policies.
 - **[Competitive Landscape Study](docs/competitive-landscape-2026-09.md):** In-depth analysis of why in-place federation outperforms centralized RAG.
@@ -782,7 +811,7 @@ AutoRAG Agent stands on the shoulders of fantastic open-source projects:
 - **[Hister](https://github.com/asciimoo/hister)** by [asciimoo](https://github.com/asciimoo) and contributors — Self-hosted browser web-history search (AGPL-3.0-or-later). Never bundled, linked, copied, or modified: users run Hister as a separate program and AutoRAG only calls its public HTTP search API (mere aggregation). Hister is always named as the backend of the `hister` datasource.
 - **[dupey](https://github.com/NomaDamas/dupey)** by [NomaDamas](https://github.com/NomaDamas) — Fast duplicate and near-duplicate document family detection.
 - **[Filee](https://github.com/KnifeLemon/Filee)** by [KnifeLemon](https://github.com/KnifeLemon) — Offline conversion of 240+ file formats (MIT). Never bundled or linked: AutoRAG downloads the SHA-256-pinned [`filee-cli`](https://github.com/NomaDamas/filee-cli) release (NomaDamas' CLI-only redistribution, MIT) into `<AUTORAG_HOME>/tools/filee-cli/` on `autorag setup` / first conversion and spawns it as a separate process. An existing `filee`/`filee-cli` on `PATH` or `filee.binaryPath` wins. Filee downloads its own engines (FFmpeg, Pandoc, LibreOffice, …) on first use.
-- **Federated CLI Authors:** External datasource tools [`lazykatok`](https://github.com/changeroa/lazykatok), [`discrawl`](https://github.com/openclaw/discrawl), [`mailcrawl`](https://github.com/NomaDamas/mailcrawl), [`wacrawl`](https://github.com/openclaw/wacrawl), [`telecrawl`](https://github.com/openclaw/telecrawl), [`slacrawl`](https://github.com/openclaw/slacrawl), [`notcrawl`](https://github.com/openclaw/notcrawl), and [`rclone`](https://rclone.org).
+- **Federated CLI Authors:** External datasource tools [`lazykatok`](https://github.com/changeroa/lazykatok), [`discrawl`](https://github.com/openclaw/discrawl), [`mailcrawl`](https://github.com/NomaDamas/mailcrawl), [`wacrawl`](https://github.com/openclaw/wacrawl), [`telecrawl`](https://github.com/openclaw/telecrawl), [`slacrawl`](https://github.com/openclaw/slacrawl), and [`notcrawl`](https://github.com/openclaw/notcrawl).
 - **[kordoc](https://github.com/chrisryugj/kordoc)** by [chrisryugj](https://github.com/chrisryugj) — HWP/HWPX/HWPML, PDF, DOCX and XLSX parsing with nested-table fidelity, used as AutoRAG's default document parser.
 
 ---
@@ -823,9 +852,11 @@ Common failures and their fix:
 | A datasource errors during refresh | `datasource-index-failed` | run that CLI's own `doctor` |
 | MinSync binary missing (search and refresh fail with `MinSync is required ...`) | — | `cargo install minsync`, or leave `minSync.autoInstall` on and re-run refresh |
 | Jikji missing | `jikji-unavailable` | check the Rust toolchain, re-run refresh |
+| Jikji older than 0.2.0 (the message names the version and the binary path) | `jikji-unsupported-version` | re-run `autorag refresh --method jikji` to install the pinned build, or `cargo install jikji-cli --version 0.2.1 --locked --force`; with `jikji.autoInstall: false` or `jikji.binaryPath`, upgrade it yourself |
 | Jikji prepare of a large root times out (`the Jikji prepare run timed out`) | `jikji-prepare-failed` | the refresh still finishes and the other roots are indexed. There is no fixed total wall: the per-root hang guard is `jikji.timeoutMs` (60 s base) plus `jikji.prepareTimeoutMsPerFile` ms per file under the root (default: `jikji.parseTimeout`, 5 s), uncapped, so a huge root is not cut by its size and only a run stuck beyond every file hitting its parse limit is killed. Set `prepareTimeoutMsPerFile: 0` for a flat `timeoutMs` wall. `jikji.parseTimeout` (seconds) is the per-file limit, forwarded to `jikji prepare --parse-timeout`; jikji skips a file that exceeds it. A timeout kills jikji's whole process group |
+| `jikji_find` miss: the `discover` retry step fails (`Jikji discover retry failed: ... (exit status N): <stderr>`) | `jikji-discover-failed` | a miss makes AutoRAG run the `jikji discover ... --after-jikji-retry --retry-proof <proof>` command jikji named (the proof is bound to that root and rewritten query; `--no-background-refresh --no-auto-prepare` keep it from reindexing or rewriting `AGENTS.md`/`CLAUDE.md`/`.cursorrules`). The message carries the CLI exit status and stderr verbatim, real paths included; the failed root is dropped and healthy roots still answer. A `next_commands` line AutoRAG cannot parse as a plain `discover` step is quoted in the same diagnostic and is never executed |
 | Windows file-name search fails during refresh | `everything-index-failed` | read the ES exit code and stderr in the message, then `autorag refresh --method everything --json` |
-| Every search does a full local search and verification, even for small talk | `query-route-fallback` | sign in to Dazzi Models with `/login`, or set `OPENROUTER_API_KEY` (Jev routing and decomposition are on by default) |
+| A search stops before answering with `[decision-model-unavailable] ... (missing-credential)` | `decision-model-unavailable` (exit 3) | sign in to Dazzi Models with `/login`, or set `OPENROUTER_API_KEY`; the error's `reason` names what failed (Jev is required) |
 
 Native datasource stores stay owned by their CLIs — AutoRAG never rebuilds them. Fix a broken archive with `lazykatok doctor`, `discrawl --json metadata`, `slacrawl --json doctor`, `wacrawl --json doctor`, `telecrawl --json doctor`, `notcrawl doctor`, or `mailcrawl doctor`, then re-run `autorag refresh --method datasources --json`.
 

package/dist/agent/agent.d.ts

@@ -6,6 +6,7 @@ import type { DatasourceIndexResult, DatasourceSkill } from "../datasource/types
 import { type DupeyCliOptions } from "../dupey/index.js";
 import { type Embedder, type ManagedEmbedder } from "../embedding/index.js";
 import { type EverythingClientOptions, type EverythingSearchRequest, type EverythingSearchResult } from "../everything/index.js";
+import { type CloudFileProbe } from "../filesystem/cloud-placeholder.js";
 import { type FSearchClientOptions, type FSearchFailureReason, type FSearchSearchRequest, type FSearchSearchResult } from "../fsearch/index.js";
 import { type JikjiFailureReason, type JikjiOptions, type JikjiPrepareOptions } from "../jikji/index.js";
 import { type LanguageTag } from "../language.js";
@@ -48,6 +49,22 @@ export interface AutoRAGRefreshLimits {
     readonly maxDepth?: number;
     /** Retries for a transient per-file I/O error (EMFILE/EBUSY/EIO/ETIMEDOUT) before the file is skipped. Default 3. */
     readonly transientRetries?: number;
+    /**
+     * Most cloud-file placeholders (files whose data is not on disk) one refresh may download.
+     * Reading a placeholder downloads it, so this bounds disk and network use per refresh; the rest wait
+     * for the next one. Default 200.
+     */
+    readonly cloudMaxFiles?: number;
+    /** Most placeholder bytes one refresh may download; the first file is always allowed. Default 536870912 (512 MiB). */
+    readonly cloudMaxBytes?: number;
+    /** Release a downloaded cloud file again once it is parsed and indexed. Default true. */
+    readonly cloudEvictAfterParse?: boolean;
+    /**
+     * Retries after a failed release of a downloaded cloud file (`evictUbiquitousItem` can be refused while the
+     * provider still holds the file), waiting 500 ms then 1 s then 2 s. After the last retry the file is left on
+     * disk and `cloud-evict-failed` is reported. `0` tries once. Default 3.
+     */
+    readonly cloudEvictRetries?: number;
 }
 /**
  * Hard caps on retrieval, baseline prefetch, and the candidate lists handed to
@@ -66,13 +83,18 @@ export interface AutoRAGRetrievalLimits {
     readonly minSyncScopedQueryTopK?: number;
     /** Instance scopes listed in one datasource tool description. Default 8. */
     readonly toolDescriptionInstanceScopes?: number;
-    /** Baseline evidence prefetched for the fast answer. */
+    /**
+     * Baseline evidence prefetched for the fast answer. Each field's default depends on reranking: with a named rerank
+     * model the pool is wide (the reranker picks the best); without one every source contributes its own top 10.
+     */
     readonly prefetch?: {
-        /** Jikji find candidate count. Default 30. */
+        /** Jikji find candidate count. Default 30 with a reranker, else 10. */
         readonly jikjiTopK?: number;
-        /** MinSync retrieve candidate count. Default 100. */
+        /** MinSync retrieve candidate count. Default 100 with a reranker, else 10. */
         readonly minSyncTopK?: number;
-        /** Max Jikji answer paths rendered into the baseline. Default 100. */
+        /** Hits requested from each selected datasource. Default `singleDatasourceTopK` (20) with a reranker, else 10. */
+        readonly datasourceTopK?: number;
+        /** Max Jikji answer paths handed to the verification prompt as starting points. Default 100. */
         readonly jikjiPathLimit?: number;
         /** Max results rendered per baseline section. Default 100. */
         readonly sectionLimit?: number;
@@ -255,15 +277,20 @@ export interface AutoRAGAgentOptions {
         fetch?: WebFetchToolOptions | false;
     }) | false;
     /**
-     * Post-merge reranking. When set (and not `false`), merged retrieval evidence
-     * is reordered by a dedicated rerank model — OpenRouter by default. A
-     * configured-but-unavailable reranker is reported as a diagnostic and the
-     * merged order is preserved. `false` disables reranking.
+     * Post-merge reranking, opt-in: it runs only when a rerank `model` is named. A block without a model is treated as
+     * off and reported once as a `rerank-disabled` diagnostic; absent or `false` is off. With reranking off the fast
+     * answer's baseline keeps each source's own top 10 (see {@link AutoRAGRetrievalLimits.prefetch}). A configured
+     * but unavailable reranker is reported as a diagnostic and the merged order is preserved.
      */
     rerank?: RerankAgentOptions | false;
     autoRefresh?: AutoRefreshOptions;
     parserOptions?: DefaultParserRegistryOptions;
     dupey?: DupeyCliOptions | false;
+    /**
+     * Cloud placeholder probe. Omitted, the platform probe is used (macOS and Windows). A programmatic seam for
+     * hosts and tests that need to stand in for this machine's cloud and network roots.
+     */
+    cloudProbe?: CloudFileProbe;
     /**
      * Filee command line behind `convert_files` and `list_file_conversions`.
      * `false` leaves both tools out. The duplicate-family check on converted
@@ -271,18 +298,20 @@ export interface AutoRAGAgentOptions {
      */
     filee?: FileeCliOptions | false;
     /**
-     * Jev (TypeSafe's judgment model: typed questions in, calibrated
-     * probabilities out). When set, the two-phase search asks Jev before the
-     * fast answer which branch handles the question (local search, file
-     * organization, an extra task, or — in a local host — its own configuration)
-     * and whether to decompose it; after the fast answer, whether
-     * verification is needed. It also exposes the `jev` tool.
+     * Jev, the decision model (typed questions in, calibrated probabilities
+     * out). It is required: before the fast answer the two-phase search asks
+     * it which branch handles the question (local search, file organization,
+     * an extra task, or — in a local host — its own configuration) and whether
+     * to decompose it; after the fast answer, whether verification is needed.
+     * It also judges citations and evidence, drives the `auto` permission
+     * gate, and backs the `jev` tool.
      *
-     * The CLI config enables this by default on OpenRouter (`buildAgentOptions`
-     * fills it in). On this programmatic option, absent or `false` keeps Jev
-     * off, so library callers never make paid network calls they did not ask for.
+     * The first question of every session probes it and fails with
+     * `DecisionModelUnavailableError` (code `decision-model-unavailable`) when
+     * it cannot answer. Omitted, the backend is resolved from the environment
+     * (the first credential present); `false` is rejected.
      */
-    jev?: JevToolOptions | false;
+    jev?: JevToolOptions;
     /**
      * The embedder shared by memory and the datasources. Omitted, it is built from `minSync.embedder`: a
      * configured remote endpoint, else MinSync's `minsync serve` daemon, else MinSync's in-process engine.
@@ -296,11 +325,10 @@ export interface AutoRAGAgentOptions {
      */
     memoryEmbedder?: Embedder | false;
     /**
-     * Agent self-configuration. When set (and Jev is on), Jev gains a `config`
-     * branch for questions about AutoRAG's own settings: the turn skips the
-     * fast answer, receives the full `autorag-setup` skill, edits
-     * `configPath` with the pi tools, and reports what it changed as its final
-     * message.
+     * Agent self-configuration. When set, Jev gains a `config` branch for
+     * questions about AutoRAG's own settings: the turn skips the fast answer,
+     * receives the full `autorag-setup` skill, edits `configPath` with the pi
+     * tools, and reports what it changed as its final message.
      */
     selfConfig?: SelfConfigOptions;
     /**
@@ -393,7 +421,7 @@ export interface AutoRAGAgentOptions {
 export interface RerankAgentOptions {
     /** Provider id. @default "openrouter" */
     provider?: string;
-    /** Wire model id. @default "voyageai/rerank-3-lite" */
+    /** Wire model id. Required: reranking runs only when a model is named; there is no default. */
     model?: string;
     /** API key. Prefer `apiKeyEnv`; this is a programmatic seam. Not used by the `dazzi` provider. */
     apiKey?: string;
@@ -497,14 +525,28 @@ export declare class AutoRAGAgent {
     private readonly parserOptions;
     private readonly dupeyOptions;
     private readonly fileeOptions;
-    /** pi extension registering the optional `jev` tool; undefined when disabled. */
+    /** Platform cloud probe; `undefined` where AutoRAG has no cloud handling. */
+    private readonly cloudProbe;
+    /** Resolved once: the machine's non-physical roots, or `[]` when they cannot be listed. */
+    private cloudRootsPromise;
+    /** pi extension registering the `jev` tool. */
     private readonly jevExtension;
-    /** Jev judge shared by the `jev` tool and the query router; undefined when disabled. */
+    /** pi extension releasing the cloud files the agent opened during a run. */
+    private readonly cloudRelease;
+    /** Jev and its sLLM fallback: the session check picks which one {@link jevJudge} asks. */
+    private readonly decisionModels;
+    /** Jev judge shared by the `jev` tool, the query router, the judgments, and the permission gate. */
     private readonly jevJudge;
+    /**
+     * Session whose first question already confirmed that the decision model answers: the bound pi
+     * session's id, or {@link UNBOUND_DECISION_SESSION} for runs without a TUI session. Undefined until a
+     * check passed; a failed check leaves it unchanged, so the next question checks again.
+     */
+    private decisionModelCheckedSession;
     /** Mode and grants of the file-change approval gate; shared by every pi session this agent creates. */
     private readonly permissions;
     private readonly permissionGate;
-    /** Self-configuration, enabled only with Jev on a local session. */
+    /** Self-configuration, enabled only on a local session. */
     private readonly selfConfig;
     /** Override for the packaged organize skill the `organize` branch loads; undefined keeps the packaged one. */
     private readonly organizeSkillPath;
@@ -532,6 +574,14 @@ export declare class AutoRAGAgent {
     private readonly memoryVectorPath;
     /** Every retrieval result of the in-flight run with the query/method that surfaced it. */
     private readonly evidenceOrigins;
+    /**
+     * Files Jikji found for the current search's queries, in Jikji's order. The baseline prefetch sets it;
+     * only the verification prompt uses it, as places to start exploring. The fast answer never sees them:
+     * a path says nothing about what a file contains, and the reranker cannot judge one.
+     */
+    private jikjiStartingPoints;
+    /** Settles once Jikji's find for every query of the current search is done. Never rejects. */
+    private jikjiDiscovery;
     /** Search-tool queries by tool call id, read back when the call ends. */
     private readonly searchToolQueries;
     /** Identifies this agent's conversation: everything judged in it is "current" memory. */
@@ -581,8 +631,8 @@ export declare class AutoRAGAgent {
      * After the final answer: map every cited evidence back to the search query
      * and method that found it, ask Jev (one batched call, evidence judged in
      * parallel) whether it really supports the answer sentence it backs, and
-     * keep only what clears the threshold. Runs without Jev
-     * store nothing; a failed judgment stores nothing and says why.
+     * keep only what clears the threshold. A failed judgment stores nothing and
+     * says why.
      */
     private recordJudgedEvidence;
     private resolveSessionModel;
@@ -615,7 +665,7 @@ export declare class AutoRAGAgent {
     private interruptibleBySteer;
     /**
      * Resolve the model's inline `[eN]` citations against this run's evidence and derive the structured results.
-     * With Jev on, every cited chunk is first checked for relevance to the sentence that cites it and to what
+     * Every cited chunk is first checked for relevance to the sentence that cites it and to what
      * the user typed in this conversation; a chunk Jev finds unrelated (a table of contents, a footer) is dropped
      * from the citations before the answer is returned or judged for memory.
      */
@@ -724,8 +774,8 @@ export declare class AutoRAGAgent {
     /**
      * Jev check run after the fast answer: does the answer need correction,
      * clarification, or further research? "No" ends the run with the fast
-     * answer as the final answer. Without Jev, or when the check fails, the run
-     * always continues into verification.
+     * answer as the final answer. When the check fails, the run continues into
+     * verification.
      */
     private shouldFollowUp;
     /**
@@ -735,8 +785,8 @@ export declare class AutoRAGAgent {
      * most five search queries with the configured decomposition model
      * (default: the session model). On the local branch, Jev also judges, per
      * registered datasource, whether it must be searched (in parallel with
-     * decomposition). Without Jev, and on any routing failure, this is today's
-     * single local search for the original question with no datasource search.
+     * decomposition). On any routing failure this is a single local search for
+     * the original question with no datasource search.
      * A non-local branch that misses the confidence floor is not a failure: it
      * becomes a full local plan, decomposition and datasource check included.
      */
@@ -753,11 +803,20 @@ export declare class AutoRAGAgent {
      */
     private selectSearchDatasources;
     /**
-     * Baseline local evidence for the fast answer. Jikji, MinSync, every
+     * Start Jikji's find for every search query without waiting for it. The fast answer never sees Jikji's
+     * files (a path says nothing about what a file contains), so nothing in the baseline depends on this; it
+     * runs alongside the fast answer and `searchDocuments` waits for it only before the follow-up check, to
+     * hand the files to the verification prompt. It stops with the run, not with a steer: a steer makes the
+     * run verify anyway, which is when the files are used. A failed find contributes no files.
+     */
+    private discoverJikjiFiles;
+    /**
+     * Baseline local evidence for the fast answer. MinSync, every
      * Jev-selected datasource, and (when Jev selected it) one web search run for
      * every search query in parallel (MinSync itself queues per workspace), the
      * per-query hits are interleaved and deduplicated into one pool, and that
-     * pool is reranked against the original question.
+     * pool is reranked against the original question. Jikji is started here too
+     * (see `discoverJikjiFiles`) but is not awaited and adds nothing to the pool.
      */
     private prefetchInitialRetrievalContext;
     /**
@@ -770,7 +829,7 @@ export declare class AutoRAGAgent {
      */
     private prefetchWebSearch;
     /**
-     * Rerank the pre-fast-answer baseline pool — Jikji answer paths, MinSync
+     * Rerank the pre-fast-answer baseline pool — MinSync
      * chunks, Jev-selected datasource chunks, and prefetched web hits — down to
      * the configured `rerank.topN`. With decomposition the pool merges every
      * search query's hits (interleaved), capped at the same per-source sizes as a
@@ -859,7 +918,7 @@ export declare class AutoRAGAgent {
      * against it; the model then verifies with thinking on and ends with the
      * final answer as a plain reply, returning only the delta.
      */
-    buildRefinementPrompt(query: string, options: RetrievalOptions, fastAnswer: string | undefined, fastAnswerDelivered: boolean, steers?: readonly string[]): string;
+    buildRefinementPrompt(query: string, options: RetrievalOptions, fastAnswer: string | undefined, fastAnswerDelivered: boolean, steers?: readonly string[], jikjiStartingPoints?: readonly string[]): string;
     buildSearchPrompt(query: string, options: RetrievalOptions, initialRetrievalContext?: string): string;
     refresh(force?: boolean, opts?: AutoRAGRefreshOptions): Promise<AutoRAGRefreshResult>;
     /**
@@ -901,6 +960,8 @@ export declare class AutoRAGAgent {
      * so the refresh result and status remain consistent.
      */
     private scanMirrorStaleness;
+    /** The machine's non-physical roots, resolved once; a probe failure reads as no roots, never an error. */
+    private cloudRoots;
     private exactDuplicateExclusions;
     syncMinSync(force?: boolean, signal?: AbortSignal, onProgress?: (progress: MinSyncSyncProgress) => void): Promise<MinSyncSyncResult>;
     /** Provider for the Windows-only everything_search tool. */
@@ -947,6 +1008,15 @@ export declare class AutoRAGAgent {
         readonly first?: boolean;
         readonly signal?: AbortSignal;
     }): Promise<JikjiFindProviderResult>;
+    /**
+     * One root's find, followed by jikji's recovery step. When `find` answers
+     * `jikji_retry`, run the `discover` command it named with the proof it
+     * issued for this root and rewritten query (re-running `find` with the proof
+     * does not advance jikji, only `discover` does). One step per find: whatever
+     * discover answers is final for the root, so the protocol cannot loop. A
+     * failed discover drops the root and reports the CLI failure verbatim.
+     */
+    private findJikjiRoot;
     /**
      * Merge per-root answer packs into one. Concatenates answer_paths/candidates
      * preserving per-root order; dedupes by normalized path. Does NOT cross-root
@@ -968,7 +1038,8 @@ export declare class AutoRAGAgent {
      * - stopAfterFind: OR
      * - agentShouldNotRerank: OR
      * - handoffAction: MOST RESTRICTIVE (direct_use < jikji_retry < raw_fallback_after_retry)
-     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry
+     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry and every root's raw_fallback_allowed
+     * - maxRawFallbackCommands: MIN over the roots that report one
      */
     private mergePolicy;
     /**

package/dist/agent/cloud-release-extension.d.ts

@@ -0,0 +1,43 @@
+import type { ExtensionFactory } from "@earendil-works/pi-coding-agent";
+import { type CloudFileProbe, type CloudRoot } from "../filesystem/cloud-placeholder.js";
+/** Direct child files one directory candidate contributes; beyond this a directory is left unwatched. */
+export declare const DEFAULT_CLOUD_RELEASE_CHILD_CAP = 500;
+/**
+ * Every absolute path in a tool input value: a string that is itself absolute (a `path` argument), and the
+ * absolute words of any string that is not (a `bash` command, where `cd "<dir>" && pdftotext "a.pdf" -` names
+ * the directory `/…/dir` and no absolute file). Object and array values are walked, so a tool that nests its
+ * arguments is covered too.
+ */
+export declare function absolutePathsIn(value: unknown): string[];
+export interface CloudReleaseExtensionOptions {
+    /** Platform cloud probe; without one the extension registers nothing (a platform AutoRAG cannot release on). */
+    readonly probe?: CloudFileProbe;
+    /**
+     * The machine's non-physical roots, resolved lazily and cached by the caller. Without one, or when it
+     * resolves to no releasable root (Linux, or a machine with no cloud folders), the extension does nothing.
+     */
+    readonly cloudRoots?: () => Promise<readonly CloudRoot[]>;
+    /** Direct child files one directory candidate contributes. Default {@link DEFAULT_CLOUD_RELEASE_CHILD_CAP}. */
+    readonly childCap?: number;
+    /** Retries after the first failed release; `0` tries once. Default {@link DEFAULT_CLOUD_EVICT_RETRIES}. */
+    readonly evictRetries?: number;
+    /** First retry wait in ms, doubled each retry. Default {@link DEFAULT_CLOUD_EVICT_BACKOFF_MS}. */
+    readonly evictBackoffMs?: number;
+    /** Wait implementation; tests inject one that records instead of sleeping. */
+    readonly sleep?: (ms: number) => Promise<void>;
+    /** One line per file that could not be released; defaults to stderr. Never thrown, never shown in the UI. */
+    readonly log?: (message: string) => void;
+}
+/**
+ * pi extension that releases cloud files the agent itself opened during an answer.
+ *
+ * A refresh releases the placeholders it downloads, but the agent's own tools (`bash`, `read`, `grep`, …) open
+ * originals too, and nothing released those. This extension watches the files a run named or was shown: before
+ * each call it stats the candidates a tool input names (an absolute `path`, the absolute words of a `bash`
+ * command, the direct children of a directory candidate) and remembers the ones that are placeholders now;
+ * after each result it does the same for the absolute paths a retrieval result printed. At `agent_end` it
+ * re-stats what it remembered and releases each file that is now on disk. A file that was already local when
+ * first seen is never watched, so only what this run hydrated is released. Without a probe or without a
+ * releasable cloud root the extension does nothing; a failed release is logged once and never thrown.
+ */
+export declare function createCloudReleaseExtension(options: CloudReleaseExtensionOptions): ExtensionFactory;

package/dist/agent/context-token-guard.d.ts

@@ -20,15 +20,20 @@ export declare function createContextTokenGuardedTransform(model: ContextTokenGu
 /** The model with its output limit held to {@link MAX_OUTPUT_TOKENS}; the same model when already within it, or when absent. */
 export declare function withBoundedOutput<T extends Model<any> | undefined>(model: T): T;
 /**
- * Keep complete retrieval candidates while the assembled request fits the
- * model budget. Removing a per-chunk cut is only safe because this drops whole
- * lower-ranked candidates (with an explicit marker) instead of hiding text.
+ * Keep the request inside the model budget by dropping retrieval chunks only: the
+ * lowest-ranked ones go first, across every ranked list in the transcript (every
+ * list's best chunk stays before any list's second), each with an explicit marker.
+ * The user's query, earlier turns, and the system prompt are never edited. When no
+ * chunk is left to drop, only the newest tool result is shortened; anything still
+ * too large is left for the provider to reject and pi's compaction to recover.
  */
 export declare function guardAgentContextMessages(model: Model<any> | undefined, messages: AgentMessage[], useReportedUsage?: boolean, fixedOverheadTokens?: number): AgentMessage[];
 /**
- * Estimate the current transcript with pi-coding-agent's conservative chars/4
- * counter. `useReportedUsage` adds the last assistant turn's provider-reported
- * usage for the settled prefix; pass false when the messages were just edited.
+ * Estimate the current transcript: pi-coding-agent's chars/4 counter as the
+ * floor, raised by a real BPE count for text that chars/4 undercounts (see
+ * `token-count.ts`). `useReportedUsage` adds the last assistant turn's
+ * provider-reported usage for the settled prefix; pass false when the messages
+ * were just edited.
  */
 export declare function estimateContextMessageTokens(messages: readonly AgentMessage[], useReportedUsage?: boolean): number;
 export {};