@autorag/librarian 2.3.0 → 2.4.1

package/README.md

@@ -16,6 +16,25 @@ AutoRAG itself is the specialized search agent, not a coordinator for other
 model roles. You configure one model, and that model owns the complete
 retrieval, reading, judgment, and curation loop.
 
+## Core values
+
+Three principles drive every design decision:
+
+1. **Never migrate your data to search it.** AutoRAG federates CLI-owned
+   stores (katok, discrawl, qmd, msgvault, rclone, …) in place. No forced
+   ingestion into a central index, no third-party server holding a copy of
+   your corpus. Results carry source-native identities
+   (`kakao:<chat>/<sender>/<chunk>`) with scope-checked access — see the
+   [competitive landscape study](docs/competitive-landscape-2026-09.md) for
+   why this is the durable differentiator.
+2. **Just works — no RAG degree required.** Install it and it works: minimal
+   configuration, no pipeline tuning, no vector-DB operations, no API keys.
+   The local embedder (EmbeddingGemma via Ollama) and MinSync auto-install
+   handle the "RAG plumbing" invisibly.
+3. **Fast by design.** One configured model owns the whole loop; retrieval
+   runs locally over MinSync CDC chunks (BM25 / vector / hybrid); interactive
+   search is optimized for low latency.
+
 ## Why AutoRAG
 
 ### The problem with search tools
@@ -65,26 +84,62 @@ Different documents need different search strategies:
 | Legal documents, specifications | BM25 (keyword ranking) | Handles domain terminology well |
 | Mixed collections | Hybrid (vector + BM25) | Combines precision and recall |
 
-AutoRAG supports **pluggable retrieval methods**. It ships with lexical **BM25** and semantic **MinSync** methods wired through the `RetrievalMethodRegistry`, and the architecture is ready for additional vector and hybrid backends. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication.
+AutoRAG supports **pluggable retrieval methods**. Local lexical BM25, semantic vector, and hybrid retrieval all go through **MinSync** over one shared CDC chunk lifecycle, wired through the `RetrievalMethodRegistry`. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication. External datasources keep their own archive/index lifecycle.
+
+BM25, vector, and hybrid are **enabled by default** whenever MinSync is enabled. Disable local indexing with `"minSync": false`, or disable only lexical search with `"bm25": false`. MinSync auto-installs a verified release into `<workspace>/.autorag/bin` on first use (`autoInstall: true`); set `"autoInstall": false` only when managing the binary yourself. Configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints, and set `minSync.maxChunkSize` (or `--minsync-max-chunk-size`) when a local embedder has a smaller context window. AutoRAG never forces TEI or any external embedding service.
 
-BM25 and MinSync are **enabled by default** — no explicit configuration is needed for standard lexical + semantic retrieval. Both can be disabled by setting `"bm25": false` or `"minSync": false` in the config file. MinSync uses a pre-installed binary (`autoInstall: false`); configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints. AutoRAG never forces TEI or any external embedding service.
+See [docs/minsync-setup.md](docs/minsync-setup.md) for automatic installation,
+managed binary paths, and the local EmbeddingGemma QA flow.
 
 ### Real directory access
 
-AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. BM25 and MinSync index parsed markdown mirrors under `.autorag`.
+AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. MinSync indexes parsed markdown mirrors under `.autorag` for BM25, vector, and hybrid retrieval.
+
+### Thin PDF extraction retry
+
+The default PDF parser performs a cheap quality check for multi-page PDFs.
+When local markdown is unusually sparse (fewer than 800 characters or fewer
+than 40 characters per detected page, for at least three pages), it retries
+through OpenDataLoader's `docling-fast` hybrid backend with `hybridMode:
+"auto"` and a 30-second timeout. Dense PDFs are not retried, and hybrid is
+never used for single-page PDFs or as the first path for images.
+
+The gate is parser-owned and can be tuned through trusted programmatic
+`parserOptions`:
+
+```typescript
+new AutoRAGAgent({
+  searchPaths: ["/path/to/documents"],
+  parserOptions: {
+    thinExtract: {
+      minPages: 3,
+      minChars: 800,
+      minCharsPerPage: 40,
+      timeoutMs: 30_000,
+      hybrid: "docling-fast",
+      hybridMode: "auto",
+    },
+  },
+});
+```
+
+If the hybrid sidecar is missing, times out, or fails, AutoRAG keeps the local
+markdown and emits `pdf-extract-thin` plus `pdf-hybrid-unavailable`
+diagnostics; refresh remains successful.
 
-### Optional Jikji discovery and indexing
+### Default Jikji discovery and indexing
 
-AutoRAG can opt into [Jikji](https://github.com/NomaDamas/jikji) as a local CLI-backed **find-first discovery and indexing** layer. Jikji is optional: AutoRAG does not vendor it, install it, or register it as a retrieval backend when enabled.
+AutoRAG uses [Jikji](https://github.com/NomaDamas/jikji) by default as a local CLI-backed **find-first discovery and indexing** layer. Jikji is not registered as a retrieval backend: it supplies bounded discovery answer packs while the librarian still reads original files directly.
 
-When Jikji is configured, AutoRAG calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly.
+The default agent calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly. If the binary or Rust toolchain is unavailable, Jikji reports a diagnostic and the normal filesystem/retrieval paths continue.
+
+New `autorag init` configs include `"jikji": {}`. To opt out, set `"jikji": false` in `config.json`; programmatic callers can pass `jikji: false`.
 
 Programmatic use:
 
 ```typescript
 const agent = new AutoRAGAgent({
   searchPaths: ["/path/to/documents"],
-  jikji: { binaryPath: "jikji" },
 });
 await agent.prepareJikji();
 ```
@@ -118,7 +173,7 @@ searchable:
 If `dupey` is not installed or fails, refresh continues without exclusion and
 reports no destructive action; install it with `cargo install dupey`.
 
-The same `.autorag/jikji.json` shape configures Jikji when present:
+The same configuration shape customizes Jikji when needed:
 
 ```json
 {
@@ -153,7 +208,7 @@ channel or group chat as its own datasource.
 
 Security defaults are intentionally strict:
 
-- datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags` and `datasourceAccess.allowedScopes`;
+- datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags`; `allowedScopes` applies only to datasource methods that advertise the `scoped` capability;
 - model/tool arguments never grant datasource tags or scopes;
 - `search_datasource_documents` accepts only `{ query, topK?, scope? }`, and `scope` can only narrow trusted access.
 
@@ -169,8 +224,9 @@ Security defaults are intentionally strict:
 | Notion | `notion` | external [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | local-first page/database/block archive + FTS5 search |
 | GitHub Issues/PRs | `github` | GitHub REST (token optional) | issues + PR bodies per `owner/repo`; public repos work unauthenticated |
 | Cloud drives | `cloud-drive` | **[`rclone`](https://rclone.org) CLI** | Incremental Google Drive Tier-1; OneDrive/network remotes; iCloud experimental |
-| Gmail / IMAP | `gmail` | Gmail REST v1, or **[`himalaya`](https://pimalaya.org) CLI** (`backend: "himalaya"`) | the himalaya backend indexes any IMAP/Maildir account it has configured — no OAuth plumbing |
+| Gmail | `gmail` | Gmail REST v1 | OAuth access token is referenced by environment variable name |
 | Local mail exports | `mail-export` | filesystem (`.mbox` / `.eml`) | classic `From_` splitting, mailparser-based; count-only warnings |
+| Mail archives | `mailcrawl` | external [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | local email sync plus BM25, semantic, and hybrid retrieval |
 | Obsidian vault | `obsidian` | external [`qmd`](https://github.com/tobi/qmd) CLI | incremental `qmd update`, BM25 `qmd search`, semantic `qmd vsearch`; vault path via `connector.vaultPath` |
 | RSS / news | `rss` | HTTP feed polling | RSS 2.0 + Atom, feed/category hierarchy, 24h dedupe window |
 
@@ -186,15 +242,16 @@ Configure them in `config.json` (CLI) or pass `datasourceSkills` programmaticall
     "slack":    { "connector": { "binaryPath": "slacrawl", "configPath": "/path/to/slacrawl.yaml", "syncSource": "primary" } },
     "notion":   { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } },
     "github":   { "connector": { "repos": ["owner/repo"] } },
-    "gmail":    { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
+    "gmail":    { "connector": { "tokenEnv": "GMAIL_ACCESS_TOKEN", "labelIds": ["INBOX"] } },
+    "mailcrawl": { "instanceId": "personal", "connector": { "account": "personal", "mailbox": "INBOX", "binaryPath": "mailcrawl" } },
     "personal-google-drive": { "type": "cloud-drive", "instanceId": "personal", "connector": { "provider": "google-drive", "remote": "personal-gdrive:", "include": ["**/*.md"] } },
     "company-onedrive": { "type": "cloud-drive", "instanceId": "work", "connector": { "provider": "onedrive", "remote": "company-onedrive:Documents" } },
     "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
     "rss":      { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
   },
   "datasourceAccess": {
-    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "cloud-drive", "obsidian", "rss"],
-    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
+    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "mailcrawl", "cloud-drive", "obsidian", "rss"],
+    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/mailcrawl/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
   }
 }
 ```
@@ -207,6 +264,18 @@ Install slacrawl with `brew install openclaw/tap/slacrawl`. AutoRAG invokes `sla
 
 Install notcrawl with `brew install openclaw/tap/notcrawl`. AutoRAG invokes `notcrawl sync` during datasource refresh and `notcrawl search --json` during retrieval. Optional trusted connector fields are `binaryPath` and `configPath`. Notion credentials and workspace definitions remain in notcrawl's own configuration rather than AutoRAG.
 
+Install and configure [`mailcrawl`](https://github.com/NomaDamas/mailcrawl)
+`@nomadamas/mailcrawl@0.1.4` or newer separately. AutoRAG invokes
+`mailcrawl sync` followed by `mailcrawl index` during datasource refresh, then
+calls `mailcrawl search` in BM25, semantic, or hybrid mode. The archive remains
+in mailcrawl's native store unless the operator explicitly sets
+`connector.dataDir`; Himalaya credentials and provider configuration remain
+owned by mailcrawl. 0.1.3 and earlier fail a repeated
+`index` after a no-op sync.
+Use `mailcrawl` for all Himalaya-backed IMAP/Maildir retrieval. The retired
+`gmail` connector option `backend: "himalaya"` is no longer registered; migrate
+that configuration to an explicit `mailcrawl` datasource.
+
 Install and authenticate rclone separately (`brew install rclone && rclone
 config` on macOS), then configure the provider-neutral `cloud-drive` skill.
 `cloud-drive` is a reusable `type`: each datasource config key is a connection
@@ -240,7 +309,6 @@ const agent = new AutoRAGAgent({
   datasourceSkills: [kakao],
   datasourceAccess: {
     allowedTags: ["kakaotalk"],
-    allowedScopes: ["/kakao/personal/**"],
   },
 });
 
@@ -271,7 +339,6 @@ const agent = new AutoRAGAgent({
   datasourceSkills: [discord],
   datasourceAccess: {
     allowedTags: ["discord"],
-    allowedScopes: ["/discord/community/**"],
   },
 });
 ```
@@ -297,7 +364,7 @@ Or through the trusted config factory:
 Two defaults are deliberate and worth keeping:
 
 - **`defaultMode: "hybrid"`** — discrawl's FTS index strips newlines without substituting a space, welding words across line breaks into a single unsearchable token (measured at ~47% of post-newline words on a real archive). Semantic recall covers that gap. See [#1413](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1413).
-- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). For workspace-managed discrawl state, AutoRAG writes these values to `.autorag/datasources/discrawl/config.toml` while preserving unrelated discrawl settings. An explicit `connector.configPath` remains operator-owned and is never rewritten. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
+- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). Configure these values in discrawl's own config; AutoRAG uses discrawl's native store unless an explicit `connector.configPath` is supplied. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources can share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
 
 A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as `/kakao/personal/chunks/<chunk-id>`, and permission tags that match your server-side access policy.
 
@@ -349,6 +416,30 @@ Config path precedence is `--config` > `AUTORAG_CONFIG` > `~/.autorag/config.jso
 
 `autorag health` checks model/provider auth before a search — it resolves the model, verifies credential presence, and optionally probes one completion call. Use it to diagnose model, provider, auth, or timeout failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness). When `autorag search` fails for a model/provider reason, the error output includes a hint pointing to `autorag health`.
 
+`autorag ui` opens a loopback-only page (`127.0.0.1`) to connect local folders and datasource skills without editing JSON. It writes the same trusted `datasources` / `datasourceAccess` fields as a hand-edited config, stores env-var *names* rather than secrets, and refuses non-loopback binds. Use `--no-open` to print the URL without launching a browser.
+
+For a deliberately deployed UI, opt in explicitly in `config.json`. Keep the
+session token in the environment, set the public URL used by the browser, and
+allow only the frontend origins that should make credentialed API requests:
+
+```json
+{
+  "ui": {
+    "host": "0.0.0.0",
+    "port": 8787,
+    "allowRemote": true,
+    "publicOrigin": "https://autorag.example.com",
+    "corsOrigins": ["https://autorag.example.com"],
+    "tokenEnv": "AUTORAG_UI_TOKEN"
+  }
+}
+```
+
+Start it with `AUTORAG_UI_TOKEN` set to a random value of at least 16
+characters. Local use remains the safe default: omit `ui` (or leave
+`allowRemote` false) and AutoRAG binds to loopback, including a working
+`localhost` URL on systems that resolve it to IPv6.
+
 ## Installation
 
 Published as `@autorag/librarian` (dist bundled with Bun, runtime Node ≥ 24 or Bun):
@@ -360,9 +451,45 @@ bun install -g @autorag/librarian   # autorag CLI
 bun add github:NomaDamas/AutoRAG-2.0
 ```
 
+PDF parsing requires **Java 11 or newer** because the bundled
+`@opendataloader/pdf` runtime is compiled for Java 11. This applies to
+Windows, macOS, and Linux. Verify the runtime that will be selected from
+`PATH` before indexing PDFs:
+
+```bash
+java -version
+```
+
+On Windows, ensure a Java 11+ installation appears before older Java
+installations in `PATH` (or set `JAVA_HOME` to the Java 11+ installation and
+start a new shell). Java 8 is not supported by the PDF parser.
+
 Git-based installs build `dist/` via the `prepare` script and require Bun on the installing machine.
 External tool binaries auto-install on first use into `<workspace>/.autorag/bin`: **MinSync** downloads a verified GitHub release asset (on by default; `minSync.autoInstall: false` to opt out), and **Jikji** compiles the [`jikji-cli`](https://crates.io/crates/jikji-cli) crate via cargo (requires the [Rust toolchain](https://rustup.rs); `jikji.autoInstall: false` to opt out). New `autorag init` configs enable Jikji find-first discovery by default. KakaoTalk (`katok`) and Discord (`discrawl`) stay manual, optional installs. All of them degrade gracefully when missing — core BM25 search works without any of them.
 
+### Experimental TUI (beta)
+
+The latest TUI work is currently available on the `feat/experimental-tui`
+branch. To use this beta version from source:
+
+```bash
+git clone --branch feat/experimental-tui https://github.com/Marker-Inc-Korea/AutoRAG.git
+cd AutoRAG
+bun install
+bun run build
+
+# Configure your document roots and model once:
+node dist/cli/index.js init --search-paths /path/to/documents
+
+# Start the beta TUI:
+node dist/cli/index.js tui
+```
+
+Inside the TUI, use `/resume` to browse saved sessions. Selecting a session
+replaces the current view with that session instead of mixing the two
+conversation histories. This branch is experimental and may change before the
+feature is included in a published release.
+
 ## Quick Start
 
 ```typescript

package/dist/agent/agent.d.ts

@@ -8,19 +8,18 @@ import type { ResultFeedback } from "../memory/memory.js";
 import { type MinSyncSyncResult, type MinSyncVectorMethodOptions } from "../minsync/index.js";
 import { type ParsedMirrorSyncResult } from "../mirror/sync.js";
 import type { DefaultParserRegistryOptions } from "../parser/index.js";
-import { type BM25MethodOptions, type BM25SyncResult } from "../retrieval/methods/bm25.js";
 import { RetrievalMethodRegistry } from "../retrieval/registry.js";
 import type { CuratedResult, RetrievalDiagnostic, RetrievalOptions, RetrievalResult } from "../retrieval/types.js";
 import { type ScanDuplicateDocumentsDetails } from "./dupey-tool.js";
 import { type JikjiFindProviderResult } from "./jikji-find-tool.js";
-import { type SearchDocumentDiagnostic, type SearchDocumentsResponse } from "./search-documents.js";
+import { type SearchDocumentDiagnostic, type SearchDocumentsResponse, type SearchDocumentsStreamEvent } from "./search-documents.js";
 import { type WatcherFactory, type WatchRefreshHandle } from "./watch-refresh.js";
 export interface AutoRefreshOptions {
     readonly intervalMs: number;
     readonly immediate?: boolean;
 }
 /** Methods that `refresh` can selectively run. Defaults to all when omitted. */
-export type RefreshMethod = "parsed" | "bm25" | "minsync" | "datasources" | "jikji";
+export type RefreshMethod = "parsed" | "minsync" | "datasources" | "jikji";
 export interface AutoRAGRefreshOptions {
     /** Restrict refresh to specific methods. Defaults to all when undefined. */
     readonly methods?: readonly RefreshMethod[];
@@ -30,13 +29,12 @@ export interface AutoRAGMinSyncRefreshResult {
     readonly synced: number;
     readonly reason?: string;
 }
-export interface AutoRAGRefreshResult extends ParsedMirrorSyncResult {
-    readonly bm25?: BM25SyncResult;
+export interface AutoRAGRefreshResult extends Omit<ParsedMirrorSyncResult, "diagnostics"> {
+    readonly diagnostics: readonly SearchDocumentDiagnostic[];
     readonly minsync?: AutoRAGMinSyncRefreshResult;
     readonly datasources?: readonly DatasourceIndexResult[];
 }
 export interface AutoRAGRefreshComponentStatus {
-    readonly bm25?: string;
     readonly minsync?: string;
     readonly jikji?: string;
     readonly datasources?: string;
@@ -77,14 +75,19 @@ export interface AutoRAGAgentOptions {
     workspacePath?: string;
     tools?: AgentTool[];
     minSync?: Omit<MinSyncVectorMethodOptions, "root"> | false;
-    bm25?: Omit<BM25MethodOptions, "root"> | false;
-    jikji?: JikjiOptions;
+    jikji?: JikjiOptions | false;
     autoRefresh?: AutoRefreshOptions;
     parserOptions?: DefaultParserRegistryOptions;
     dupey?: DupeyCliOptions | false;
     excludeExactDuplicates?: boolean;
     datasourceSkills?: readonly DatasourceSkill[];
     datasourceAccess?: DatasourceAccessContextOptions;
+    /** Non-fatal diagnostics from config/agent construction (e.g. skipped unknown datasources). */
+    startupDiagnostics?: readonly SearchDocumentDiagnostic[];
+    /** Maximum time a model/tool search may run before it is aborted. */
+    searchTimeoutMs?: number;
+    /** Maximum number of retrieval/tool executions allowed in one search. */
+    maxSearchToolCalls?: number;
 }
 export interface AutoRAGSearchSession {
     readonly agent: Agent;
@@ -119,6 +122,10 @@ export declare class AutoRAGAgent {
     private resultCapture;
     private autoRefreshTimer;
     private refreshing;
+    private jikjiPrepareInFlight;
+    private jikjiReady;
+    private minSyncPrepareInFlight;
+    private minSyncReady;
     private refreshState;
     private readonly searchPaths;
     private readonly configuredSearchPaths;
@@ -130,22 +137,27 @@ export declare class AutoRAGAgent {
     private readonly merger;
     private readonly datasourceFilter;
     private readonly minSyncMethod;
-    private readonly bm25Method;
     private readonly jikjiClient;
     private readonly datasourceSkills;
     private readonly datasourceAccessOptions;
+    private readonly startupDiagnostics;
     private readonly datasourceAgentSkills;
     private readonly parserOptions;
     private readonly dupeyOptions;
     private readonly excludeExactDuplicates;
     private readonly baseSystemPromptConfig;
     private readonly droppedCallerToolNames;
+    private readonly searchTimeoutMs;
+    private readonly maxSearchToolCalls;
+    private searchToolCallCount;
+    private readonly sourceSearchCallCounts;
     constructor(options: AutoRAGAgentOptions);
     scanDuplicateDocuments(): Promise<ScanDuplicateDocumentsDetails>;
     private withMemoryContext;
     private resolveSessionModel;
     private createSearchSession;
     private configureSearchSession;
+    private withPerSourceSearchLimit;
     private recordSearchToolEvent;
     private currentSystemPromptConfig;
     subscribe(listener: Parameters<Agent["subscribe"]>[0]): () => void;
@@ -166,6 +178,13 @@ export declare class AutoRAGAgent {
     recordFeedbackByIds(usefulFeedbackIds: readonly string[], notUsefulFeedbackIds?: readonly string[]): void;
     getResultRegistry(sessionId?: string): ReadonlyMap<number, CuratedResult>;
     searchDocuments(query: string, options?: RetrievalOptions): Promise<SearchDocumentsResponse>;
+    /**
+     * Stream bounded progress updates while retaining the stable
+     * {@link searchDocuments} promise API. Progress is sourced from model text
+     * deltas, so callers can render it in a TUI, CLI, or parent agent and can
+     * stop by calling {@link abort}.
+     */
+    searchDocumentsStream(query: string, options?: RetrievalOptions): AsyncGenerator<SearchDocumentsStreamEvent, void, void>;
     private datasourceAccessContext;
     /**
      * Build the Pi agent-skill list for datasource skills authorized by the
@@ -180,9 +199,17 @@ export declare class AutoRAGAgent {
      */
     loadDatasourceSkill(name: string): Skill | undefined;
     private indexDatasources;
-    /** Path-opaque component diagnostics (e.g. BM25 readiness) for the search response. */
+    /** Path-opaque component diagnostics for the search response. */
     private collectComponentDiagnostics;
-    buildSearchPrompt(query: string, options: RetrievalOptions): string;
+    private prefetchInitialRetrievalContext;
+    /**
+     * Prepare MinSync lazily without holding up the current search. Parsed
+     * mirrors are built first because MinSync stages and indexes those mirrors.
+     */
+    /** Prepare MinSync in the background; the returned promise is for tests. */
+    scheduleMinSyncPrepareForTest(): Promise<MinSyncSyncResult | undefined> | undefined;
+    private scheduleMinSyncPrepare;
+    buildSearchPrompt(query: string, options: RetrievalOptions, initialRetrievalContext?: string): string;
     refresh(force?: boolean, opts?: AutoRAGRefreshOptions): Promise<AutoRAGRefreshResult>;
     /**
      * Path-opaque snapshot of corpus freshness and the last refresh outcome. Runs
@@ -209,10 +236,15 @@ export declare class AutoRAGAgent {
      */
     private scanMirrorStaleness;
     private exactDuplicateExclusions;
-    syncBM25(): Promise<BM25SyncResult | undefined>;
     syncMinSync(): Promise<MinSyncSyncResult | undefined>;
     prepareJikji(): Promise<readonly AutoRAGJikjiPrepareResult[] | undefined>;
     private executeJikjiPrepare;
+    /**
+     * Start Jikji preparation without delaying the current search turn. A
+     * subsequent turn can use the prepared index, while the child process keeps
+     * running after emit_autorag_results has terminated the agent loop.
+     */
+    private scheduleJikjiPrepare;
     private sanitizeJikjiPrepareResult;
     /**
      * Provider method for the jikji_find tool. Runs JikjiClient.find over all
@@ -271,7 +303,7 @@ export declare class AutoRAGAgent {
         results: RetrievalResult[];
         diagnostics: RetrievalDiagnostic[];
     }>;
-    /** The retrieval method registry (posix active; vector/bm25/hybrid pluggable). */
+    /** The retrieval method registry (posix, MinSync, and datasource methods). */
     getMethodRegistry(): RetrievalMethodRegistry;
     getSystemPrompt(): string;
     private rerankWithMemory;

package/dist/agent/bash-tool.d.ts

@@ -17,7 +17,9 @@ export interface BashToolOptions {
 export interface BashToolDetails {
     readonly method: "bash";
     readonly command: string;
+    readonly aborted: boolean;
     readonly exitCode: number | undefined;
+    readonly terminationFailed: boolean;
     readonly timedOut: boolean;
     readonly truncated: boolean;
 }

package/dist/agent/index.d.ts

@@ -7,6 +7,6 @@ export { createEmitResultsTool, EMIT_AUTORAG_RESULTS_TOOL_NAME } from "./emit-re
 export { createJikjiFindTool, JIKJI_FIND_TOOL_NAME, type JikjiFindDetails, type JikjiFindPerRootPolicy, type JikjiFindProvider, type JikjiFindProviderResult, type MergedJikjiPolicy, } from "./jikji-find-tool.js";
 export { createSearchAllDocumentsTool, SEARCH_ALL_DOCUMENTS_TOOL_NAME, type SearchAllDocumentsDetails, type SearchAllDocumentsProvider, type SearchAllDocumentsResult, } from "./search-all-tool.js";
 export { createSearchDatasourceDocumentsTool, type DatasourceSearchProvider, SEARCH_DATASOURCE_DOCUMENTS_TOOL_NAME, type SearchDatasourceDocumentsDetails, } from "./search-datasource-tool.js";
-export type { SearchDocumentDiagnostic, SearchDocumentDiagnosticCode, SearchDocumentDiagnosticSeverity, SearchDocumentEvidence, SearchDocumentResult, SearchDocumentsResponse, SearchDocumentWarning, } from "./search-documents.js";
+export type { SearchDocumentDiagnostic, SearchDocumentDiagnosticCode, SearchDocumentDiagnosticSeverity, SearchDocumentEvidence, SearchDocumentResult, SearchDocumentsResponse, SearchDocumentsStreamEvent, SearchDocumentWarning, } from "./search-documents.js";
 export { createSearchMinSyncDocumentsTool, SEARCH_MINSYNC_DOCUMENTS_TOOL_NAME, type SearchMinSyncDocumentsDetails, } from "./search-minsync-tool.js";
 export { buildSystemPrompt, type SystemPromptConfig } from "./system-prompt.js";

package/dist/agent/search-documents.d.ts

@@ -9,12 +9,12 @@ export type SearchDocumentDiagnosticSeverity = "info" | "warning" | "error";
  * (#21/#22) does not require another source-breaking type change; codes not yet
  * emitted are still valid members.
  */
-export type SearchDocumentDiagnosticCode = "empty-query" | "unknown-warning" | "caller-tool-dropped" | "bm25-unavailable" | "bm25-degraded-fallback" | "minsync-unavailable" | "parser-skipped" | "parser-failed" | "duplicate-excluded" | "unsupported-file" | "stale-index" | "deleted-mirror" | "retrieval-method-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "refresh-failed" | "watch-failed" | "watch-limited";
+export type SearchDocumentDiagnosticCode = "empty-query" | "unknown-warning" | "caller-tool-dropped" | "minsync-unavailable" | "parser-skipped" | "parser-failed" | "pdf-java-version" | "duplicate-excluded" | "unsupported-file" | "stale-index" | "deleted-mirror" | "pdf-extract-thin" | "pdf-hybrid-unavailable" | "retrieval-method-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "refresh-failed" | "watch-failed" | "watch-limited" | "unknown-datasource-skill";
 export interface SearchDocumentDiagnostic {
     readonly code: SearchDocumentDiagnosticCode;
     readonly severity: SearchDocumentDiagnosticSeverity;
     readonly message: string;
-    /** Component label (e.g. "sanitizer", "bm25") or opaque virtual path — never a real filesystem path. */
+    /** Component label (e.g. "sanitizer", "minsync") or opaque virtual path — never a real filesystem path. */
     readonly source?: string;
 }
 export interface SearchDocumentEvidence {
@@ -43,6 +43,15 @@ export interface SearchDocumentsResponse {
      */
     readonly diagnostics?: readonly SearchDocumentDiagnostic[];
 }
+export type SearchDocumentsStreamEvent = {
+    readonly type: "progress";
+    readonly sessionId: string;
+    readonly query: string;
+    readonly text: string;
+} | {
+    readonly type: "complete";
+    readonly response: SearchDocumentsResponse;
+};
 type SearchSessions = Map<string, {
     query: string;
     registry: Map<number, CuratedResult>;
@@ -51,7 +60,7 @@ type ReadonlySearchSessions = ReadonlyMap<string, {
     query: string;
     registry: ReadonlyMap<number, CuratedResult>;
 }>;
-export declare function createEmptySearchDocumentsResponse(sessionId: string, query: string, sessions: SearchSessions): SearchDocumentsResponse;
+export declare function createEmptySearchDocumentsResponse(sessionId: string, query: string, sessions: SearchSessions, diagnostics?: readonly SearchDocumentDiagnostic[]): SearchDocumentsResponse;
 export declare function recordStructuredResultsSession(sessionId: string, query: string, details: AutoRAGResultsDetails, sessions: SearchSessions, memory: RetrievalMemory, componentDiagnostics?: readonly SearchDocumentDiagnostic[]): SearchDocumentsResponse;
 export declare function recordNumberedFeedback(sessions: ReadonlySearchSessions, memory: RetrievalMemory, sessionId: string, usefulNumbers: readonly number[], notUsefulNumbers: readonly number[]): void;
 export {};

package/dist/agent/search-minsync-tool.d.ts

@@ -1,14 +1,14 @@
 import type { AgentTool } from "@earendil-works/pi-agent-core";
 import { Type } from "typebox";
 import type { MinSyncVectorMethod } from "../minsync/method.js";
-export declare const SEARCH_MINSYNC_DOCUMENTS_TOOL_NAME = "search_minsync_documents";
+export declare const SEARCH_MINSYNC_DOCUMENTS_TOOL_NAME = "semantic_search_local_docs";
 declare const searchMinSyncSchema: Type.TObject<{
     query: Type.TString;
     topK: Type.TOptional<Type.TInteger>;
     scope: Type.TOptional<Type.TString>;
 }>;
 export interface SearchMinSyncDocumentsDetails {
-    readonly method: "search_minsync_documents";
+    readonly method: "semantic_search_local_docs";
     readonly resultCount: number;
     readonly sources: readonly string[];
     readonly available: boolean;

package/dist/cli/commands/evidence.d.ts

@@ -0,0 +1,3 @@
+import type { CommandContext } from "./types.js";
+/** Show the persisted source/chunk evidence attached to a search session. */
+export declare function runEvidence(ctx: CommandContext): Promise<number>;

package/dist/cli/commands/index.d.ts

@@ -1,8 +1,8 @@
 import type { CommandContext } from "./types.js";
 /**
- * Run the `autorag index` command. `reset` removes the parsed mirror, BM25,
- * and MinSync index directories under `.autorag` (preserving `bin`,
- * `datasources`, and the memory file). `rebuild` resets then re-runs a forced
+ * Run the `autorag index` command. `reset` removes the parsed mirror
+ * and MinSync indexes
+ * (preserving `bin`, `datasources`, and the memory file). `rebuild` resets then re-runs a forced
  * refresh. Returns 0 on success, 2 for usage/decline errors, 1 for runtime
  * errors including path-escape guard violations.
  */

package/dist/cli/commands/refresh.d.ts

@@ -2,13 +2,12 @@ import { type RefreshMethod } from "../../agent/agent.js";
 import type { CommandContext } from "./types.js";
 /**
  * `autorag refresh` — parse configured search paths and resync every active
- * index (parsed mirror, BM25, MinSync, datasources, jikji). Model-free: no LLM
+ * index (parsed mirror, MinSync, datasources, jikji). Model-free: no LLM
  * is constructed. Output is rendered through the path-opaque refresh renderer;
  * the raw result (which carries absolute `indexPath`) is never printed.
  *
- * `--method bm25,minsync,parsed` restricts which methods run. When omitted, all
- * methods run. Parsed mirrors are always synced when BM25 or MinSync is selected
- * (they index over the parsed mirrors).
+ * `--method minsync,parsed` restricts which methods run. When omitted, all
+ * methods run. Parsed mirrors are always synced when MinSync is selected.
  */
 export declare function runRefresh(ctx: CommandContext): Promise<number>;
 /**

package/dist/cli/commands/search.d.ts

@@ -21,10 +21,10 @@ export declare function classifySearchHealthHint(error: unknown): SearchHealthHi
 /**
  * Search-command dependencies. Tests inject `agentFactory` to bypass real
  * model construction and AutoRAGAgent instantiation, returning a stub whose
- * `searchDocuments` resolves a canned {@link SearchDocumentsResponse}.
+ * `searchDocumentsStream` yields progress and a canned completion.
  */
 export interface SearchDeps {
-    agentFactory?: (opts: AutoRAGAgentOptions) => Pick<AutoRAGAgent, "searchDocuments">;
+    agentFactory?: (opts: AutoRAGAgentOptions) => Pick<AutoRAGAgent, "searchDocumentsStream">;
     modelResolver?: (config: CliConfig) => ResolvedAgentModel;
 }
 /**

package/dist/cli/commands/tui.d.ts

@@ -0,0 +1,36 @@
+import type { AgentEvent } from "@earendil-works/pi-agent-core";
+import { AutoRAGAgent, type AutoRAGAgentOptions } from "../../agent/agent.js";
+import { type CliConfig, type ResolvedAgentModel } from "../config.js";
+import { type TuiSessionRecord, type TuiSessionStore } from "../tui-session-store.js";
+import type { CommandContext } from "./types.js";
+export { parseSlashCommand } from "../tui-commands.js";
+export interface TuiDriver {
+    submissions: string[];
+    rendered: string[];
+    started: boolean;
+    stopped: boolean;
+    onSubmit?: (text: string) => void | Promise<void>;
+    onExit?: () => void;
+    onEvent?: (event: AgentEvent) => void;
+    onInput?: (data: string) => void;
+}
+export interface TuiPresenter {
+    handle(event: AgentEvent): void;
+    lines(): readonly string[];
+    working(): boolean;
+    setWorking(active: boolean): void;
+    beginRun(): void;
+    interrupt(): void;
+    status(): string;
+    reset(): void;
+}
+type TuiAgent = Pick<AutoRAGAgent, "searchDocumentsStream"> & Partial<Pick<AutoRAGAgent, "subscribe" | "abort">>;
+export interface TuiDeps {
+    agentFactory?: (opts?: AutoRAGAgentOptions) => TuiAgent;
+    modelResolver?: (config: CliConfig) => ResolvedAgentModel;
+    tuiFactory?: (ctx: CommandContext) => TuiDriver;
+    sessionStore?: TuiSessionStore;
+}
+export declare function createTuiPresenter(): TuiPresenter;
+export declare function renderTuiSessionView(session: TuiSessionRecord | undefined, input: string): string;
+export declare function runTui(ctx: CommandContext, deps?: TuiDeps): Promise<number>;

package/dist/cli/commands/ui.d.ts

@@ -0,0 +1,13 @@
+import { startUiServer, type UiServer } from "../../ui/server.js";
+import type { CommandContext } from "./types.js";
+export interface UiCommandDeps {
+    readonly startServer?: typeof startUiServer;
+    readonly openBrowser?: (url: string) => Promise<void>;
+    readonly waitUntilStopped?: (server: UiServer) => Promise<void>;
+    readonly createToken?: () => string;
+}
+/**
+ * `autorag ui` — local loopback dashboard for folders and datasource connections.
+ */
+export declare function runUi(ctx: CommandContext, deps?: UiCommandDeps): Promise<number>;
+export declare function openBrowser(url: string): Promise<void>;

package/dist/cli/commands/watch.d.ts

@@ -1,6 +1,6 @@
 import type { CommandContext } from "./types.js";
 /**
- * `autorag watch` — keep parsed/BM25/MinSync indexes current.
+ * `autorag watch` — keep parsed/MinSync indexes current.
  *
  * - default: long-running recursive fs watch with debounced refresh
  * - `--once`: single refresh tick (for cron / launchd / Task Scheduler)