@autorag/librarian 2.3.0 → 2.4.0

package/README.md

@@ -65,26 +65,62 @@ Different documents need different search strategies:
 | Legal documents, specifications | BM25 (keyword ranking) | Handles domain terminology well |
 | Mixed collections | Hybrid (vector + BM25) | Combines precision and recall |
 
-AutoRAG supports **pluggable retrieval methods**. It ships with lexical **BM25** and semantic **MinSync** methods wired through the `RetrievalMethodRegistry`, and the architecture is ready for additional vector and hybrid backends. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication.
+AutoRAG supports **pluggable retrieval methods**. Local lexical BM25, semantic vector, and hybrid retrieval all go through **MinSync** over one shared CDC chunk lifecycle, wired through the `RetrievalMethodRegistry`. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication. External datasources keep their own archive/index lifecycle.
 
-BM25 and MinSync are **enabled by default** — no explicit configuration is needed for standard lexical + semantic retrieval. Both can be disabled by setting `"bm25": false` or `"minSync": false` in the config file. MinSync uses a pre-installed binary (`autoInstall: false`); configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints. AutoRAG never forces TEI or any external embedding service.
+BM25, vector, and hybrid are **enabled by default** whenever MinSync is enabled. Disable local indexing with `"minSync": false`, or disable only lexical search with `"bm25": false`. MinSync auto-installs a verified release into `<workspace>/.autorag/bin` on first use (`autoInstall: true`); set `"autoInstall": false` only when managing the binary yourself. Configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints, and set `minSync.maxChunkSize` (or `--minsync-max-chunk-size`) when a local embedder has a smaller context window. AutoRAG never forces TEI or any external embedding service.
+
+See [docs/minsync-setup.md](docs/minsync-setup.md) for automatic installation,
+managed binary paths, and the local EmbeddingGemma QA flow.
 
 ### Real directory access
 
-AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. BM25 and MinSync index parsed markdown mirrors under `.autorag`.
+AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. MinSync indexes parsed markdown mirrors under `.autorag` for BM25, vector, and hybrid retrieval.
+
+### Thin PDF extraction retry
+
+The default PDF parser performs a cheap quality check for multi-page PDFs.
+When local markdown is unusually sparse (fewer than 800 characters or fewer
+than 40 characters per detected page, for at least three pages), it retries
+through OpenDataLoader's `docling-fast` hybrid backend with `hybridMode:
+"auto"` and a 30-second timeout. Dense PDFs are not retried, and hybrid is
+never used for single-page PDFs or as the first path for images.
+
+The gate is parser-owned and can be tuned through trusted programmatic
+`parserOptions`:
+
+```typescript
+new AutoRAGAgent({
+  searchPaths: ["/path/to/documents"],
+  parserOptions: {
+    thinExtract: {
+      minPages: 3,
+      minChars: 800,
+      minCharsPerPage: 40,
+      timeoutMs: 30_000,
+      hybrid: "docling-fast",
+      hybridMode: "auto",
+    },
+  },
+});
+```
+
+If the hybrid sidecar is missing, times out, or fails, AutoRAG keeps the local
+markdown and emits `pdf-extract-thin` plus `pdf-hybrid-unavailable`
+diagnostics; refresh remains successful.
+
+### Default Jikji discovery and indexing
 
-### Optional Jikji discovery and indexing
+AutoRAG uses [Jikji](https://github.com/NomaDamas/jikji) by default as a local CLI-backed **find-first discovery and indexing** layer. Jikji is not registered as a retrieval backend: it supplies bounded discovery answer packs while the librarian still reads original files directly.
 
-AutoRAG can opt into [Jikji](https://github.com/NomaDamas/jikji) as a local CLI-backed **find-first discovery and indexing** layer. Jikji is optional: AutoRAG does not vendor it, install it, or register it as a retrieval backend when enabled.
+The default agent calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly. If the binary or Rust toolchain is unavailable, Jikji reports a diagnostic and the normal filesystem/retrieval paths continue.
 
-When Jikji is configured, AutoRAG calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly.
+New `autorag init` configs include `"jikji": {}`. To opt out, set `"jikji": false` in `config.json`; programmatic callers can pass `jikji: false`.
 
 Programmatic use:
 
 ```typescript
 const agent = new AutoRAGAgent({
   searchPaths: ["/path/to/documents"],
-  jikji: { binaryPath: "jikji" },
 });
 await agent.prepareJikji();
 ```
@@ -118,7 +154,7 @@ searchable:
 If `dupey` is not installed or fails, refresh continues without exclusion and
 reports no destructive action; install it with `cargo install dupey`.
 
-The same `.autorag/jikji.json` shape configures Jikji when present:
+The same configuration shape customizes Jikji when needed:
 
 ```json
 {
@@ -153,7 +189,7 @@ channel or group chat as its own datasource.
 
 Security defaults are intentionally strict:
 
-- datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags` and `datasourceAccess.allowedScopes`;
+- datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags`; `allowedScopes` applies only to datasource methods that advertise the `scoped` capability;
 - model/tool arguments never grant datasource tags or scopes;
 - `search_datasource_documents` accepts only `{ query, topK?, scope? }`, and `scope` can only narrow trusted access.
 
@@ -171,6 +207,7 @@ Security defaults are intentionally strict:
 | Cloud drives | `cloud-drive` | **[`rclone`](https://rclone.org) CLI** | Incremental Google Drive Tier-1; OneDrive/network remotes; iCloud experimental |
 | Gmail / IMAP | `gmail` | Gmail REST v1, or **[`himalaya`](https://pimalaya.org) CLI** (`backend: "himalaya"`) | the himalaya backend indexes any IMAP/Maildir account it has configured — no OAuth plumbing |
 | Local mail exports | `mail-export` | filesystem (`.mbox` / `.eml`) | classic `From_` splitting, mailparser-based; count-only warnings |
+| Mail archives | `mailcrawl` | external [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | local email sync plus BM25, semantic, and hybrid retrieval |
 | Obsidian vault | `obsidian` | external [`qmd`](https://github.com/tobi/qmd) CLI | incremental `qmd update`, BM25 `qmd search`, semantic `qmd vsearch`; vault path via `connector.vaultPath` |
 | RSS / news | `rss` | HTTP feed polling | RSS 2.0 + Atom, feed/category hierarchy, 24h dedupe window |
 
@@ -187,14 +224,15 @@ Configure them in `config.json` (CLI) or pass `datasourceSkills` programmaticall
     "notion":   { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } },
     "github":   { "connector": { "repos": ["owner/repo"] } },
     "gmail":    { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
+    "mailcrawl": { "instanceId": "personal", "connector": { "account": "personal", "mailbox": "INBOX", "binaryPath": "mailcrawl" } },
     "personal-google-drive": { "type": "cloud-drive", "instanceId": "personal", "connector": { "provider": "google-drive", "remote": "personal-gdrive:", "include": ["**/*.md"] } },
     "company-onedrive": { "type": "cloud-drive", "instanceId": "work", "connector": { "provider": "onedrive", "remote": "company-onedrive:Documents" } },
     "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
     "rss":      { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
   },
   "datasourceAccess": {
-    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "cloud-drive", "obsidian", "rss"],
-    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
+    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "mailcrawl", "cloud-drive", "obsidian", "rss"],
+    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/mailcrawl/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
   }
 }
 ```
@@ -207,6 +245,15 @@ Install slacrawl with `brew install openclaw/tap/slacrawl`. AutoRAG invokes `sla
 
 Install notcrawl with `brew install openclaw/tap/notcrawl`. AutoRAG invokes `notcrawl sync` during datasource refresh and `notcrawl search --json` during retrieval. Optional trusted connector fields are `binaryPath` and `configPath`. Notion credentials and workspace definitions remain in notcrawl's own configuration rather than AutoRAG.
 
+Install and configure [`mailcrawl`](https://github.com/NomaDamas/mailcrawl)
+`@nomadamas/mailcrawl@0.1.4` or newer separately. AutoRAG invokes
+`mailcrawl sync` followed by `mailcrawl index` during datasource refresh, then
+calls `mailcrawl search` in BM25, semantic, or hybrid mode. The archive remains
+in mailcrawl's native store unless the operator explicitly sets
+`connector.dataDir`; Himalaya credentials and provider configuration remain
+owned by mailcrawl. 0.1.3 and earlier fail a repeated
+`index` after a no-op sync.
+
 Install and authenticate rclone separately (`brew install rclone && rclone
 config` on macOS), then configure the provider-neutral `cloud-drive` skill.
 `cloud-drive` is a reusable `type`: each datasource config key is a connection
@@ -240,7 +287,6 @@ const agent = new AutoRAGAgent({
   datasourceSkills: [kakao],
   datasourceAccess: {
     allowedTags: ["kakaotalk"],
-    allowedScopes: ["/kakao/personal/**"],
   },
 });
 
@@ -271,7 +317,6 @@ const agent = new AutoRAGAgent({
   datasourceSkills: [discord],
   datasourceAccess: {
     allowedTags: ["discord"],
-    allowedScopes: ["/discord/community/**"],
   },
 });
 ```
@@ -297,7 +342,7 @@ Or through the trusted config factory:
 Two defaults are deliberate and worth keeping:
 
 - **`defaultMode: "hybrid"`** — discrawl's FTS index strips newlines without substituting a space, welding words across line breaks into a single unsearchable token (measured at ~47% of post-newline words on a real archive). Semantic recall covers that gap. See [#1413](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1413).
-- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). For workspace-managed discrawl state, AutoRAG writes these values to `.autorag/datasources/discrawl/config.toml` while preserving unrelated discrawl settings. An explicit `connector.configPath` remains operator-owned and is never rewritten. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
+- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). Configure these values in discrawl's own config; AutoRAG uses discrawl's native store unless an explicit `connector.configPath` is supplied. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources can share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
 
 A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as `/kakao/personal/chunks/<chunk-id>`, and permission tags that match your server-side access policy.
 
@@ -349,6 +394,30 @@ Config path precedence is `--config` > `AUTORAG_CONFIG` > `~/.autorag/config.jso
 
 `autorag health` checks model/provider auth before a search — it resolves the model, verifies credential presence, and optionally probes one completion call. Use it to diagnose model, provider, auth, or timeout failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness). When `autorag search` fails for a model/provider reason, the error output includes a hint pointing to `autorag health`.
 
+`autorag ui` opens a loopback-only page (`127.0.0.1`) to connect local folders and datasource skills without editing JSON. It writes the same trusted `datasources` / `datasourceAccess` fields as a hand-edited config, stores env-var *names* rather than secrets, and refuses non-loopback binds. Use `--no-open` to print the URL without launching a browser.
+
+For a deliberately deployed UI, opt in explicitly in `config.json`. Keep the
+session token in the environment, set the public URL used by the browser, and
+allow only the frontend origins that should make credentialed API requests:
+
+```json
+{
+  "ui": {
+    "host": "0.0.0.0",
+    "port": 8787,
+    "allowRemote": true,
+    "publicOrigin": "https://autorag.example.com",
+    "corsOrigins": ["https://autorag.example.com"],
+    "tokenEnv": "AUTORAG_UI_TOKEN"
+  }
+}
+```
+
+Start it with `AUTORAG_UI_TOKEN` set to a random value of at least 16
+characters. Local use remains the safe default: omit `ui` (or leave
+`allowRemote` false) and AutoRAG binds to loopback, including a working
+`localhost` URL on systems that resolve it to IPv6.
+
 ## Installation
 
 Published as `@autorag/librarian` (dist bundled with Bun, runtime Node ≥ 24 or Bun):
@@ -360,9 +429,45 @@ bun install -g @autorag/librarian   # autorag CLI
 bun add github:NomaDamas/AutoRAG-2.0
 ```
 
+PDF parsing requires **Java 11 or newer** because the bundled
+`@opendataloader/pdf` runtime is compiled for Java 11. This applies to
+Windows, macOS, and Linux. Verify the runtime that will be selected from
+`PATH` before indexing PDFs:
+
+```bash
+java -version
+```
+
+On Windows, ensure a Java 11+ installation appears before older Java
+installations in `PATH` (or set `JAVA_HOME` to the Java 11+ installation and
+start a new shell). Java 8 is not supported by the PDF parser.
+
 Git-based installs build `dist/` via the `prepare` script and require Bun on the installing machine.
 External tool binaries auto-install on first use into `<workspace>/.autorag/bin`: **MinSync** downloads a verified GitHub release asset (on by default; `minSync.autoInstall: false` to opt out), and **Jikji** compiles the [`jikji-cli`](https://crates.io/crates/jikji-cli) crate via cargo (requires the [Rust toolchain](https://rustup.rs); `jikji.autoInstall: false` to opt out). New `autorag init` configs enable Jikji find-first discovery by default. KakaoTalk (`katok`) and Discord (`discrawl`) stay manual, optional installs. All of them degrade gracefully when missing — core BM25 search works without any of them.
 
+### Experimental TUI (beta)
+
+The latest TUI work is currently available on the `feat/experimental-tui`
+branch. To use this beta version from source:
+
+```bash
+git clone --branch feat/experimental-tui https://github.com/Marker-Inc-Korea/AutoRAG.git
+cd AutoRAG
+bun install
+bun run build
+
+# Configure your document roots and model once:
+node dist/cli/index.js init --search-paths /path/to/documents
+
+# Start the beta TUI:
+node dist/cli/index.js tui
+```
+
+Inside the TUI, use `/resume` to browse saved sessions. Selecting a session
+replaces the current view with that session instead of mixing the two
+conversation histories. This branch is experimental and may change before the
+feature is included in a published release.
+
 ## Quick Start
 
 ```typescript

package/dist/agent/agent.d.ts

@@ -5,10 +5,10 @@ import type { DatasourceIndexResult, DatasourceSkill } from "../datasource/types
 import { type DupeyCliOptions } from "../dupey/index.js";
 import { type JikjiFailureReason, type JikjiOptions } from "../jikji/index.js";
 import type { ResultFeedback } from "../memory/memory.js";
-import { type MinSyncSyncResult, type MinSyncVectorMethodOptions } from "../minsync/index.js";
+import { type MinSyncBM25MethodOptions, type MinSyncSyncResult, type MinSyncVectorMethodOptions } from "../minsync/index.js";
 import { type ParsedMirrorSyncResult } from "../mirror/sync.js";
 import type { DefaultParserRegistryOptions } from "../parser/index.js";
-import { type BM25MethodOptions, type BM25SyncResult } from "../retrieval/methods/bm25.js";
+import { type BM25SyncResult } from "../retrieval/methods/bm25.js";
 import { RetrievalMethodRegistry } from "../retrieval/registry.js";
 import type { CuratedResult, RetrievalDiagnostic, RetrievalOptions, RetrievalResult } from "../retrieval/types.js";
 import { type ScanDuplicateDocumentsDetails } from "./dupey-tool.js";
@@ -30,7 +30,8 @@ export interface AutoRAGMinSyncRefreshResult {
     readonly synced: number;
     readonly reason?: string;
 }
-export interface AutoRAGRefreshResult extends ParsedMirrorSyncResult {
+export interface AutoRAGRefreshResult extends Omit<ParsedMirrorSyncResult, "diagnostics"> {
+    readonly diagnostics: readonly SearchDocumentDiagnostic[];
     readonly bm25?: BM25SyncResult;
     readonly minsync?: AutoRAGMinSyncRefreshResult;
     readonly datasources?: readonly DatasourceIndexResult[];
@@ -77,14 +78,20 @@ export interface AutoRAGAgentOptions {
     workspacePath?: string;
     tools?: AgentTool[];
     minSync?: Omit<MinSyncVectorMethodOptions, "root"> | false;
-    bm25?: Omit<BM25MethodOptions, "root"> | false;
-    jikji?: JikjiOptions;
+    bm25?: Omit<MinSyncBM25MethodOptions, "root"> | false;
+    jikji?: JikjiOptions | false;
     autoRefresh?: AutoRefreshOptions;
     parserOptions?: DefaultParserRegistryOptions;
     dupey?: DupeyCliOptions | false;
     excludeExactDuplicates?: boolean;
     datasourceSkills?: readonly DatasourceSkill[];
     datasourceAccess?: DatasourceAccessContextOptions;
+    /** Non-fatal diagnostics from config/agent construction (e.g. skipped unknown datasources). */
+    startupDiagnostics?: readonly SearchDocumentDiagnostic[];
+    /** Maximum time a model/tool search may run before it is aborted. */
+    searchTimeoutMs?: number;
+    /** Maximum number of retrieval/tool executions allowed in one search. */
+    maxSearchToolCalls?: number;
 }
 export interface AutoRAGSearchSession {
     readonly agent: Agent;
@@ -134,12 +141,16 @@ export declare class AutoRAGAgent {
     private readonly jikjiClient;
     private readonly datasourceSkills;
     private readonly datasourceAccessOptions;
+    private readonly startupDiagnostics;
     private readonly datasourceAgentSkills;
     private readonly parserOptions;
     private readonly dupeyOptions;
     private readonly excludeExactDuplicates;
     private readonly baseSystemPromptConfig;
     private readonly droppedCallerToolNames;
+    private readonly searchTimeoutMs;
+    private readonly maxSearchToolCalls;
+    private searchToolCallCount;
     constructor(options: AutoRAGAgentOptions);
     scanDuplicateDocuments(): Promise<ScanDuplicateDocumentsDetails>;
     private withMemoryContext;
@@ -182,7 +193,8 @@ export declare class AutoRAGAgent {
     private indexDatasources;
     /** Path-opaque component diagnostics (e.g. BM25 readiness) for the search response. */
     private collectComponentDiagnostics;
-    buildSearchPrompt(query: string, options: RetrievalOptions): string;
+    private prefetchInitialRetrievalContext;
+    buildSearchPrompt(query: string, options: RetrievalOptions, initialRetrievalContext?: string): string;
     refresh(force?: boolean, opts?: AutoRAGRefreshOptions): Promise<AutoRAGRefreshResult>;
     /**
      * Path-opaque snapshot of corpus freshness and the last refresh outcome. Runs
@@ -209,7 +221,7 @@ export declare class AutoRAGAgent {
      */
     private scanMirrorStaleness;
     private exactDuplicateExclusions;
-    syncBM25(): Promise<BM25SyncResult | undefined>;
+    syncBM25(minsync?: MinSyncSyncResult): Promise<BM25SyncResult | undefined>;
     syncMinSync(): Promise<MinSyncSyncResult | undefined>;
     prepareJikji(): Promise<readonly AutoRAGJikjiPrepareResult[] | undefined>;
     private executeJikjiPrepare;

package/dist/agent/bash-tool.d.ts

@@ -17,7 +17,9 @@ export interface BashToolOptions {
 export interface BashToolDetails {
     readonly method: "bash";
     readonly command: string;
+    readonly aborted: boolean;
     readonly exitCode: number | undefined;
+    readonly terminationFailed: boolean;
     readonly timedOut: boolean;
     readonly truncated: boolean;
 }

package/dist/agent/search-bm25-tool.d.ts

@@ -1,7 +1,8 @@
 import type { AgentTool } from "@earendil-works/pi-agent-core";
 import { Type } from "typebox";
-import type { BM25Method, BM25Status } from "../retrieval/methods/bm25.js";
-export declare const SEARCH_BM25_DOCUMENTS_TOOL_NAME = "search_bm25_documents";
+import type { BM25Status } from "../retrieval/methods/bm25.js";
+import type { RetrievalResult } from "../retrieval/types.js";
+export declare const SEARCH_BM25_DOCUMENTS_TOOL_NAME = "lexical_search_local_docs";
 declare const searchBM25Schema: Type.TObject<{
     query: Type.TString;
     topK: Type.TOptional<Type.TInteger>;
@@ -14,5 +15,12 @@ export interface SearchBM25DocumentsDetails {
     readonly readiness: BM25Status["readiness"];
     readonly engine: BM25Status["engine"];
 }
-export declare function createSearchBM25DocumentsTool(getMethod: () => BM25Method | undefined, resolveScope?: (scope: string | undefined) => string | undefined): AgentTool<typeof searchBM25Schema, SearchBM25DocumentsDetails>;
+export interface BM25SearchMethod {
+    getStatus(): BM25Status;
+    retrieve(query: string, options: {
+        topK?: number;
+        scope?: string;
+    }): Promise<RetrievalResult[]>;
+}
+export declare function createSearchBM25DocumentsTool(getMethod: () => BM25SearchMethod | undefined, resolveScope?: (scope: string | undefined) => string | undefined): AgentTool<typeof searchBM25Schema, SearchBM25DocumentsDetails>;
 export {};

package/dist/agent/search-documents.d.ts

@@ -9,7 +9,7 @@ export type SearchDocumentDiagnosticSeverity = "info" | "warning" | "error";
  * (#21/#22) does not require another source-breaking type change; codes not yet
  * emitted are still valid members.
  */
-export type SearchDocumentDiagnosticCode = "empty-query" | "unknown-warning" | "caller-tool-dropped" | "bm25-unavailable" | "bm25-degraded-fallback" | "minsync-unavailable" | "parser-skipped" | "parser-failed" | "duplicate-excluded" | "unsupported-file" | "stale-index" | "deleted-mirror" | "retrieval-method-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "refresh-failed" | "watch-failed" | "watch-limited";
+export type SearchDocumentDiagnosticCode = "empty-query" | "unknown-warning" | "caller-tool-dropped" | "bm25-unavailable" | "bm25-degraded-fallback" | "minsync-unavailable" | "parser-skipped" | "parser-failed" | "pdf-java-version" | "duplicate-excluded" | "unsupported-file" | "stale-index" | "deleted-mirror" | "pdf-extract-thin" | "pdf-hybrid-unavailable" | "retrieval-method-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "refresh-failed" | "watch-failed" | "watch-limited" | "unknown-datasource-skill";
 export interface SearchDocumentDiagnostic {
     readonly code: SearchDocumentDiagnosticCode;
     readonly severity: SearchDocumentDiagnosticSeverity;
@@ -51,7 +51,7 @@ type ReadonlySearchSessions = ReadonlyMap<string, {
     query: string;
     registry: ReadonlyMap<number, CuratedResult>;
 }>;
-export declare function createEmptySearchDocumentsResponse(sessionId: string, query: string, sessions: SearchSessions): SearchDocumentsResponse;
+export declare function createEmptySearchDocumentsResponse(sessionId: string, query: string, sessions: SearchSessions, diagnostics?: readonly SearchDocumentDiagnostic[]): SearchDocumentsResponse;
 export declare function recordStructuredResultsSession(sessionId: string, query: string, details: AutoRAGResultsDetails, sessions: SearchSessions, memory: RetrievalMemory, componentDiagnostics?: readonly SearchDocumentDiagnostic[]): SearchDocumentsResponse;
 export declare function recordNumberedFeedback(sessions: ReadonlySearchSessions, memory: RetrievalMemory, sessionId: string, usefulNumbers: readonly number[], notUsefulNumbers: readonly number[]): void;
 export {};

package/dist/agent/search-minsync-tool.d.ts

@@ -1,14 +1,14 @@
 import type { AgentTool } from "@earendil-works/pi-agent-core";
 import { Type } from "typebox";
 import type { MinSyncVectorMethod } from "../minsync/method.js";
-export declare const SEARCH_MINSYNC_DOCUMENTS_TOOL_NAME = "search_minsync_documents";
+export declare const SEARCH_MINSYNC_DOCUMENTS_TOOL_NAME = "semantic_search_local_docs";
 declare const searchMinSyncSchema: Type.TObject<{
     query: Type.TString;
     topK: Type.TOptional<Type.TInteger>;
     scope: Type.TOptional<Type.TString>;
 }>;
 export interface SearchMinSyncDocumentsDetails {
-    readonly method: "search_minsync_documents";
+    readonly method: "semantic_search_local_docs";
     readonly resultCount: number;
     readonly sources: readonly string[];
     readonly available: boolean;

package/dist/cli/commands/index.d.ts

@@ -1,8 +1,8 @@
 import type { CommandContext } from "./types.js";
 /**
- * Run the `autorag index` command. `reset` removes the parsed mirror, BM25,
- * and MinSync index directories under `.autorag` (preserving `bin`,
- * `datasources`, and the memory file). `rebuild` resets then re-runs a forced
+ * Run the `autorag index` command. `reset` removes the parsed mirror
+ * MinSync, and leftover legacy BM25 index directories under `.autorag`
+ * (preserving `bin`, `datasources`, and the memory file). `rebuild` resets then re-runs a forced
  * refresh. Returns 0 on success, 2 for usage/decline errors, 1 for runtime
  * errors including path-escape guard violations.
  */

package/dist/cli/commands/tui.d.ts

@@ -0,0 +1,36 @@
+import type { AgentEvent } from "@earendil-works/pi-agent-core";
+import { AutoRAGAgent, type AutoRAGAgentOptions } from "../../agent/agent.js";
+import { type CliConfig, type ResolvedAgentModel } from "../config.js";
+import { type TuiSessionRecord, type TuiSessionStore } from "../tui-session-store.js";
+import type { CommandContext } from "./types.js";
+export { parseSlashCommand } from "../tui-commands.js";
+export interface TuiDriver {
+    submissions: string[];
+    rendered: string[];
+    started: boolean;
+    stopped: boolean;
+    onSubmit?: (text: string) => void | Promise<void>;
+    onExit?: () => void;
+    onEvent?: (event: AgentEvent) => void;
+    onInput?: (data: string) => void;
+}
+export interface TuiPresenter {
+    handle(event: AgentEvent): void;
+    lines(): readonly string[];
+    working(): boolean;
+    setWorking(active: boolean): void;
+    beginRun(): void;
+    interrupt(): void;
+    status(): string;
+    reset(): void;
+}
+type TuiAgent = Pick<AutoRAGAgent, "searchDocuments"> & Partial<Pick<AutoRAGAgent, "subscribe" | "abort">>;
+export interface TuiDeps {
+    agentFactory?: (opts?: AutoRAGAgentOptions) => TuiAgent;
+    modelResolver?: (config: CliConfig) => ResolvedAgentModel;
+    tuiFactory?: (ctx: CommandContext) => TuiDriver;
+    sessionStore?: TuiSessionStore;
+}
+export declare function createTuiPresenter(): TuiPresenter;
+export declare function renderTuiSessionView(session: TuiSessionRecord | undefined, input: string): string;
+export declare function runTui(ctx: CommandContext, deps?: TuiDeps): Promise<number>;

package/dist/cli/commands/ui.d.ts

@@ -0,0 +1,13 @@
+import { startUiServer, type UiServer } from "../../ui/server.js";
+import type { CommandContext } from "./types.js";
+export interface UiCommandDeps {
+    readonly startServer?: typeof startUiServer;
+    readonly openBrowser?: (url: string) => Promise<void>;
+    readonly waitUntilStopped?: (server: UiServer) => Promise<void>;
+    readonly createToken?: () => string;
+}
+/**
+ * `autorag ui` — local loopback dashboard for folders and datasource connections.
+ */
+export declare function runUi(ctx: CommandContext, deps?: UiCommandDeps): Promise<number>;
+export declare function openBrowser(url: string): Promise<void>;

package/dist/cli/config.d.ts

@@ -4,7 +4,6 @@ import { type LoadLocalAutoRAGModelOptions } from "../agent/local-model.js";
 import type { DatasourceAccessContextOptions } from "../datasource/access-context.js";
 import { type DatasourcesConfig } from "../datasource/skills/factory.js";
 import type { EnsureMinSyncBinaryOptions, MinSyncEmbedderConfig } from "../minsync/index.js";
-import type { BM25Engine, BM25FallbackMode } from "../retrieval/methods/bm25.js";
 export declare const DEFAULT_CONFIG_FILENAME = "config.json";
 export declare const LEGACY_CONFIG_FILENAME = "autorag.config.json";
 export { AUTORAG_HOME_ENV, resolveAutoRAGHome } from "../config/home.js";
@@ -17,14 +16,10 @@ export declare class ConfigError extends Error {
  */
 export interface Bm25MethodConfig {
     enabled?: boolean;
-    indexPath?: string;
-    fallback?: BM25FallbackMode;
-    forceEngine?: Exclude<BM25Engine, "none">;
-    importBinding?: () => Promise<typeof import("@pngwasi/node-tantivy-binding")>;
 }
 /**
  * Typed MinSync method config persisted in `config.json`. Missing `enabled`
- * means enabled (true); `autoInstall` defaults to false. `embedder` carries
+ * means enabled (true); `autoInstall` defaults to true. `embedder` carries
  * the MinSync vector embedder settings validated by {@link normalizeEmbedder}.
  */
 export interface MinSyncMethodConfig {
@@ -32,6 +27,7 @@ export interface MinSyncMethodConfig {
     autoInstall?: boolean;
     binaryPath?: string;
     workspacePath?: string;
+    maxChunkSize?: number;
     installer?: Omit<EnsureMinSyncBinaryOptions, "root">;
     embedder?: MinSyncEmbedderConfig;
 }
@@ -45,6 +41,14 @@ export interface NormalizedIndexingConfig {
     bm25: Bm25MethodConfig;
     minSync: MinSyncMethodConfig;
 }
+export interface UiConfig {
+    host?: string;
+    port?: number;
+    allowRemote?: boolean;
+    publicOrigin?: string;
+    corsOrigins?: string[];
+    tokenEnv?: string;
+}
 export interface AgentModelConfig {
     /** Provider identity used for auth lookup and Model.provider (e.g. openrouter, fireworks, ollama). */
     provider: string;
@@ -79,7 +83,7 @@ export interface CliConfig {
     model?: AgentModelConfig;
     minSync?: MinSyncMethodConfig;
     bm25?: Bm25MethodConfig;
-    jikji?: Record<string, unknown>;
+    jikji?: Record<string, unknown> | false;
     parserOptions?: Record<string, unknown>;
     dupey?: {
         enabled?: boolean;
@@ -91,6 +95,8 @@ export interface CliConfig {
     datasources?: DatasourcesConfig;
     /** Trusted datasource allow-tags/allow-scopes. Absent ⇒ default-deny. */
     datasourceAccess?: DatasourceAccessContextOptions;
+    /** Optional local/deployment settings for the datasource UI. */
+    ui?: UiConfig;
 }
 export interface ResolveConfigInput {
     flags: Record<string, string | boolean | undefined>;
@@ -115,7 +121,7 @@ export declare function normalizeEmbedder(raw: unknown, path: string): MinSyncEm
 /**
  * Normalize raw indexing method config into a fully-populated shape.
  *
- * - `undefined` / missing key => `{ enabled: true }` (minSync `autoInstall: false`)
+ * - `undefined` / missing key => `{ enabled: true, autoInstall: true }`
  * - `false` => `{ enabled: false }` (disabled marker)
  * - object merges with `enabled: true` default and is validated
  *
@@ -166,6 +172,9 @@ export interface ResolvedAgentModelDetailed {
 }
 export declare function resolveAgentModel(config: CliConfig, localOptions?: LoadLocalAutoRAGModelOptions): ResolvedAgentModel;
 export declare function resolveAgentModelDetailed(config: CliConfig, localOptions?: LoadLocalAutoRAGModelOptions): ResolvedAgentModelDetailed;
+export declare function readRawConfigObject(path: string): Record<string, unknown>;
+/** Atomically replace a config file, preserving the caller's JSON object. */
+export declare function writeConfigObject(path: string, config: unknown): void;
 export declare function writeDefaultConfig(path: string, partial: Partial<CliConfig>, opts?: {
     force?: boolean;
     atomicCreate?: boolean;