@autorag/librarian 2.2.0 → 2.4.0

package/README.md

@@ -10,7 +10,11 @@
 
 AutoRAG searches your PDFs, wikis, notes, research papers, and knowledge bases — then curates the results into clean, numbered knowledge units. No raw grep dumps. Just answers.
 
-AutoRAG is a customized [Pi](https://github.com/earendil-works/pi-mono) agent — the Pi agent loop configured into a librarian. Searches use a two-tier workflow: a parent orchestrator delegates exploration to explorer agents. The roles and providers are independently configured from the models available in the user's authenticated runtime; AutoRAG does not ship a private provider default.
+AutoRAG is a customized [Pi](https://github.com/earendil-works/pi-mono) agent — the Pi agent loop configured into a librarian. The AutoRAG librarian retrieves candidates, reads source files directly, judges the evidence, and curates the structured answer. Its model and provider come from the user's authenticated runtime; AutoRAG does not ship a private provider default.
+
+AutoRAG itself is the specialized search agent, not a coordinator for other
+model roles. You configure one model, and that model owns the complete
+retrieval, reading, judgment, and curation loop.
 
 ## Why AutoRAG
 
@@ -61,43 +65,96 @@ Different documents need different search strategies:
 | Legal documents, specifications | BM25 (keyword ranking) | Handles domain terminology well |
 | Mixed collections | Hybrid (vector + BM25) | Combines precision and recall |
 
-AutoRAG supports **pluggable retrieval methods**. It ships with lexical **BM25** and semantic **MinSync** methods wired through the `RetrievalMethodRegistry`, and the architecture is ready for additional vector and hybrid backends. The parent orchestrator owns process-bound retrieval tools and gives bounded seed packs to explorers; explorers use read-only `read`/`grep`/`find`/`ls` tools to inspect the underlying documents. The `ResultMerger` handles cross-method score normalization and deduplication — you get one unified result set regardless of how many methods contributed.
+AutoRAG supports **pluggable retrieval methods**. Local lexical BM25, semantic vector, and hybrid retrieval all go through **MinSync** over one shared CDC chunk lifecycle, wired through the `RetrievalMethodRegistry`. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication. External datasources keep their own archive/index lifecycle.
+
+BM25, vector, and hybrid are **enabled by default** whenever MinSync is enabled. Disable local indexing with `"minSync": false`, or disable only lexical search with `"bm25": false`. MinSync auto-installs a verified release into `<workspace>/.autorag/bin` on first use (`autoInstall: true`); set `"autoInstall": false` only when managing the binary yourself. Configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints, and set `minSync.maxChunkSize` (or `--minsync-max-chunk-size`) when a local embedder has a smaller context window. AutoRAG never forces TEI or any external embedding service.
 
-BM25 and MinSync are **enabled by default** — no explicit configuration is needed for standard lexical + semantic retrieval. Both can be disabled by setting `"bm25": false` or `"minSync": false` in the config file. MinSync uses a pre-installed binary (`autoInstall: false`); configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints. AutoRAG never forces TEI or any external embedding service.
+See [docs/minsync-setup.md](docs/minsync-setup.md) for automatic installation,
+managed binary paths, and the local EmbeddingGemma QA flow.
 
 ### Real directory access
 
-AutoRAG reads configured source directories through delegated explorer tasks. Each explorer is assigned exactly one normalized configured search root as its `cwd`; the top-level `subagent` invocation sets `agentScope: "user"` and `artifacts: false` exactly once for single, `tasks`, `chain`, or `parallel` dispatch, and nested explorer task items omit both fields. Project-local `.pi-subagents` debug artifacts are disabled. Explorers use read-only `read`/`grep`/`find`/`ls`; the parent orchestrator owns retrieval seed tools and must delegate document reading before curating. Curated answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. BM25 and MinSync index parsed markdown mirrors under `.autorag`.
+AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. MinSync indexes parsed markdown mirrors under `.autorag` for BM25, vector, and hybrid retrieval.
 
-Each explorer `task` contains an Assignment V1 block — a sentinel-wrapped JSON body with `originalQuery` (the caller query verbatim), `method` (the selected retrieval path), and `queryVariants` (a nonempty array), followed by canonical role lines requiring `retrievedAt` and temporal metadata:
+### Thin PDF extraction retry
 
-```text
-<<<AUTORAG_ASSIGNMENT_V1>>>
-{"originalQuery":"<caller query verbatim>","method":"<selected retrieval method>","queryVariants":["<variant 1>","<variant 2>"]}
-<<<END_AUTORAG_ASSIGNMENT_V1>>>
-Required handoff: include retrievedAt.
-Required handoff: include temporal metadata.
+The default PDF parser performs a cheap quality check for multi-page PDFs.
+When local markdown is unusually sparse (fewer than 800 characters or fewer
+than 40 characters per detected page, for at least three pages), it retries
+through OpenDataLoader's `docling-fast` hybrid backend with `hybridMode:
+"auto"` and a 30-second timeout. Dense PDFs are not retried, and hybrid is
+never used for single-page PDFs or as the first path for images.
+
+The gate is parser-owned and can be tuned through trusted programmatic
+`parserOptions`:
+
+```typescript
+new AutoRAGAgent({
+  searchPaths: ["/path/to/documents"],
+  parserOptions: {
+    thinExtract: {
+      minPages: 3,
+      minChars: 800,
+      minCharsPerPage: 40,
+      timeoutMs: 30_000,
+      hybrid: "docling-fast",
+      hybridMode: "auto",
+    },
+  },
+});
 ```
 
-A legacy labeled format (`Original query:`, `Selected retrieval method:`, `Query variants:`) is accepted for compatibility. Missing or null top-level `artifacts`, `agentScope`, and leaf `model` fields are safely autofilled before validation; explicit wrong values (`artifacts: true`, `agentScope: "project"`, a non-configured model) remain rejected. There is no single-agent fallback. See [docs/subagent-orchestration.md](docs/subagent-orchestration.md) for the full dispatch contract, templates, anti-examples, and stable coded dispatch errors.
+If the hybrid sidecar is missing, times out, or fails, AutoRAG keeps the local
+markdown and emits `pdf-extract-thin` plus `pdf-hybrid-unavailable`
+diagnostics; refresh remains successful.
+
+### Default Jikji discovery and indexing
 
-### Optional Jikji discovery and indexing
+AutoRAG uses [Jikji](https://github.com/NomaDamas/jikji) by default as a local CLI-backed **find-first discovery and indexing** layer. Jikji is not registered as a retrieval backend: it supplies bounded discovery answer packs while the librarian still reads original files directly.
 
-AutoRAG can opt into [Jikji](https://github.com/NomaDamas/jikji) as a local CLI-backed **find-first discovery and indexing** layer. Jikji is optional: AutoRAG does not vendor it, install it, or register it as a retrieval backend when enabled.
+The default agent calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly. If the binary or Rust toolchain is unavailable, Jikji reports a diagnostic and the normal filesystem/retrieval paths continue.
 
-When Jikji is configured, AutoRAG calls `jikji find ROOT "query" --json` via a policy-aware `jikji_find` tool as the first local-discovery action. The tool parses and validates the upstream answer-pack and honors its `handoff_action` (`direct_use` / `jikji_retry` / `raw_fallback_after_retry`), `tool_call_policy` (`stop_after_find`, `forbidden_tools`, `allowed_followups`), and `agent_should_not_rerank`. Explorer `read`/`grep`/`find`/`ls` discovery is the fallback only when the answer-pack permits raw fallback (`raw_fallback_after_retry`, after the required retry) or when Jikji is unavailable/unconfigured. `prepare`/`refresh` remain for indexing only and do not answer queries directly.
+New `autorag init` configs include `"jikji": {}`. To opt out, set `"jikji": false` in `config.json`; programmatic callers can pass `jikji: false`.
 
 Programmatic use:
 
 ```typescript
 const agent = new AutoRAGAgent({
   searchPaths: ["/path/to/documents"],
-  jikji: { binaryPath: "jikji" },
 });
 await agent.prepareJikji();
 ```
 
-The same `.autorag/jikji.json` shape configures Jikji when present:
+### Duplicate document management with dupey
+
+AutoRAG can use the external [`dupey`](https://github.com/NomaDamas/dupey) CLI
+to detect exact, near, and containment document families.
+
+```bash
+autorag duplicates /path/to/documents
+autorag duplicates --json
+```
+
+The command is read-only: it reports exact duplicate groups and review
+guidance, but never moves or deletes source files. The `scan_duplicate_documents`
+Agent tool exposes the same read-only scan to the orchestrator.
+
+Exact duplicate exclusion is enabled by default during parsed-mirror refresh.
+For each exact canonical-text hash, the newest filesystem copy is indexed and
+older copies are omitted. Disable it in `config.json` when both copies must be
+searchable:
+
+```json
+{
+  "dupey": { "enabled": true },
+  "excludeExactDuplicates": false
+}
+```
+
+If `dupey` is not installed or fails, refresh continues without exclusion and
+reports no destructive action; install it with `cargo install dupey`.
+
+The same configuration shape customizes Jikji when needed:
 
 ```json
 {
@@ -122,9 +179,17 @@ The upstream Rust `PrepareArgs` defines reference defaults that AutoRAG does not
 
 Datasource skills let AutoRAG search external, server-configured data sources through the same retrieval pipeline as local documents. A skill describes what it indexes, how it should be refreshed, what source instances exist, and which permission tags/scopes bound access. Retrieval still flows through `RetrievalMethodRegistry` → `ParallelRetriever` → datasource result filtering → `ResultMerger`; datasource skills do not create a parallel search path.
 
+Every datasource can be registered multiple times through a connection alias:
+use the config key as the unique name and set `type` to the reusable backend
+(`gmail`, `github`, `slack`, `discord`, `kakao`, `cloud-drive`, and so on).
+Each alias becomes an independently loadable agent skill with its own source
+scope and workspace namespace. Chat aliases search all channels by default;
+trusted `channels.ids` / `channels.names` allowlists can expose a particular
+channel or group chat as its own datasource.
+
 Security defaults are intentionally strict:
 
-- datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags` and `datasourceAccess.allowedScopes`;
+- datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags`; `allowedScopes` applies only to datasource methods that advertise the `scoped` capability;
 - model/tool arguments never grant datasource tags or scopes;
 - `search_datasource_documents` accepts only `{ query, topK?, scope? }`, and `scope` can only narrow trusted access.
 
@@ -139,9 +204,10 @@ Security defaults are intentionally strict:
 | Discord | `discord` | external [`discrawl`](https://github.com/openclaw/discrawl) CLI | guild/channel/thread/DM archive; FTS5 + semantic + hybrid retrieval, incremental sync |
 | Notion | `notion` | external [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | local-first page/database/block archive + FTS5 search |
 | GitHub Issues/PRs | `github` | GitHub REST (token optional) | issues + PR bodies per `owner/repo`; public repos work unauthenticated |
-| Google Drive | `gdrive` | Drive REST v3, or **[`rclone`](https://rclone.org) CLI** (`backend: "rclone"`) | Docs/Sheets exported as text; the rclone backend also opens any of rclone's 70+ remotes |
+| Cloud drives | `cloud-drive` | **[`rclone`](https://rclone.org) CLI** | Incremental Google Drive Tier-1; OneDrive/network remotes; iCloud experimental |
 | Gmail / IMAP | `gmail` | Gmail REST v1, or **[`himalaya`](https://pimalaya.org) CLI** (`backend: "himalaya"`) | the himalaya backend indexes any IMAP/Maildir account it has configured — no OAuth plumbing |
 | Local mail exports | `mail-export` | filesystem (`.mbox` / `.eml`) | classic `From_` splitting, mailparser-based; count-only warnings |
+| Mail archives | `mailcrawl` | external [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | local email sync plus BM25, semantic, and hybrid retrieval |
 | Obsidian vault | `obsidian` | external [`qmd`](https://github.com/tobi/qmd) CLI | incremental `qmd update`, BM25 `qmd search`, semantic `qmd vsearch`; vault path via `connector.vaultPath` |
 | RSS / news | `rss` | HTTP feed polling | RSS 2.0 + Atom, feed/category hierarchy, 24h dedupe window |
 
@@ -158,13 +224,15 @@ Configure them in `config.json` (CLI) or pass `datasourceSkills` programmaticall
     "notion":   { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } },
     "github":   { "connector": { "repos": ["owner/repo"] } },
     "gmail":    { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
-    "gdrive":   { "connector": { "backend": "rclone", "remote": "gdrive:" } },
+    "mailcrawl": { "instanceId": "personal", "connector": { "account": "personal", "mailbox": "INBOX", "binaryPath": "mailcrawl" } },
+    "personal-google-drive": { "type": "cloud-drive", "instanceId": "personal", "connector": { "provider": "google-drive", "remote": "personal-gdrive:", "include": ["**/*.md"] } },
+    "company-onedrive": { "type": "cloud-drive", "instanceId": "work", "connector": { "provider": "onedrive", "remote": "company-onedrive:Documents" } },
     "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
     "rss":      { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
   },
   "datasourceAccess": {
-    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "gdrive", "obsidian", "rss"],
-    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/gdrive/**", "/obsidian/**", "/rss/**"]
+    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "mailcrawl", "cloud-drive", "obsidian", "rss"],
+    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/mailcrawl/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
   }
 }
 ```
@@ -177,6 +245,30 @@ Install slacrawl with `brew install openclaw/tap/slacrawl`. AutoRAG invokes `sla
 
 Install notcrawl with `brew install openclaw/tap/notcrawl`. AutoRAG invokes `notcrawl sync` during datasource refresh and `notcrawl search --json` during retrieval. Optional trusted connector fields are `binaryPath` and `configPath`. Notion credentials and workspace definitions remain in notcrawl's own configuration rather than AutoRAG.
 
+Install and configure [`mailcrawl`](https://github.com/NomaDamas/mailcrawl)
+`@nomadamas/mailcrawl@0.1.4` or newer separately. AutoRAG invokes
+`mailcrawl sync` followed by `mailcrawl index` during datasource refresh, then
+calls `mailcrawl search` in BM25, semantic, or hybrid mode. The archive remains
+in mailcrawl's native store unless the operator explicitly sets
+`connector.dataDir`; Himalaya credentials and provider configuration remain
+owned by mailcrawl. 0.1.3 and earlier fail a repeated
+`index` after a no-op sync.
+
+Install and authenticate rclone separately (`brew install rclone && rclone
+config` on macOS), then configure the provider-neutral `cloud-drive` skill.
+`cloud-drive` is a reusable `type`: each datasource config key is a connection
+alias and becomes a separate agent skill and scope. Multiple Google accounts,
+or Google Drive plus OneDrive/iCloud, can therefore be loaded and searched
+independently.
+AutoRAG inventories with `rclone lsjson`, keeps a workspace-local manifest and
+managed mirror, and downloads only added or changed indexable files. Google
+Drive is Tier-1. OneDrive, Dropbox, SMB/SFTP/WebDAV, and mounted drives share
+the same manifest contract. iCloud Drive is experimental because rclone marks
+that backend Tier 4 and Apple ID/2FA sessions periodically require
+reauthentication. See [docs/datasource-skills.md](docs/datasource-skills.md)
+for filtering, size, concurrency, bandwidth, dry-run, and agent tool-calling
+details.
+
 #### KakaoTalk (katok)
 
 KakaoTalk was the first datasource skill. It uses the external [`katok`](https://github.com/NomaDamas/katok) CLI only — AutoRAG never reads KakaoTalk databases directly. `katok` failures return diagnostics instead of throwing, and remote embedding egress configuration is rejected before the CLI is spawned.
@@ -195,7 +287,6 @@ const agent = new AutoRAGAgent({
   datasourceSkills: [kakao],
   datasourceAccess: {
     allowedTags: ["kakaotalk"],
-    allowedScopes: ["/kakao/personal/**"],
   },
 });
 
@@ -226,7 +317,6 @@ const agent = new AutoRAGAgent({
   datasourceSkills: [discord],
   datasourceAccess: {
     allowedTags: ["discord"],
-    allowedScopes: ["/discord/community/**"],
   },
 });
 ```
@@ -252,7 +342,7 @@ Or through the trusted config factory:
 Two defaults are deliberate and worth keeping:
 
 - **`defaultMode: "hybrid"`** — discrawl's FTS index strips newlines without substituting a space, welding words across line breaks into a single unsearchable token (measured at ~47% of post-newline words on a real archive). Semantic recall covers that gap. See [#1413](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1413).
-- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). For workspace-managed discrawl state, AutoRAG writes these values to `.autorag/datasources/discrawl/config.toml` while preserving unrelated discrawl settings. An explicit `connector.configPath` remains operator-owned and is never rewritten. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
+- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). Configure these values in discrawl's own config; AutoRAG uses discrawl's native store unless an explicit `connector.configPath` is supplied. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources can share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
 
 A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as `/kakao/personal/chunks/<chunk-id>`, and permission tags that match your server-side access policy.
 
@@ -260,7 +350,7 @@ A datasource skill should provide polling/cron metadata for routine indexing, so
 
 AutoRAG is built for **non-code document retrieval**: manuals, legal docs, internal wikis, meeting notes, research literature, knowledge bases, PDFs.
 
-Code repositories work too (the explorer's `grep` is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
+Code repositories work too (direct `grep` is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
 
 ## Configuration and state
 
@@ -270,38 +360,63 @@ The default home state is kept outside the workspace:
 ~/.autorag/
 ├── config.json
 ├── memory.json
-├── logs/
-│   └── runs.jsonl
-└── pi-agent/
-    ├── auth.json
-    ├── models.json
-    ├── settings.json
-    └── sessions/
+└── logs/
+    └── runs.jsonl
 ```
 
-`config.json` selects sources, the workspace, memory path, retrieval settings, and the two agent models. Configure the roles independently with `agents.orchestrator` and `agents.explorer`. Provider and model IDs must refer to models available in the user's authenticated runtime:
+`config.json` selects sources, the workspace, memory path, retrieval settings, and the agent model. Provider and model IDs must refer to a model available in the user's authenticated runtime:
 
 ```json
 {
   "searchPaths": ["/path/to/documents"],
   "workspacePath": "/path/to/workspace",
   "memoryPath": "/Users/you/.autorag/memory.json",
-  "agents": {
-    "orchestrator": { "provider": "provider-name", "id": "reasoning-model" },
-    "explorer": { "provider": "provider-name", "id": "exploration-model" }
-  }
+  "model": { "provider": "provider-name", "id": "reasoning-model" }
 }
 ```
 
-`autorag init` leaves `agents` unset when no role-model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure both roles explicitly.
+`autorag init` leaves `model` unset when no model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure the model explicitly.
+
+For fast interactive search, prefer a model with reliable tool calling, high
+output TPS, and low first-token latency. A query can require several short
+model turns while AutoRAG alternates between retrieval tools and direct source
+reading, so model throughput has a visible effect on end-to-end response time.
+It does not accelerate BM25, MinSync, Jikji, filesystem access, or indexing
+itself. Larger reasoning models remain useful for difficult synthesis,
+conflicting evidence, and specialized domain judgment, but they are not a
+requirement for ordinary retrieval.
 
 Config path precedence is `--config` > `AUTORAG_CONFIG` > `~/.autorag/config.json`. When the home config is absent and `<cwd>/autorag.config.json` exists, AutoRAG copies the legacy file to `~/.autorag/config.json` without deleting or modifying the legacy file. The legacy cwd file is a migration source, not the default location.
 
-`memory.json` stores retrieval memory, `logs/runs.jsonl` records run events, and durable Pi models, settings, and sessions stay under `~/.autorag/pi-agent`. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under `<workspace>/.autorag`.
+`memory.json` stores retrieval memory and `logs/runs.jsonl` records run events. Model authentication remains with the user's configured provider or authenticated local runtime. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under `<workspace>/.autorag`.
 
 `autorag refresh` and `autorag index reset|rebuild` accept `--method <csv>` (e.g. `--method bm25,minsync,parsed`) to scope which indexing methods run or which index directories are removed. When omitted, all methods run. `autorag init` accepts `--embedder-*` flags to configure the MinSync embedder endpoint in the config file.
 
-`autorag health` checks model/provider auth and explorer subagent setup before a search — it resolves both role models, verifies credential presence, and optionally probes a completion call per role. Use it to diagnose model, provider, auth, timeout, or subagent dispatch failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness); it does not check models or subagent dispatch. When `autorag search` fails for a model/provider/subagent reason, the error output includes a hint pointing to `autorag health`.
+`autorag health` checks model/provider auth before a search — it resolves the model, verifies credential presence, and optionally probes one completion call. Use it to diagnose model, provider, auth, or timeout failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness). When `autorag search` fails for a model/provider reason, the error output includes a hint pointing to `autorag health`.
+
+`autorag ui` opens a loopback-only page (`127.0.0.1`) to connect local folders and datasource skills without editing JSON. It writes the same trusted `datasources` / `datasourceAccess` fields as a hand-edited config, stores env-var *names* rather than secrets, and refuses non-loopback binds. Use `--no-open` to print the URL without launching a browser.
+
+For a deliberately deployed UI, opt in explicitly in `config.json`. Keep the
+session token in the environment, set the public URL used by the browser, and
+allow only the frontend origins that should make credentialed API requests:
+
+```json
+{
+  "ui": {
+    "host": "0.0.0.0",
+    "port": 8787,
+    "allowRemote": true,
+    "publicOrigin": "https://autorag.example.com",
+    "corsOrigins": ["https://autorag.example.com"],
+    "tokenEnv": "AUTORAG_UI_TOKEN"
+  }
+}
+```
+
+Start it with `AUTORAG_UI_TOKEN` set to a random value of at least 16
+characters. Local use remains the safe default: omit `ui` (or leave
+`allowRemote` false) and AutoRAG binds to loopback, including a working
+`localhost` URL on systems that resolve it to IPv6.
 
 ## Installation
 
@@ -314,9 +429,45 @@ bun install -g @autorag/librarian   # autorag CLI
 bun add github:NomaDamas/AutoRAG-2.0
 ```
 
+PDF parsing requires **Java 11 or newer** because the bundled
+`@opendataloader/pdf` runtime is compiled for Java 11. This applies to
+Windows, macOS, and Linux. Verify the runtime that will be selected from
+`PATH` before indexing PDFs:
+
+```bash
+java -version
+```
+
+On Windows, ensure a Java 11+ installation appears before older Java
+installations in `PATH` (or set `JAVA_HOME` to the Java 11+ installation and
+start a new shell). Java 8 is not supported by the PDF parser.
+
 Git-based installs build `dist/` via the `prepare` script and require Bun on the installing machine.
 External tool binaries auto-install on first use into `<workspace>/.autorag/bin`: **MinSync** downloads a verified GitHub release asset (on by default; `minSync.autoInstall: false` to opt out), and **Jikji** compiles the [`jikji-cli`](https://crates.io/crates/jikji-cli) crate via cargo (requires the [Rust toolchain](https://rustup.rs); `jikji.autoInstall: false` to opt out). New `autorag init` configs enable Jikji find-first discovery by default. KakaoTalk (`katok`) and Discord (`discrawl`) stay manual, optional installs. All of them degrade gracefully when missing — core BM25 search works without any of them.
 
+### Experimental TUI (beta)
+
+The latest TUI work is currently available on the `feat/experimental-tui`
+branch. To use this beta version from source:
+
+```bash
+git clone --branch feat/experimental-tui https://github.com/Marker-Inc-Korea/AutoRAG.git
+cd AutoRAG
+bun install
+bun run build
+
+# Configure your document roots and model once:
+node dist/cli/index.js init --search-paths /path/to/documents
+
+# Start the beta TUI:
+node dist/cli/index.js tui
+```
+
+Inside the TUI, use `/resume` to browse saved sessions. Selecting a session
+replaces the current view with that session instead of mixing the two
+conversation histories. This branch is experimental and may change before the
+feature is included in a published release.
+
 ## Quick Start
 
 ```typescript
@@ -345,15 +496,11 @@ agent.recordFeedbackByNumbers(response.sessionId, [1, 3], [2]);
            │
            ▼
     ┌──────────────┐
-    │ Plan + seed  │ ← parent check_memory, Jikji, and retrieval tools
-    └──────┬───────┘
-           ▼
-    ┌──────────────┐
-    │ Delegate     │ ← pi-subagents → gpt-5.6-luna explorer
+    │ Plan + search│ ← check_memory, Jikji, and retrieval tools
     └──────┬───────┘
            ▼
     ┌──────────────┐
-    │ Search + read│ ← explorer read/grep/find/ls
+    │ Direct read  │ ← bash find/grep/cat
     └──────┬──────┘
            ▼
     ┌─────────────┐

package/dist/agent/agent.d.ts

@@ -2,15 +2,16 @@ import { Agent, type AgentTool, type Skill } from "@earendil-works/pi-agent-core
 import type { Api, Model } from "@earendil-works/pi-ai";
 import { type DatasourceAccessContextOptions } from "../datasource/access-context.js";
 import type { DatasourceIndexResult, DatasourceSkill } from "../datasource/types.js";
+import { type DupeyCliOptions } from "../dupey/index.js";
 import { type JikjiFailureReason, type JikjiOptions } from "../jikji/index.js";
 import type { ResultFeedback } from "../memory/memory.js";
-import { type MinSyncSyncResult, type MinSyncVectorMethodOptions } from "../minsync/index.js";
+import { type MinSyncBM25MethodOptions, type MinSyncSyncResult, type MinSyncVectorMethodOptions } from "../minsync/index.js";
 import { type ParsedMirrorSyncResult } from "../mirror/sync.js";
 import type { DefaultParserRegistryOptions } from "../parser/index.js";
-import { type BM25MethodOptions, type BM25SyncResult } from "../retrieval/methods/bm25.js";
+import { type BM25SyncResult } from "../retrieval/methods/bm25.js";
 import { RetrievalMethodRegistry } from "../retrieval/registry.js";
 import type { CuratedResult, RetrievalDiagnostic, RetrievalOptions, RetrievalResult } from "../retrieval/types.js";
-import { type MandatorySubagentSessionOptions } from "../subagents/runtime.js";
+import { type ScanDuplicateDocumentsDetails } from "./dupey-tool.js";
 import { type JikjiFindProviderResult } from "./jikji-find-tool.js";
 import { type SearchDocumentDiagnostic, type SearchDocumentsResponse } from "./search-documents.js";
 import { type WatcherFactory, type WatchRefreshHandle } from "./watch-refresh.js";
@@ -29,7 +30,8 @@ export interface AutoRAGMinSyncRefreshResult {
     readonly synced: number;
     readonly reason?: string;
 }
-export interface AutoRAGRefreshResult extends ParsedMirrorSyncResult {
+export interface AutoRAGRefreshResult extends Omit<ParsedMirrorSyncResult, "diagnostics"> {
+    readonly diagnostics: readonly SearchDocumentDiagnostic[];
     readonly bm25?: BM25SyncResult;
     readonly minsync?: AutoRAGMinSyncRefreshResult;
     readonly datasources?: readonly DatasourceIndexResult[];
@@ -68,7 +70,6 @@ export interface AutoRAGWatchRefreshOptions {
 export type AutoRAGWatchRefreshHandle = WatchRefreshHandle;
 export interface AutoRAGAgentOptions {
     model?: Model<Api>;
-    explorerModel?: Model<Api>;
     apiKey?: string;
     providerApiKeys?: Readonly<Record<string, string>>;
     searchPaths: string[];
@@ -77,13 +78,20 @@ export interface AutoRAGAgentOptions {
     workspacePath?: string;
     tools?: AgentTool[];
     minSync?: Omit<MinSyncVectorMethodOptions, "root"> | false;
-    bm25?: Omit<BM25MethodOptions, "root"> | false;
-    jikji?: JikjiOptions;
+    bm25?: Omit<MinSyncBM25MethodOptions, "root"> | false;
+    jikji?: JikjiOptions | false;
     autoRefresh?: AutoRefreshOptions;
     parserOptions?: DefaultParserRegistryOptions;
+    dupey?: DupeyCliOptions | false;
+    excludeExactDuplicates?: boolean;
     datasourceSkills?: readonly DatasourceSkill[];
     datasourceAccess?: DatasourceAccessContextOptions;
-    sessionFactory?: AutoRAGSessionFactory;
+    /** Non-fatal diagnostics from config/agent construction (e.g. skipped unknown datasources). */
+    startupDiagnostics?: readonly SearchDocumentDiagnostic[];
+    /** Maximum time a model/tool search may run before it is aborted. */
+    searchTimeoutMs?: number;
+    /** Maximum number of retrieval/tool executions allowed in one search. */
+    maxSearchToolCalls?: number;
 }
 export interface AutoRAGSearchSession {
     readonly agent: Agent;
@@ -91,7 +99,6 @@ export interface AutoRAGSearchSession {
     abort(): Promise<void> | void;
     dispose(): void;
 }
-export type AutoRAGSessionFactory = (options: MandatorySubagentSessionOptions) => Promise<AutoRAGSearchSession>;
 export type AutoRAGJikjiPrepareResult = {
     readonly ok: true;
     readonly code: number;
@@ -106,17 +113,10 @@ export declare class AutoRAGAgent {
     private readonly innerAgent;
     private readonly tools;
     private readonly configuredModel;
-    private readonly configuredExplorerModel;
     private readonly apiKey;
     private readonly providerApiKeys;
-    private readonly sessionFactory;
     private readonly listeners;
     private activeSession;
-    private readonly pendingSubagentCalls;
-    private readonly dispatchSequenceBySession;
-    private readonly dispatchArgsTracker;
-    private dispatchSyntheticCounter;
-    private successfulExplorerCalls;
     private readonly memory;
     private readonly runLogger;
     private lastQuery;
@@ -131,7 +131,6 @@ export declare class AutoRAGAgent {
     private readonly configuredSearchPaths;
     private retrievalScopeBindings;
     private readonly datasourceVirtualScopePrefixes;
-    private readonly allowedExplorerRoots;
     private readonly workspaceProjectRoot;
     private readonly methodRegistry;
     private readonly retriever;
@@ -142,25 +141,23 @@ export declare class AutoRAGAgent {
     private readonly jikjiClient;
     private readonly datasourceSkills;
     private readonly datasourceAccessOptions;
+    private readonly startupDiagnostics;
     private readonly datasourceAgentSkills;
     private readonly parserOptions;
+    private readonly dupeyOptions;
+    private readonly excludeExactDuplicates;
     private readonly baseSystemPromptConfig;
-    /** Run-scoped merged Jikji policy, set during searchDocuments; cleared after. */
-    private activeJikjiPolicy;
-    private activeExplorerModel;
-    /** Run-scoped jikji_find call count for the two-phase raw-fallback gate. */
-    private jikjiFindCallCount;
     private readonly droppedCallerToolNames;
+    private readonly searchTimeoutMs;
+    private readonly maxSearchToolCalls;
+    private searchToolCallCount;
     constructor(options: AutoRAGAgentOptions);
+    scanDuplicateDocuments(): Promise<ScanDuplicateDocumentsDetails>;
     private withMemoryContext;
     private resolveSessionModel;
+    private createSearchSession;
     private configureSearchSession;
     private recordSearchToolEvent;
-    private nextDispatchSequence;
-    private resolveToolCallId;
-    private emitDispatchAutofilled;
-    private emitDispatchRejected;
-    private emitDispatchRejectedFromRejection;
     private currentSystemPromptConfig;
     subscribe(listener: Parameters<Agent["subscribe"]>[0]): () => void;
     abort(): void;
@@ -177,6 +174,7 @@ export declare class AutoRAGAgent {
     submitFeedback(sessionId: string | undefined, satisfied: boolean): void;
     recordResultFeedback(feedback: ResultFeedback[]): void;
     recordFeedbackByNumbers(sessionId: string, usefulNumbers: number[], notUsefulNumbers?: number[]): void;
+    recordFeedbackByIds(usefulFeedbackIds: readonly string[], notUsefulFeedbackIds?: readonly string[]): void;
     getResultRegistry(sessionId?: string): ReadonlyMap<number, CuratedResult>;
     searchDocuments(query: string, options?: RetrievalOptions): Promise<SearchDocumentsResponse>;
     private datasourceAccessContext;
@@ -195,7 +193,8 @@ export declare class AutoRAGAgent {
     private indexDatasources;
     /** Path-opaque component diagnostics (e.g. BM25 readiness) for the search response. */
     private collectComponentDiagnostics;
-    buildSearchPrompt(query: string, options: RetrievalOptions, explorerModel?: string): string;
+    private prefetchInitialRetrievalContext;
+    buildSearchPrompt(query: string, options: RetrievalOptions, initialRetrievalContext?: string): string;
     refresh(force?: boolean, opts?: AutoRAGRefreshOptions): Promise<AutoRAGRefreshResult>;
     /**
      * Path-opaque snapshot of corpus freshness and the last refresh outcome. Runs
@@ -221,7 +220,8 @@ export declare class AutoRAGAgent {
      * so the refresh result and status remain consistent.
      */
     private scanMirrorStaleness;
-    syncBM25(): Promise<BM25SyncResult | undefined>;
+    private exactDuplicateExclusions;
+    syncBM25(minsync?: MinSyncSyncResult): Promise<BM25SyncResult | undefined>;
     syncMinSync(): Promise<MinSyncSyncResult | undefined>;
     prepareJikji(): Promise<readonly AutoRAGJikjiPrepareResult[] | undefined>;
     private executeJikjiPrepare;
@@ -229,12 +229,9 @@ export declare class AutoRAGAgent {
     /**
      * Provider method for the jikji_find tool. Runs JikjiClient.find over all
      * configured search roots, normalizes answer paths against planned source
-     * roots, merges per-root answer packs using least-privilege (restrictive-
-     * wins) semantics, and sets the run-scoped activeJikjiPolicy. Only mutates
-     * run state (activeJikjiPolicy / jikjiFindCallCount) when a searchDocuments
-     * run is active; direct out-of-run calls compute and return the result
-     * without persisting. When jikji is unavailable or all roots fail, returns
-     * an unavailable result and leaves the policy undefined (bash is allowed).
+     * roots, and merges per-root answer packs using least-privilege
+     * (restrictive-wins) semantics. Jikji policy metadata remains visible to the
+     * model, but it does not gate the librarian's direct file-reading tools.
      */
     findJikji(query: string, opts?: {
         readonly topK?: number;
@@ -253,20 +250,9 @@ export declare class AutoRAGAgent {
      * - stopAfterFind: OR
      * - agentShouldNotRerank: OR
      * - handoffAction: MOST RESTRICTIVE (direct_use < jikji_retry < raw_fallback_after_retry)
-     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry AND callCount>=2
+     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry
      */
     private mergePolicy;
-    /**
-     * Deny-by-default bash gate. When jikji is configured but no jikji_find has
-     * run yet this run (count===0), bash is blocked so the agent discovers local
-     * files via jikji_find first. After a find, the run-scoped activeJikjiPolicy
-     * applies: when no policy is active, bash behaves exactly as before
-     * (allowed). Under an active policy, bash is denied unless handoffAction is
-     * raw_fallback_after_retry AND rawFallbackAllowed is true (after a second
-     * jikji_find). When jikji is not configured, the find-first branch is
-     * skipped entirely (bash unchanged).
-     */
-    private bashGate;
     /**
      * Programmatic retrieval across all registered methods, merged via min-max
      * normalization + source dedup. Activates the RetrievalMethodRegistry /
@@ -300,6 +286,7 @@ export declare class AutoRAGAgent {
     /** The retrieval method registry (posix active; vector/bm25/hybrid pluggable). */
     getMethodRegistry(): RetrievalMethodRegistry;
     getSystemPrompt(): string;
+    private rerankWithMemory;
     private normalizeRetrievalOptions;
     private resolveRetrievalScope;
 }

package/dist/agent/bash-tool.d.ts

@@ -13,24 +13,15 @@ export interface BashToolOptions {
     readonly timeoutMs?: number;
     /** Maximum captured output bytes before truncation. */
     readonly maxOutputBytes?: number;
-    /**
-     * Optional deny-by-default gate. When present and returns
-     * `{ allowed: false }`, the command is NOT spawned; the tool returns the
-     * gate message with `blockedByJikjiPolicy: true`. Used by the Jikji policy
-     * to enforce stop_after_find / direct_use / forbidden bash restrictions.
-     */
-    readonly gate?: () => {
-        allowed: boolean;
-        message: string;
-    };
 }
 export interface BashToolDetails {
     readonly method: "bash";
     readonly command: string;
+    readonly aborted: boolean;
     readonly exitCode: number | undefined;
+    readonly terminationFailed: boolean;
     readonly timedOut: boolean;
     readonly truncated: boolean;
-    readonly blockedByJikjiPolicy?: boolean;
 }
 /**
  * Real shell access for the librarian agent. AutoRAG navigates and reads the