@autorag/librarian 2.1.0 → 2.3.0

package/README.md

@@ -10,7 +10,11 @@
 
 AutoRAG searches your PDFs, wikis, notes, research papers, and knowledge bases — then curates the results into clean, numbered knowledge units. No raw grep dumps. Just answers.
 
-AutoRAG is a customized [Pi](https://github.com/earendil-works/pi-mono) agent — the Pi agent loop configured into a librarian. Searches use a two-tier workflow: a parent orchestrator delegates exploration to explorer agents. The roles and providers are independently configured from the models available in the user's authenticated runtime; AutoRAG does not ship a private provider default.
+AutoRAG is a customized [Pi](https://github.com/earendil-works/pi-mono) agent — the Pi agent loop configured into a librarian. The AutoRAG librarian retrieves candidates, reads source files directly, judges the evidence, and curates the structured answer. Its model and provider come from the user's authenticated runtime; AutoRAG does not ship a private provider default.
+
+AutoRAG itself is the specialized search agent, not a coordinator for other
+model roles. You configure one model, and that model owns the complete
+retrieval, reading, judgment, and curation loop.
 
 ## Why AutoRAG
 
@@ -61,31 +65,19 @@ Different documents need different search strategies:
 | Legal documents, specifications | BM25 (keyword ranking) | Handles domain terminology well |
 | Mixed collections | Hybrid (vector + BM25) | Combines precision and recall |
 
-AutoRAG supports **pluggable retrieval methods**. It ships with lexical **BM25** and semantic **MinSync** methods wired through the `RetrievalMethodRegistry`, and the architecture is ready for additional vector and hybrid backends. The parent orchestrator owns process-bound retrieval tools and gives bounded seed packs to explorers; explorers use read-only `read`/`grep`/`find`/`ls` tools to inspect the underlying documents. The `ResultMerger` handles cross-method score normalization and deduplication — you get one unified result set regardless of how many methods contributed.
+AutoRAG supports **pluggable retrieval methods**. It ships with lexical **BM25** and semantic **MinSync** methods wired through the `RetrievalMethodRegistry`, and the architecture is ready for additional vector and hybrid backends. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication.
 
 BM25 and MinSync are **enabled by default** — no explicit configuration is needed for standard lexical + semantic retrieval. Both can be disabled by setting `"bm25": false` or `"minSync": false` in the config file. MinSync uses a pre-installed binary (`autoInstall: false`); configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints. AutoRAG never forces TEI or any external embedding service.
 
 ### Real directory access
 
-AutoRAG reads configured source directories through delegated explorer tasks. Each explorer is assigned exactly one normalized configured search root as its `cwd`; the top-level `subagent` invocation sets `agentScope: "user"` and `artifacts: false` exactly once for single, `tasks`, `chain`, or `parallel` dispatch, and nested explorer task items omit both fields. Project-local `.pi-subagents` debug artifacts are disabled. Explorers use read-only `read`/`grep`/`find`/`ls`; the parent orchestrator owns retrieval seed tools and must delegate document reading before curating. Curated answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. BM25 and MinSync index parsed markdown mirrors under `.autorag`.
-
-Each explorer `task` contains an Assignment V1 block — a sentinel-wrapped JSON body with `originalQuery` (the caller query verbatim), `method` (the selected retrieval path), and `queryVariants` (a nonempty array), followed by canonical role lines requiring `retrievedAt` and temporal metadata:
-
-```text
-<<<AUTORAG_ASSIGNMENT_V1>>>
-{"originalQuery":"<caller query verbatim>","method":"<selected retrieval method>","queryVariants":["<variant 1>","<variant 2>"]}
-<<<END_AUTORAG_ASSIGNMENT_V1>>>
-Required handoff: include retrievedAt.
-Required handoff: include temporal metadata.
-```
-
-A legacy labeled format (`Original query:`, `Selected retrieval method:`, `Query variants:`) is accepted for compatibility. Missing or null top-level `artifacts`, `agentScope`, and leaf `model` fields are safely autofilled before validation; explicit wrong values (`artifacts: true`, `agentScope: "project"`, a non-configured model) remain rejected. There is no single-agent fallback. See [docs/subagent-orchestration.md](docs/subagent-orchestration.md) for the full dispatch contract, templates, anti-examples, and stable coded dispatch errors.
+AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. BM25 and MinSync index parsed markdown mirrors under `.autorag`.
 
 ### Optional Jikji discovery and indexing
 
 AutoRAG can opt into [Jikji](https://github.com/NomaDamas/jikji) as a local CLI-backed **find-first discovery and indexing** layer. Jikji is optional: AutoRAG does not vendor it, install it, or register it as a retrieval backend when enabled.
 
-When Jikji is configured, AutoRAG calls `jikji find ROOT "query" --json` via a policy-aware `jikji_find` tool as the first local-discovery action. The tool parses and validates the upstream answer-pack and honors its `handoff_action` (`direct_use` / `jikji_retry` / `raw_fallback_after_retry`), `tool_call_policy` (`stop_after_find`, `forbidden_tools`, `allowed_followups`), and `agent_should_not_rerank`. Explorer `read`/`grep`/`find`/`ls` discovery is the fallback only when the answer-pack permits raw fallback (`raw_fallback_after_retry`, after the required retry) or when Jikji is unavailable/unconfigured. `prepare`/`refresh` remain for indexing only and do not answer queries directly.
+When Jikji is configured, AutoRAG calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly.
 
 Programmatic use:
 
@@ -97,6 +89,35 @@ const agent = new AutoRAGAgent({
 await agent.prepareJikji();
 ```
 
+### Duplicate document management with dupey
+
+AutoRAG can use the external [`dupey`](https://github.com/NomaDamas/dupey) CLI
+to detect exact, near, and containment document families.
+
+```bash
+autorag duplicates /path/to/documents
+autorag duplicates --json
+```
+
+The command is read-only: it reports exact duplicate groups and review
+guidance, but never moves or deletes source files. The `scan_duplicate_documents`
+Agent tool exposes the same read-only scan to the orchestrator.
+
+Exact duplicate exclusion is enabled by default during parsed-mirror refresh.
+For each exact canonical-text hash, the newest filesystem copy is indexed and
+older copies are omitted. Disable it in `config.json` when both copies must be
+searchable:
+
+```json
+{
+  "dupey": { "enabled": true },
+  "excludeExactDuplicates": false
+}
+```
+
+If `dupey` is not installed or fails, refresh continues without exclusion and
+reports no destructive action; install it with `cargo install dupey`.
+
 The same `.autorag/jikji.json` shape configures Jikji when present:
 
 ```json
@@ -122,6 +143,14 @@ The upstream Rust `PrepareArgs` defines reference defaults that AutoRAG does not
 
 Datasource skills let AutoRAG search external, server-configured data sources through the same retrieval pipeline as local documents. A skill describes what it indexes, how it should be refreshed, what source instances exist, and which permission tags/scopes bound access. Retrieval still flows through `RetrievalMethodRegistry` → `ParallelRetriever` → datasource result filtering → `ResultMerger`; datasource skills do not create a parallel search path.
 
+Every datasource can be registered multiple times through a connection alias:
+use the config key as the unique name and set `type` to the reusable backend
+(`gmail`, `github`, `slack`, `discord`, `kakao`, `cloud-drive`, and so on).
+Each alias becomes an independently loadable agent skill with its own source
+scope and workspace namespace. Chat aliases search all channels by default;
+trusted `channels.ids` / `channels.names` allowlists can expose a particular
+channel or group chat as its own datasource.
+
 Security defaults are intentionally strict:
 
 - datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags` and `datasourceAccess.allowedScopes`;
@@ -133,37 +162,66 @@ Security defaults are intentionally strict:
 | Datasource | Skill | Connects via | Notes |
 |---|---|---|---|
 | KakaoTalk | `katok` | external [`katok`](https://github.com/NomaDamas/katok) CLI | first datasource skill; AutoRAG never reads KakaoTalk databases directly |
-| Slack | `slack` | Slack Web API (bot token) | workspace/channel history; per-channel scope failures degrade to warnings |
-| Discord | `discord` | Discord REST v10 (bot token) | guild/channel messages; Hangul channel names work as scopes |
-| Notion | `notion` | Notion API (integration token) | pages/databases shared with the integration; block-tree text |
+| WhatsApp | `whatsapp` | external [`wacrawl`](https://github.com/openclaw/wacrawl) CLI | local-first incremental archive + FTS5 search; live desktop ingestion is macOS-only |
+| Telegram | `telegram` | external [`telecrawl`](https://github.com/openclaw/telecrawl) CLI | local-first archive + FTS5 search; live desktop ingestion is macOS-only |
+| Slack | `slack` | external [`slacrawl`](https://github.com/openclaw/slacrawl) CLI | local-first workspace/channel/thread archive + FTS5 search |
+| Discord | `discord` | external [`discrawl`](https://github.com/openclaw/discrawl) CLI | guild/channel/thread/DM archive; FTS5 + semantic + hybrid retrieval, incremental sync |
+| Notion | `notion` | external [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | local-first page/database/block archive + FTS5 search |
 | GitHub Issues/PRs | `github` | GitHub REST (token optional) | issues + PR bodies per `owner/repo`; public repos work unauthenticated |
-| Google Drive | `gdrive` | Drive REST v3, or **[`rclone`](https://rclone.org) CLI** (`backend: "rclone"`) | Docs/Sheets exported as text; the rclone backend also opens any of rclone's 70+ remotes |
+| Cloud drives | `cloud-drive` | **[`rclone`](https://rclone.org) CLI** | Incremental Google Drive Tier-1; OneDrive/network remotes; iCloud experimental |
 | Gmail / IMAP | `gmail` | Gmail REST v1, or **[`himalaya`](https://pimalaya.org) CLI** (`backend: "himalaya"`) | the himalaya backend indexes any IMAP/Maildir account it has configured — no OAuth plumbing |
 | Local mail exports | `mail-export` | filesystem (`.mbox` / `.eml`) | classic `From_` splitting, mailparser-based; count-only warnings |
-| Obsidian vault | `obsidian` | filesystem (markdown) | frontmatter/inline tags, wiki links `[[...]]`, embeds `![[...]]` |
+| Obsidian vault | `obsidian` | external [`qmd`](https://github.com/tobi/qmd) CLI | incremental `qmd update`, BM25 `qmd search`, semantic `qmd vsearch`; vault path via `connector.vaultPath` |
 | RSS / news | `rss` | HTTP feed polling | RSS 2.0 + Atom, feed/category hierarchy, 24h dedupe window |
 
-All of them share one framework: a trusted connector fetches documents at refresh time, chunks persist under `<workspace>/.autorag/datasources/<skill>/<instance>/`, and queries run against a local BM25 lexical index (Korean-aware prefix matching) through `search_datasource_documents`. Tokens are referenced by environment variable name only (e.g. `SLACK_BOT_TOKEN`), never stored in config. Auth/permission/rate-limit failures surface as path/PII-opaque diagnostics. See [docs/manual-qa-datasources.md](docs/manual-qa-datasources.md) for the QA harnesses.
+Connector-backed skills fetch documents into AutoRAG's local chunk store. External-crawler skills such as KakaoTalk, WhatsApp, Telegram, Slack, and Notion leave incremental archive and FTS ownership with their CLI and map query results into the same retrieval pipeline. Obsidian uses the external `qmd` CLI (incremental update + BM25 + semantic). Tokens are referenced by environment variable name only, never stored in config. Process/API failures surface as path/PII-opaque diagnostics. See [docs/manual-qa-datasources.md](docs/manual-qa-datasources.md) for the QA harnesses.
 
 Configure them in `config.json` (CLI) or pass `datasourceSkills` programmatically:
 
 ```jsonc
 {
   "datasources": {
-    "slack":    { "connector": { "tokenEnv": "SLACK_BOT_TOKEN" } },
+    "whatsapp": { "instanceId": "personal", "connector": { "binaryPath": "wacrawl" } },
+    "telegram": { "instanceId": "personal", "connector": { "binaryPath": "telecrawl" } },
+    "slack":    { "connector": { "binaryPath": "slacrawl", "configPath": "/path/to/slacrawl.yaml", "syncSource": "primary" } },
+    "notion":   { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } },
     "github":   { "connector": { "repos": ["owner/repo"] } },
     "gmail":    { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
-    "gdrive":   { "connector": { "backend": "rclone", "remote": "gdrive:" } },
+    "personal-google-drive": { "type": "cloud-drive", "instanceId": "personal", "connector": { "provider": "google-drive", "remote": "personal-gdrive:", "include": ["**/*.md"] } },
+    "company-onedrive": { "type": "cloud-drive", "instanceId": "work", "connector": { "provider": "onedrive", "remote": "company-onedrive:Documents" } },
     "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
     "rss":      { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
   },
   "datasourceAccess": {
-    "allowedTags": ["slack", "github", "gmail", "gdrive", "obsidian", "rss"],
-    "allowedScopes": ["/slack/**", "/github/**", "/gmail/**", "/gdrive/**", "/obsidian/**", "/rss/**"]
+    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "cloud-drive", "obsidian", "rss"],
+    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
   }
 }
 ```
 
+Install wacrawl with `brew install openclaw/tap/wacrawl`. AutoRAG invokes `wacrawl sync` during datasource refresh and `wacrawl --json --sync never search` during retrieval. Optional trusted connector fields are `binaryPath`, `databasePath`, and `sourcePath`. The child process receives only a restricted environment; unrelated model/provider secrets are not forwarded. Live WhatsApp Desktop discovery requires macOS and the permissions documented by wacrawl, while an existing portable archive can be queried on other supported platforms.
+
+Install telecrawl with `brew install openclaw/tap/telecrawl`. AutoRAG invokes `telecrawl import` during datasource refresh and `telecrawl --json search` during retrieval. It uses the same optional trusted connector fields and restricted child environment as wacrawl. Live Telegram Desktop discovery requires macOS and the permissions documented by telecrawl, while an existing portable archive can be queried on other supported platforms.
+
+Install slacrawl with `brew install openclaw/tap/slacrawl`. AutoRAG invokes `slacrawl sync` during datasource refresh and `slacrawl --json search` during retrieval. Optional trusted connector fields are `binaryPath`, `configPath`, and `syncSource`. Slack credentials and source definitions remain in slacrawl's own configuration rather than AutoRAG.
+
+Install notcrawl with `brew install openclaw/tap/notcrawl`. AutoRAG invokes `notcrawl sync` during datasource refresh and `notcrawl search --json` during retrieval. Optional trusted connector fields are `binaryPath` and `configPath`. Notion credentials and workspace definitions remain in notcrawl's own configuration rather than AutoRAG.
+
+Install and authenticate rclone separately (`brew install rclone && rclone
+config` on macOS), then configure the provider-neutral `cloud-drive` skill.
+`cloud-drive` is a reusable `type`: each datasource config key is a connection
+alias and becomes a separate agent skill and scope. Multiple Google accounts,
+or Google Drive plus OneDrive/iCloud, can therefore be loaded and searched
+independently.
+AutoRAG inventories with `rclone lsjson`, keeps a workspace-local manifest and
+managed mirror, and downloads only added or changed indexable files. Google
+Drive is Tier-1. OneDrive, Dropbox, SMB/SFTP/WebDAV, and mounted drives share
+the same manifest contract. iCloud Drive is experimental because rclone marks
+that backend Tier 4 and Apple ID/2FA sessions periodically require
+reauthentication. See [docs/datasource-skills.md](docs/datasource-skills.md)
+for filtering, size, concurrency, bandwidth, dry-run, and agent tool-calling
+details.
+
 #### KakaoTalk (katok)
 
 KakaoTalk was the first datasource skill. It uses the external [`katok`](https://github.com/NomaDamas/katok) CLI only — AutoRAG never reads KakaoTalk databases directly. `katok` failures return diagnostics instead of throwing, and remote embedding egress configuration is rejected before the CLI is spawned.
@@ -190,19 +248,64 @@ await agent.refresh(); // refreshes parsed mirrors, BM25/MinSync, and datasource
 const results = await agent.searchDatasourceDocuments("meeting with Mina", { topK: 5 });
 ```
 
-A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as `/kakao/personal/chunks/<chunk-id>`, and permission tags that match your server-side access policy.
+#### Discord (discrawl)
 
-### Primary target: document collections
+Discord uses the external [`discrawl`](https://github.com/openclaw/discrawl) CLI, which owns the SQLite archive, the FTS5 index, and the message vectors. AutoRAG never calls the Discord API itself.
 
-AutoRAG is built for **non-code document retrieval**: manuals, legal docs, internal wikis, meeting notes, research literature, knowledge bases, PDFs.
+```bash
+brew install openclaw/tap/discrawl
+```
 
-Code repositories work too (the explorer's `grep` is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
+Two archive sources are supported. `wiretap` (the default) reads the local Discord Desktop cache and needs **no token at all**; `discord` uses a bot token, which is the ToS-sanctioned automation path. AutoRAG refuses to spawn the CLI when a Discord *user* token is present in the environment — automating a user account violates Discord's Community Guidelines and can get the account terminated.
 
-#### Legacy HWP5 parsing
+```typescript
+import { AutoRAGAgent, DiscrawlClient, DiscrawlSkill } from "@autorag/librarian";
+
+const discord = new DiscrawlSkill({
+  client: new DiscrawlClient({ source: "wiretap", root: process.cwd() }),
+  instanceId: "community",
+});
+
+const agent = new AutoRAGAgent({
+  searchPaths: ["/path/to/documents"],
+  datasourceSkills: [discord],
+  datasourceAccess: {
+    allowedTags: ["discord"],
+    allowedScopes: ["/discord/community/**"],
+  },
+});
+```
+
+Or through the trusted config factory:
+
+```json
+{
+  "datasources": {
+    "discord": {
+      "instanceId": "community",
+      "connector": {
+        "source": "wiretap",
+        "embeddingProvider": "ollama",
+        "embeddingModel": "embeddinggemma",
+        "defaultMode": "hybrid"
+      }
+    }
+  }
+}
+```
+
+Two defaults are deliberate and worth keeping:
 
-AutoRAG supports searchable Markdown extraction from legacy binary `.hwp` (HWP5) files for body paragraphs and top-level table cells. Extraction is powered by the MIT-licensed [`rhwp`](https://github.com/edwardkim/rhwp) WebAssembly parser.
+- **`defaultMode: "hybrid"`** — discrawl's FTS index strips newlines without substituting a space, welding words across line breaks into a single unsearchable token (measured at ~47% of post-newline words on a real archive). Semantic recall covers that gap. See [#1413](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1413).
+- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). For workspace-managed discrawl state, AutoRAG writes these values to `.autorag/datasources/discrawl/config.toml` while preserving unrelated discrawl settings. An explicit `connector.configPath` remains operator-owned and is never rewritten. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
 
-This is scoped text extraction, not full-fidelity document conversion. Headers, footers, notes, text boxes, nested tables, images, equations, and OCR are explicitly deferred and are not currently extracted from HWP5 files.
+A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as `/kakao/personal/chunks/<chunk-id>`, and permission tags that match your server-side access policy.
+
+### Primary target: document collections
+
+AutoRAG is built for **non-code document retrieval**: manuals, legal docs, internal wikis, meeting notes, research literature, knowledge bases, PDFs.
+
+Code repositories work too (direct `grep` is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
 
 ## Configuration and state
 
@@ -212,38 +315,39 @@ The default home state is kept outside the workspace:
 ~/.autorag/
 ├── config.json
 ├── memory.json
-├── logs/
-│   └── runs.jsonl
-└── pi-agent/
-    ├── auth.json
-    ├── models.json
-    ├── settings.json
-    └── sessions/
+└── logs/
+    └── runs.jsonl
 ```
 
-`config.json` selects sources, the workspace, memory path, retrieval settings, and the two agent models. Configure the roles independently with `agents.orchestrator` and `agents.explorer`. Provider and model IDs must refer to models available in the user's authenticated runtime:
+`config.json` selects sources, the workspace, memory path, retrieval settings, and the agent model. Provider and model IDs must refer to a model available in the user's authenticated runtime:
 
 ```json
 {
   "searchPaths": ["/path/to/documents"],
   "workspacePath": "/path/to/workspace",
   "memoryPath": "/Users/you/.autorag/memory.json",
-  "agents": {
-    "orchestrator": { "provider": "provider-name", "id": "reasoning-model" },
-    "explorer": { "provider": "provider-name", "id": "exploration-model" }
-  }
+  "model": { "provider": "provider-name", "id": "reasoning-model" }
 }
 ```
 
-`autorag init` leaves `agents` unset when no role-model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure both roles explicitly.
+`autorag init` leaves `model` unset when no model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure the model explicitly.
+
+For fast interactive search, prefer a model with reliable tool calling, high
+output TPS, and low first-token latency. A query can require several short
+model turns while AutoRAG alternates between retrieval tools and direct source
+reading, so model throughput has a visible effect on end-to-end response time.
+It does not accelerate BM25, MinSync, Jikji, filesystem access, or indexing
+itself. Larger reasoning models remain useful for difficult synthesis,
+conflicting evidence, and specialized domain judgment, but they are not a
+requirement for ordinary retrieval.
 
 Config path precedence is `--config` > `AUTORAG_CONFIG` > `~/.autorag/config.json`. When the home config is absent and `<cwd>/autorag.config.json` exists, AutoRAG copies the legacy file to `~/.autorag/config.json` without deleting or modifying the legacy file. The legacy cwd file is a migration source, not the default location.
 
-`memory.json` stores retrieval memory, `logs/runs.jsonl` records run events, and durable Pi models, settings, and sessions stay under `~/.autorag/pi-agent`. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under `<workspace>/.autorag`.
+`memory.json` stores retrieval memory and `logs/runs.jsonl` records run events. Model authentication remains with the user's configured provider or authenticated local runtime. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under `<workspace>/.autorag`.
 
 `autorag refresh` and `autorag index reset|rebuild` accept `--method <csv>` (e.g. `--method bm25,minsync,parsed`) to scope which indexing methods run or which index directories are removed. When omitted, all methods run. `autorag init` accepts `--embedder-*` flags to configure the MinSync embedder endpoint in the config file.
 
-`autorag health` checks model/provider auth and explorer subagent setup before a search — it resolves both role models, verifies credential presence, and optionally probes a completion call per role. Use it to diagnose model, provider, auth, timeout, or subagent dispatch failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness); it does not check models or subagent dispatch. When `autorag search` fails for a model/provider/subagent reason, the error output includes a hint pointing to `autorag health`.
+`autorag health` checks model/provider auth before a search — it resolves the model, verifies credential presence, and optionally probes one completion call. Use it to diagnose model, provider, auth, or timeout failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness). When `autorag search` fails for a model/provider reason, the error output includes a hint pointing to `autorag health`.
 
 ## Installation
 
@@ -257,7 +361,7 @@ bun add github:NomaDamas/AutoRAG-2.0
 ```
 
 Git-based installs build `dist/` via the `prepare` script and require Bun on the installing machine.
-External tool binaries auto-install on first use into `<workspace>/.autorag/bin`: **MinSync** downloads a verified GitHub release asset (on by default; `minSync.autoInstall: false` to opt out), and **Jikji** compiles the [`jikji-cli`](https://crates.io/crates/jikji-cli) crate via cargo (requires the [Rust toolchain](https://rustup.rs); `jikji.autoInstall: false` to opt out). New `autorag init` configs enable Jikji find-first discovery by default. KakaoTalk (`katok`) stays a manual, optional install. All of them degrade gracefully when missing — core BM25 search works without any of them.
+External tool binaries auto-install on first use into `<workspace>/.autorag/bin`: **MinSync** downloads a verified GitHub release asset (on by default; `minSync.autoInstall: false` to opt out), and **Jikji** compiles the [`jikji-cli`](https://crates.io/crates/jikji-cli) crate via cargo (requires the [Rust toolchain](https://rustup.rs); `jikji.autoInstall: false` to opt out). New `autorag init` configs enable Jikji find-first discovery by default. KakaoTalk (`katok`) and Discord (`discrawl`) stay manual, optional installs. All of them degrade gracefully when missing — core BM25 search works without any of them.
 
 ## Quick Start
 
@@ -287,15 +391,11 @@ agent.recordFeedbackByNumbers(response.sessionId, [1, 3], [2]);
            │
            ▼
     ┌──────────────┐
-    │ Plan + seed  │ ← parent check_memory, Jikji, and retrieval tools
-    └──────┬───────┘
-           ▼
-    ┌──────────────┐
-    │ Delegate     │ ← pi-subagents → gpt-5.6-luna explorer
+    │ Plan + search│ ← check_memory, Jikji, and retrieval tools
     └──────┬───────┘
            ▼
     ┌──────────────┐
-    │ Search + read│ ← explorer read/grep/find/ls
+    │ Direct read  │ ← bash find/grep/cat
     └──────┬──────┘
            ▼
     ┌─────────────┐

package/dist/agent/agent.d.ts

@@ -2,6 +2,7 @@ import { Agent, type AgentTool, type Skill } from "@earendil-works/pi-agent-core
 import type { Api, Model } from "@earendil-works/pi-ai";
 import { type DatasourceAccessContextOptions } from "../datasource/access-context.js";
 import type { DatasourceIndexResult, DatasourceSkill } from "../datasource/types.js";
+import { type DupeyCliOptions } from "../dupey/index.js";
 import { type JikjiFailureReason, type JikjiOptions } from "../jikji/index.js";
 import type { ResultFeedback } from "../memory/memory.js";
 import { type MinSyncSyncResult, type MinSyncVectorMethodOptions } from "../minsync/index.js";
@@ -10,7 +11,7 @@ import type { DefaultParserRegistryOptions } from "../parser/index.js";
 import { type BM25MethodOptions, type BM25SyncResult } from "../retrieval/methods/bm25.js";
 import { RetrievalMethodRegistry } from "../retrieval/registry.js";
 import type { CuratedResult, RetrievalDiagnostic, RetrievalOptions, RetrievalResult } from "../retrieval/types.js";
-import { type MandatorySubagentSessionOptions } from "../subagents/runtime.js";
+import { type ScanDuplicateDocumentsDetails } from "./dupey-tool.js";
 import { type JikjiFindProviderResult } from "./jikji-find-tool.js";
 import { type SearchDocumentDiagnostic, type SearchDocumentsResponse } from "./search-documents.js";
 import { type WatcherFactory, type WatchRefreshHandle } from "./watch-refresh.js";
@@ -24,8 +25,14 @@ export interface AutoRAGRefreshOptions {
     /** Restrict refresh to specific methods. Defaults to all when undefined. */
     readonly methods?: readonly RefreshMethod[];
 }
+export interface AutoRAGMinSyncRefreshResult {
+    readonly ok: boolean;
+    readonly synced: number;
+    readonly reason?: string;
+}
 export interface AutoRAGRefreshResult extends ParsedMirrorSyncResult {
     readonly bm25?: BM25SyncResult;
+    readonly minsync?: AutoRAGMinSyncRefreshResult;
     readonly datasources?: readonly DatasourceIndexResult[];
 }
 export interface AutoRAGRefreshComponentStatus {
@@ -62,7 +69,6 @@ export interface AutoRAGWatchRefreshOptions {
 export type AutoRAGWatchRefreshHandle = WatchRefreshHandle;
 export interface AutoRAGAgentOptions {
     model?: Model<Api>;
-    explorerModel?: Model<Api>;
     apiKey?: string;
     providerApiKeys?: Readonly<Record<string, string>>;
     searchPaths: string[];
@@ -75,9 +81,10 @@ export interface AutoRAGAgentOptions {
     jikji?: JikjiOptions;
     autoRefresh?: AutoRefreshOptions;
     parserOptions?: DefaultParserRegistryOptions;
+    dupey?: DupeyCliOptions | false;
+    excludeExactDuplicates?: boolean;
     datasourceSkills?: readonly DatasourceSkill[];
     datasourceAccess?: DatasourceAccessContextOptions;
-    sessionFactory?: AutoRAGSessionFactory;
 }
 export interface AutoRAGSearchSession {
     readonly agent: Agent;
@@ -85,7 +92,6 @@ export interface AutoRAGSearchSession {
     abort(): Promise<void> | void;
     dispose(): void;
 }
-export type AutoRAGSessionFactory = (options: MandatorySubagentSessionOptions) => Promise<AutoRAGSearchSession>;
 export type AutoRAGJikjiPrepareResult = {
     readonly ok: true;
     readonly code: number;
@@ -100,17 +106,10 @@ export declare class AutoRAGAgent {
     private readonly innerAgent;
     private readonly tools;
     private readonly configuredModel;
-    private readonly configuredExplorerModel;
     private readonly apiKey;
     private readonly providerApiKeys;
-    private readonly sessionFactory;
     private readonly listeners;
     private activeSession;
-    private readonly pendingSubagentCalls;
-    private readonly dispatchSequenceBySession;
-    private readonly dispatchArgsTracker;
-    private dispatchSyntheticCounter;
-    private successfulExplorerCalls;
     private readonly memory;
     private readonly runLogger;
     private lastQuery;
@@ -122,7 +121,9 @@ export declare class AutoRAGAgent {
     private refreshing;
     private refreshState;
     private readonly searchPaths;
-    private readonly allowedExplorerRoots;
+    private readonly configuredSearchPaths;
+    private retrievalScopeBindings;
+    private readonly datasourceVirtualScopePrefixes;
     private readonly workspaceProjectRoot;
     private readonly methodRegistry;
     private readonly retriever;
@@ -135,23 +136,17 @@ export declare class AutoRAGAgent {
     private readonly datasourceAccessOptions;
     private readonly datasourceAgentSkills;
     private readonly parserOptions;
+    private readonly dupeyOptions;
+    private readonly excludeExactDuplicates;
     private readonly baseSystemPromptConfig;
-    /** Run-scoped merged Jikji policy, set during searchDocuments; cleared after. */
-    private activeJikjiPolicy;
-    private activeExplorerModel;
-    /** Run-scoped jikji_find call count for the two-phase raw-fallback gate. */
-    private jikjiFindCallCount;
     private readonly droppedCallerToolNames;
     constructor(options: AutoRAGAgentOptions);
+    scanDuplicateDocuments(): Promise<ScanDuplicateDocumentsDetails>;
     private withMemoryContext;
     private resolveSessionModel;
+    private createSearchSession;
     private configureSearchSession;
     private recordSearchToolEvent;
-    private nextDispatchSequence;
-    private resolveToolCallId;
-    private emitDispatchAutofilled;
-    private emitDispatchRejected;
-    private emitDispatchRejectedFromRejection;
     private currentSystemPromptConfig;
     subscribe(listener: Parameters<Agent["subscribe"]>[0]): () => void;
     abort(): void;
@@ -168,6 +163,7 @@ export declare class AutoRAGAgent {
     submitFeedback(sessionId: string | undefined, satisfied: boolean): void;
     recordResultFeedback(feedback: ResultFeedback[]): void;
     recordFeedbackByNumbers(sessionId: string, usefulNumbers: number[], notUsefulNumbers?: number[]): void;
+    recordFeedbackByIds(usefulFeedbackIds: readonly string[], notUsefulFeedbackIds?: readonly string[]): void;
     getResultRegistry(sessionId?: string): ReadonlyMap<number, CuratedResult>;
     searchDocuments(query: string, options?: RetrievalOptions): Promise<SearchDocumentsResponse>;
     private datasourceAccessContext;
@@ -186,7 +182,7 @@ export declare class AutoRAGAgent {
     private indexDatasources;
     /** Path-opaque component diagnostics (e.g. BM25 readiness) for the search response. */
     private collectComponentDiagnostics;
-    buildSearchPrompt(query: string, options: RetrievalOptions, explorerModel?: string): string;
+    buildSearchPrompt(query: string, options: RetrievalOptions): string;
     refresh(force?: boolean, opts?: AutoRAGRefreshOptions): Promise<AutoRAGRefreshResult>;
     /**
      * Path-opaque snapshot of corpus freshness and the last refresh outcome. Runs
@@ -212,6 +208,7 @@ export declare class AutoRAGAgent {
      * so the refresh result and status remain consistent.
      */
     private scanMirrorStaleness;
+    private exactDuplicateExclusions;
     syncBM25(): Promise<BM25SyncResult | undefined>;
     syncMinSync(): Promise<MinSyncSyncResult | undefined>;
     prepareJikji(): Promise<readonly AutoRAGJikjiPrepareResult[] | undefined>;
@@ -220,12 +217,9 @@ export declare class AutoRAGAgent {
     /**
      * Provider method for the jikji_find tool. Runs JikjiClient.find over all
      * configured search roots, normalizes answer paths against planned source
-     * roots, merges per-root answer packs using least-privilege (restrictive-
-     * wins) semantics, and sets the run-scoped activeJikjiPolicy. Only mutates
-     * run state (activeJikjiPolicy / jikjiFindCallCount) when a searchDocuments
-     * run is active; direct out-of-run calls compute and return the result
-     * without persisting. When jikji is unavailable or all roots fail, returns
-     * an unavailable result and leaves the policy undefined (bash is allowed).
+     * roots, and merges per-root answer packs using least-privilege
+     * (restrictive-wins) semantics. Jikji policy metadata remains visible to the
+     * model, but it does not gate the librarian's direct file-reading tools.
      */
     findJikji(query: string, opts?: {
         readonly topK?: number;
@@ -244,20 +238,9 @@ export declare class AutoRAGAgent {
      * - stopAfterFind: OR
      * - agentShouldNotRerank: OR
      * - handoffAction: MOST RESTRICTIVE (direct_use < jikji_retry < raw_fallback_after_retry)
-     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry AND callCount>=2
+     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry
      */
     private mergePolicy;
-    /**
-     * Deny-by-default bash gate. When jikji is configured but no jikji_find has
-     * run yet this run (count===0), bash is blocked so the agent discovers local
-     * files via jikji_find first. After a find, the run-scoped activeJikjiPolicy
-     * applies: when no policy is active, bash behaves exactly as before
-     * (allowed). Under an active policy, bash is denied unless handoffAction is
-     * raw_fallback_after_retry AND rawFallbackAllowed is true (after a second
-     * jikji_find). When jikji is not configured, the find-first branch is
-     * skipped entirely (bash unchanged).
-     */
-    private bashGate;
     /**
      * Programmatic retrieval across all registered methods, merged via min-max
      * normalization + source dedup. Activates the RetrievalMethodRegistry /
@@ -291,4 +274,7 @@ export declare class AutoRAGAgent {
     /** The retrieval method registry (posix active; vector/bm25/hybrid pluggable). */
     getMethodRegistry(): RetrievalMethodRegistry;
     getSystemPrompt(): string;
+    private rerankWithMemory;
+    private normalizeRetrievalOptions;
+    private resolveRetrievalScope;
 }

package/dist/agent/bash-tool.d.ts

@@ -13,16 +13,6 @@ export interface BashToolOptions {
     readonly timeoutMs?: number;
     /** Maximum captured output bytes before truncation. */
     readonly maxOutputBytes?: number;
-    /**
-     * Optional deny-by-default gate. When present and returns
-     * `{ allowed: false }`, the command is NOT spawned; the tool returns the
-     * gate message with `blockedByJikjiPolicy: true`. Used by the Jikji policy
-     * to enforce stop_after_find / direct_use / forbidden bash restrictions.
-     */
-    readonly gate?: () => {
-        allowed: boolean;
-        message: string;
-    };
 }
 export interface BashToolDetails {
     readonly method: "bash";
@@ -30,7 +20,6 @@ export interface BashToolDetails {
     readonly exitCode: number | undefined;
     readonly timedOut: boolean;
     readonly truncated: boolean;
-    readonly blockedByJikjiPolicy?: boolean;
 }
 /**
  * Real shell access for the librarian agent. AutoRAG navigates and reads the

package/dist/agent/dupey-tool.d.ts

@@ -0,0 +1,15 @@
+import type { AgentTool } from "@earendil-works/pi-agent-core";
+import { Type } from "typebox";
+import type { DupeyScanResult } from "../dupey/index.js";
+export declare const SCAN_DUPLICATE_DOCUMENTS_TOOL_NAME = "scan_duplicate_documents";
+export interface ScanDuplicateDocumentsDetails {
+    readonly scans: readonly DupeyScanResult[];
+    readonly familyCount: number;
+    readonly exactDuplicateCount: number;
+}
+export interface ScanDuplicateDocumentsProvider {
+    scanDuplicateDocuments(): Promise<ScanDuplicateDocumentsDetails>;
+}
+declare const scanDuplicateDocumentsSchema: Type.TObject<{}>;
+export declare function createScanDuplicateDocumentsTool(provider: ScanDuplicateDocumentsProvider): AgentTool<typeof scanDuplicateDocumentsSchema, ScanDuplicateDocumentsDetails>;
+export {};

package/dist/agent/emit-results-tool.d.ts

@@ -27,6 +27,13 @@ declare const emitResultsSchema: Type.TObject<{
             chunkIndex: Type.TOptional<Type.TInteger>;
             lineNumber: Type.TOptional<Type.TInteger>;
             stableEvidenceId: Type.TOptional<Type.TString>;
+            retrieverMix: Type.TOptional<Type.TArray<Type.TString>>;
+            parserType: Type.TOptional<Type.TString>;
+            documentType: Type.TOptional<Type.TString>;
+            documentArea: Type.TOptional<Type.TString>;
+            evidenceType: Type.TOptional<Type.TString>;
+            evidenceLocation: Type.TOptional<Type.TString>;
+            confidence: Type.TOptional<Type.TNumber>;
         }>>>;
     }>>;
     warnings: Type.TOptional<Type.TArray<Type.TString>>;
@@ -51,6 +58,13 @@ export interface AutoRAGEvidenceRef {
     readonly chunkIndex?: number;
     readonly lineNumber?: number;
     readonly stableEvidenceId?: string;
+    readonly retrieverMix?: readonly string[];
+    readonly parserType?: string;
+    readonly documentType?: string;
+    readonly documentArea?: string;
+    readonly evidenceType?: string;
+    readonly evidenceLocation?: string;
+    readonly confidence?: number;
 }
 export interface AutoRAGMappingEntry {
     readonly number: number;

package/dist/agent/index.d.ts

@@ -1,6 +1,7 @@
 export type { AutoRAGAgentOptions, AutoRefreshOptions } from "./agent.js";
 export { AutoRAGAgent } from "./agent.js";
 export { buildDatasourceSkillsPrompt, createLoadDatasourceSkillTool, type DatasourceSkillProvider, datasourceSkillLocation, formatDatasourceSkillInvocation, LOAD_DATASOURCE_SKILL_TOOL_NAME, type LoadDatasourceSkillDetails, toDatasourceAgentSkill, } from "./datasource-skill.js";
+export { createScanDuplicateDocumentsTool, SCAN_DUPLICATE_DOCUMENTS_TOOL_NAME, type ScanDuplicateDocumentsDetails, type ScanDuplicateDocumentsProvider, } from "./dupey-tool.js";
 export type { AutoRAGEmittedResult, AutoRAGMappingEntry, AutoRAGResultsDetails, } from "./emit-results-tool.js";
 export { createEmitResultsTool, EMIT_AUTORAG_RESULTS_TOOL_NAME } from "./emit-results-tool.js";
 export { createJikjiFindTool, JIKJI_FIND_TOOL_NAME, type JikjiFindDetails, type JikjiFindPerRootPolicy, type JikjiFindProvider, type JikjiFindProviderResult, type MergedJikjiPolicy, } from "./jikji-find-tool.js";

package/dist/agent/local-model.d.ts

@@ -0,0 +1,12 @@
+import type { Model } from "@earendil-works/pi-ai";
+export interface LocalAutoRAGModel {
+    readonly provider: string;
+    readonly apiKey: string;
+    readonly model: Model<"openai-responses">;
+}
+export interface LoadLocalAutoRAGModelOptions {
+    readonly configPath?: string;
+    readonly env?: Readonly<Record<string, string | undefined>>;
+    readonly modelId?: string;
+}
+export declare function loadLocalAutoRAGModel(options?: LoadLocalAutoRAGModelOptions): LocalAutoRAGModel;