@autorag/librarian 2.2.0 → 2.3.0

package/README.md

@@ -10,7 +10,11 @@
 
 AutoRAG searches your PDFs, wikis, notes, research papers, and knowledge bases — then curates the results into clean, numbered knowledge units. No raw grep dumps. Just answers.
 
-AutoRAG is a customized [Pi](https://github.com/earendil-works/pi-mono) agent — the Pi agent loop configured into a librarian. Searches use a two-tier workflow: a parent orchestrator delegates exploration to explorer agents. The roles and providers are independently configured from the models available in the user's authenticated runtime; AutoRAG does not ship a private provider default.
+AutoRAG is a customized [Pi](https://github.com/earendil-works/pi-mono) agent — the Pi agent loop configured into a librarian. The AutoRAG librarian retrieves candidates, reads source files directly, judges the evidence, and curates the structured answer. Its model and provider come from the user's authenticated runtime; AutoRAG does not ship a private provider default.
+
+AutoRAG itself is the specialized search agent, not a coordinator for other
+model roles. You configure one model, and that model owns the complete
+retrieval, reading, judgment, and curation loop.
 
 ## Why AutoRAG
 
@@ -61,31 +65,19 @@ Different documents need different search strategies:
 | Legal documents, specifications | BM25 (keyword ranking) | Handles domain terminology well |
 | Mixed collections | Hybrid (vector + BM25) | Combines precision and recall |
 
-AutoRAG supports **pluggable retrieval methods**. It ships with lexical **BM25** and semantic **MinSync** methods wired through the `RetrievalMethodRegistry`, and the architecture is ready for additional vector and hybrid backends. The parent orchestrator owns process-bound retrieval tools and gives bounded seed packs to explorers; explorers use read-only `read`/`grep`/`find`/`ls` tools to inspect the underlying documents. The `ResultMerger` handles cross-method score normalization and deduplication — you get one unified result set regardless of how many methods contributed.
+AutoRAG supports **pluggable retrieval methods**. It ships with lexical **BM25** and semantic **MinSync** methods wired through the `RetrievalMethodRegistry`, and the architecture is ready for additional vector and hybrid backends. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication.
 
 BM25 and MinSync are **enabled by default** — no explicit configuration is needed for standard lexical + semantic retrieval. Both can be disabled by setting `"bm25": false` or `"minSync": false` in the config file. MinSync uses a pre-installed binary (`autoInstall: false`); configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints. AutoRAG never forces TEI or any external embedding service.
 
 ### Real directory access
 
-AutoRAG reads configured source directories through delegated explorer tasks. Each explorer is assigned exactly one normalized configured search root as its `cwd`; the top-level `subagent` invocation sets `agentScope: "user"` and `artifacts: false` exactly once for single, `tasks`, `chain`, or `parallel` dispatch, and nested explorer task items omit both fields. Project-local `.pi-subagents` debug artifacts are disabled. Explorers use read-only `read`/`grep`/`find`/`ls`; the parent orchestrator owns retrieval seed tools and must delegate document reading before curating. Curated answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. BM25 and MinSync index parsed markdown mirrors under `.autorag`.
-
-Each explorer `task` contains an Assignment V1 block — a sentinel-wrapped JSON body with `originalQuery` (the caller query verbatim), `method` (the selected retrieval path), and `queryVariants` (a nonempty array), followed by canonical role lines requiring `retrievedAt` and temporal metadata:
-
-```text
-<<<AUTORAG_ASSIGNMENT_V1>>>
-{"originalQuery":"<caller query verbatim>","method":"<selected retrieval method>","queryVariants":["<variant 1>","<variant 2>"]}
-<<<END_AUTORAG_ASSIGNMENT_V1>>>
-Required handoff: include retrievedAt.
-Required handoff: include temporal metadata.
-```
-
-A legacy labeled format (`Original query:`, `Selected retrieval method:`, `Query variants:`) is accepted for compatibility. Missing or null top-level `artifacts`, `agentScope`, and leaf `model` fields are safely autofilled before validation; explicit wrong values (`artifacts: true`, `agentScope: "project"`, a non-configured model) remain rejected. There is no single-agent fallback. See [docs/subagent-orchestration.md](docs/subagent-orchestration.md) for the full dispatch contract, templates, anti-examples, and stable coded dispatch errors.
+AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. BM25 and MinSync index parsed markdown mirrors under `.autorag`.
 
 ### Optional Jikji discovery and indexing
 
 AutoRAG can opt into [Jikji](https://github.com/NomaDamas/jikji) as a local CLI-backed **find-first discovery and indexing** layer. Jikji is optional: AutoRAG does not vendor it, install it, or register it as a retrieval backend when enabled.
 
-When Jikji is configured, AutoRAG calls `jikji find ROOT "query" --json` via a policy-aware `jikji_find` tool as the first local-discovery action. The tool parses and validates the upstream answer-pack and honors its `handoff_action` (`direct_use` / `jikji_retry` / `raw_fallback_after_retry`), `tool_call_policy` (`stop_after_find`, `forbidden_tools`, `allowed_followups`), and `agent_should_not_rerank`. Explorer `read`/`grep`/`find`/`ls` discovery is the fallback only when the answer-pack permits raw fallback (`raw_fallback_after_retry`, after the required retry) or when Jikji is unavailable/unconfigured. `prepare`/`refresh` remain for indexing only and do not answer queries directly.
+When Jikji is configured, AutoRAG calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly.
 
 Programmatic use:
 
@@ -97,6 +89,35 @@ const agent = new AutoRAGAgent({
 await agent.prepareJikji();
 ```
 
+### Duplicate document management with dupey
+
+AutoRAG can use the external [`dupey`](https://github.com/NomaDamas/dupey) CLI
+to detect exact, near, and containment document families.
+
+```bash
+autorag duplicates /path/to/documents
+autorag duplicates --json
+```
+
+The command is read-only: it reports exact duplicate groups and review
+guidance, but never moves or deletes source files. The `scan_duplicate_documents`
+Agent tool exposes the same read-only scan to the orchestrator.
+
+Exact duplicate exclusion is enabled by default during parsed-mirror refresh.
+For each exact canonical-text hash, the newest filesystem copy is indexed and
+older copies are omitted. Disable it in `config.json` when both copies must be
+searchable:
+
+```json
+{
+  "dupey": { "enabled": true },
+  "excludeExactDuplicates": false
+}
+```
+
+If `dupey` is not installed or fails, refresh continues without exclusion and
+reports no destructive action; install it with `cargo install dupey`.
+
 The same `.autorag/jikji.json` shape configures Jikji when present:
 
 ```json
@@ -122,6 +143,14 @@ The upstream Rust `PrepareArgs` defines reference defaults that AutoRAG does not
 
 Datasource skills let AutoRAG search external, server-configured data sources through the same retrieval pipeline as local documents. A skill describes what it indexes, how it should be refreshed, what source instances exist, and which permission tags/scopes bound access. Retrieval still flows through `RetrievalMethodRegistry` → `ParallelRetriever` → datasource result filtering → `ResultMerger`; datasource skills do not create a parallel search path.
 
+Every datasource can be registered multiple times through a connection alias:
+use the config key as the unique name and set `type` to the reusable backend
+(`gmail`, `github`, `slack`, `discord`, `kakao`, `cloud-drive`, and so on).
+Each alias becomes an independently loadable agent skill with its own source
+scope and workspace namespace. Chat aliases search all channels by default;
+trusted `channels.ids` / `channels.names` allowlists can expose a particular
+channel or group chat as its own datasource.
+
 Security defaults are intentionally strict:
 
 - datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags` and `datasourceAccess.allowedScopes`;
@@ -139,7 +168,7 @@ Security defaults are intentionally strict:
 | Discord | `discord` | external [`discrawl`](https://github.com/openclaw/discrawl) CLI | guild/channel/thread/DM archive; FTS5 + semantic + hybrid retrieval, incremental sync |
 | Notion | `notion` | external [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | local-first page/database/block archive + FTS5 search |
 | GitHub Issues/PRs | `github` | GitHub REST (token optional) | issues + PR bodies per `owner/repo`; public repos work unauthenticated |
-| Google Drive | `gdrive` | Drive REST v3, or **[`rclone`](https://rclone.org) CLI** (`backend: "rclone"`) | Docs/Sheets exported as text; the rclone backend also opens any of rclone's 70+ remotes |
+| Cloud drives | `cloud-drive` | **[`rclone`](https://rclone.org) CLI** | Incremental Google Drive Tier-1; OneDrive/network remotes; iCloud experimental |
 | Gmail / IMAP | `gmail` | Gmail REST v1, or **[`himalaya`](https://pimalaya.org) CLI** (`backend: "himalaya"`) | the himalaya backend indexes any IMAP/Maildir account it has configured — no OAuth plumbing |
 | Local mail exports | `mail-export` | filesystem (`.mbox` / `.eml`) | classic `From_` splitting, mailparser-based; count-only warnings |
 | Obsidian vault | `obsidian` | external [`qmd`](https://github.com/tobi/qmd) CLI | incremental `qmd update`, BM25 `qmd search`, semantic `qmd vsearch`; vault path via `connector.vaultPath` |
@@ -158,13 +187,14 @@ Configure them in `config.json` (CLI) or pass `datasourceSkills` programmaticall
     "notion":   { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } },
     "github":   { "connector": { "repos": ["owner/repo"] } },
     "gmail":    { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
-    "gdrive":   { "connector": { "backend": "rclone", "remote": "gdrive:" } },
+    "personal-google-drive": { "type": "cloud-drive", "instanceId": "personal", "connector": { "provider": "google-drive", "remote": "personal-gdrive:", "include": ["**/*.md"] } },
+    "company-onedrive": { "type": "cloud-drive", "instanceId": "work", "connector": { "provider": "onedrive", "remote": "company-onedrive:Documents" } },
     "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
     "rss":      { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
   },
   "datasourceAccess": {
-    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "gdrive", "obsidian", "rss"],
-    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/gdrive/**", "/obsidian/**", "/rss/**"]
+    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "cloud-drive", "obsidian", "rss"],
+    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
   }
 }
 ```
@@ -177,6 +207,21 @@ Install slacrawl with `brew install openclaw/tap/slacrawl`. AutoRAG invokes `sla
 
 Install notcrawl with `brew install openclaw/tap/notcrawl`. AutoRAG invokes `notcrawl sync` during datasource refresh and `notcrawl search --json` during retrieval. Optional trusted connector fields are `binaryPath` and `configPath`. Notion credentials and workspace definitions remain in notcrawl's own configuration rather than AutoRAG.
 
+Install and authenticate rclone separately (`brew install rclone && rclone
+config` on macOS), then configure the provider-neutral `cloud-drive` skill.
+`cloud-drive` is a reusable `type`: each datasource config key is a connection
+alias and becomes a separate agent skill and scope. Multiple Google accounts,
+or Google Drive plus OneDrive/iCloud, can therefore be loaded and searched
+independently.
+AutoRAG inventories with `rclone lsjson`, keeps a workspace-local manifest and
+managed mirror, and downloads only added or changed indexable files. Google
+Drive is Tier-1. OneDrive, Dropbox, SMB/SFTP/WebDAV, and mounted drives share
+the same manifest contract. iCloud Drive is experimental because rclone marks
+that backend Tier 4 and Apple ID/2FA sessions periodically require
+reauthentication. See [docs/datasource-skills.md](docs/datasource-skills.md)
+for filtering, size, concurrency, bandwidth, dry-run, and agent tool-calling
+details.
+
 #### KakaoTalk (katok)
 
 KakaoTalk was the first datasource skill. It uses the external [`katok`](https://github.com/NomaDamas/katok) CLI only — AutoRAG never reads KakaoTalk databases directly. `katok` failures return diagnostics instead of throwing, and remote embedding egress configuration is rejected before the CLI is spawned.
@@ -260,7 +305,7 @@ A datasource skill should provide polling/cron metadata for routine indexing, so
 
 AutoRAG is built for **non-code document retrieval**: manuals, legal docs, internal wikis, meeting notes, research literature, knowledge bases, PDFs.
 
-Code repositories work too (the explorer's `grep` is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
+Code repositories work too (direct `grep` is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
 
 ## Configuration and state
 
@@ -270,38 +315,39 @@ The default home state is kept outside the workspace:
 ~/.autorag/
 ├── config.json
 ├── memory.json
-├── logs/
-│   └── runs.jsonl
-└── pi-agent/
-    ├── auth.json
-    ├── models.json
-    ├── settings.json
-    └── sessions/
+└── logs/
+    └── runs.jsonl
 ```
 
-`config.json` selects sources, the workspace, memory path, retrieval settings, and the two agent models. Configure the roles independently with `agents.orchestrator` and `agents.explorer`. Provider and model IDs must refer to models available in the user's authenticated runtime:
+`config.json` selects sources, the workspace, memory path, retrieval settings, and the agent model. Provider and model IDs must refer to a model available in the user's authenticated runtime:
 
 ```json
 {
   "searchPaths": ["/path/to/documents"],
   "workspacePath": "/path/to/workspace",
   "memoryPath": "/Users/you/.autorag/memory.json",
-  "agents": {
-    "orchestrator": { "provider": "provider-name", "id": "reasoning-model" },
-    "explorer": { "provider": "provider-name", "id": "exploration-model" }
-  }
+  "model": { "provider": "provider-name", "id": "reasoning-model" }
 }
 ```
 
-`autorag init` leaves `agents` unset when no role-model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure both roles explicitly.
+`autorag init` leaves `model` unset when no model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure the model explicitly.
+
+For fast interactive search, prefer a model with reliable tool calling, high
+output TPS, and low first-token latency. A query can require several short
+model turns while AutoRAG alternates between retrieval tools and direct source
+reading, so model throughput has a visible effect on end-to-end response time.
+It does not accelerate BM25, MinSync, Jikji, filesystem access, or indexing
+itself. Larger reasoning models remain useful for difficult synthesis,
+conflicting evidence, and specialized domain judgment, but they are not a
+requirement for ordinary retrieval.
 
 Config path precedence is `--config` > `AUTORAG_CONFIG` > `~/.autorag/config.json`. When the home config is absent and `<cwd>/autorag.config.json` exists, AutoRAG copies the legacy file to `~/.autorag/config.json` without deleting or modifying the legacy file. The legacy cwd file is a migration source, not the default location.
 
-`memory.json` stores retrieval memory, `logs/runs.jsonl` records run events, and durable Pi models, settings, and sessions stay under `~/.autorag/pi-agent`. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under `<workspace>/.autorag`.
+`memory.json` stores retrieval memory and `logs/runs.jsonl` records run events. Model authentication remains with the user's configured provider or authenticated local runtime. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under `<workspace>/.autorag`.
 
 `autorag refresh` and `autorag index reset|rebuild` accept `--method <csv>` (e.g. `--method bm25,minsync,parsed`) to scope which indexing methods run or which index directories are removed. When omitted, all methods run. `autorag init` accepts `--embedder-*` flags to configure the MinSync embedder endpoint in the config file.
 
-`autorag health` checks model/provider auth and explorer subagent setup before a search — it resolves both role models, verifies credential presence, and optionally probes a completion call per role. Use it to diagnose model, provider, auth, timeout, or subagent dispatch failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness); it does not check models or subagent dispatch. When `autorag search` fails for a model/provider/subagent reason, the error output includes a hint pointing to `autorag health`.
+`autorag health` checks model/provider auth before a search — it resolves the model, verifies credential presence, and optionally probes one completion call. Use it to diagnose model, provider, auth, or timeout failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness). When `autorag search` fails for a model/provider reason, the error output includes a hint pointing to `autorag health`.
 
 ## Installation
 
@@ -345,15 +391,11 @@ agent.recordFeedbackByNumbers(response.sessionId, [1, 3], [2]);
            │
            ▼
     ┌──────────────┐
-    │ Plan + seed  │ ← parent check_memory, Jikji, and retrieval tools
-    └──────┬───────┘
-           ▼
-    ┌──────────────┐
-    │ Delegate     │ ← pi-subagents → gpt-5.6-luna explorer
+    │ Plan + search│ ← check_memory, Jikji, and retrieval tools
     └──────┬───────┘
            ▼
     ┌──────────────┐
-    │ Search + read│ ← explorer read/grep/find/ls
+    │ Direct read  │ ← bash find/grep/cat
     └──────┬──────┘
            ▼
     ┌─────────────┐

package/dist/agent/agent.d.ts

@@ -2,6 +2,7 @@ import { Agent, type AgentTool, type Skill } from "@earendil-works/pi-agent-core
 import type { Api, Model } from "@earendil-works/pi-ai";
 import { type DatasourceAccessContextOptions } from "../datasource/access-context.js";
 import type { DatasourceIndexResult, DatasourceSkill } from "../datasource/types.js";
+import { type DupeyCliOptions } from "../dupey/index.js";
 import { type JikjiFailureReason, type JikjiOptions } from "../jikji/index.js";
 import type { ResultFeedback } from "../memory/memory.js";
 import { type MinSyncSyncResult, type MinSyncVectorMethodOptions } from "../minsync/index.js";
@@ -10,7 +11,7 @@ import type { DefaultParserRegistryOptions } from "../parser/index.js";
 import { type BM25MethodOptions, type BM25SyncResult } from "../retrieval/methods/bm25.js";
 import { RetrievalMethodRegistry } from "../retrieval/registry.js";
 import type { CuratedResult, RetrievalDiagnostic, RetrievalOptions, RetrievalResult } from "../retrieval/types.js";
-import { type MandatorySubagentSessionOptions } from "../subagents/runtime.js";
+import { type ScanDuplicateDocumentsDetails } from "./dupey-tool.js";
 import { type JikjiFindProviderResult } from "./jikji-find-tool.js";
 import { type SearchDocumentDiagnostic, type SearchDocumentsResponse } from "./search-documents.js";
 import { type WatcherFactory, type WatchRefreshHandle } from "./watch-refresh.js";
@@ -68,7 +69,6 @@ export interface AutoRAGWatchRefreshOptions {
 export type AutoRAGWatchRefreshHandle = WatchRefreshHandle;
 export interface AutoRAGAgentOptions {
     model?: Model<Api>;
-    explorerModel?: Model<Api>;
     apiKey?: string;
     providerApiKeys?: Readonly<Record<string, string>>;
     searchPaths: string[];
@@ -81,9 +81,10 @@ export interface AutoRAGAgentOptions {
     jikji?: JikjiOptions;
     autoRefresh?: AutoRefreshOptions;
     parserOptions?: DefaultParserRegistryOptions;
+    dupey?: DupeyCliOptions | false;
+    excludeExactDuplicates?: boolean;
     datasourceSkills?: readonly DatasourceSkill[];
     datasourceAccess?: DatasourceAccessContextOptions;
-    sessionFactory?: AutoRAGSessionFactory;
 }
 export interface AutoRAGSearchSession {
     readonly agent: Agent;
@@ -91,7 +92,6 @@ export interface AutoRAGSearchSession {
     abort(): Promise<void> | void;
     dispose(): void;
 }
-export type AutoRAGSessionFactory = (options: MandatorySubagentSessionOptions) => Promise<AutoRAGSearchSession>;
 export type AutoRAGJikjiPrepareResult = {
     readonly ok: true;
     readonly code: number;
@@ -106,17 +106,10 @@ export declare class AutoRAGAgent {
     private readonly innerAgent;
     private readonly tools;
     private readonly configuredModel;
-    private readonly configuredExplorerModel;
     private readonly apiKey;
     private readonly providerApiKeys;
-    private readonly sessionFactory;
     private readonly listeners;
     private activeSession;
-    private readonly pendingSubagentCalls;
-    private readonly dispatchSequenceBySession;
-    private readonly dispatchArgsTracker;
-    private dispatchSyntheticCounter;
-    private successfulExplorerCalls;
     private readonly memory;
     private readonly runLogger;
     private lastQuery;
@@ -131,7 +124,6 @@ export declare class AutoRAGAgent {
     private readonly configuredSearchPaths;
     private retrievalScopeBindings;
     private readonly datasourceVirtualScopePrefixes;
-    private readonly allowedExplorerRoots;
     private readonly workspaceProjectRoot;
     private readonly methodRegistry;
     private readonly retriever;
@@ -144,23 +136,17 @@ export declare class AutoRAGAgent {
     private readonly datasourceAccessOptions;
     private readonly datasourceAgentSkills;
     private readonly parserOptions;
+    private readonly dupeyOptions;
+    private readonly excludeExactDuplicates;
     private readonly baseSystemPromptConfig;
-    /** Run-scoped merged Jikji policy, set during searchDocuments; cleared after. */
-    private activeJikjiPolicy;
-    private activeExplorerModel;
-    /** Run-scoped jikji_find call count for the two-phase raw-fallback gate. */
-    private jikjiFindCallCount;
     private readonly droppedCallerToolNames;
     constructor(options: AutoRAGAgentOptions);
+    scanDuplicateDocuments(): Promise<ScanDuplicateDocumentsDetails>;
     private withMemoryContext;
     private resolveSessionModel;
+    private createSearchSession;
     private configureSearchSession;
     private recordSearchToolEvent;
-    private nextDispatchSequence;
-    private resolveToolCallId;
-    private emitDispatchAutofilled;
-    private emitDispatchRejected;
-    private emitDispatchRejectedFromRejection;
     private currentSystemPromptConfig;
     subscribe(listener: Parameters<Agent["subscribe"]>[0]): () => void;
     abort(): void;
@@ -177,6 +163,7 @@ export declare class AutoRAGAgent {
     submitFeedback(sessionId: string | undefined, satisfied: boolean): void;
     recordResultFeedback(feedback: ResultFeedback[]): void;
     recordFeedbackByNumbers(sessionId: string, usefulNumbers: number[], notUsefulNumbers?: number[]): void;
+    recordFeedbackByIds(usefulFeedbackIds: readonly string[], notUsefulFeedbackIds?: readonly string[]): void;
     getResultRegistry(sessionId?: string): ReadonlyMap<number, CuratedResult>;
     searchDocuments(query: string, options?: RetrievalOptions): Promise<SearchDocumentsResponse>;
     private datasourceAccessContext;
@@ -195,7 +182,7 @@ export declare class AutoRAGAgent {
     private indexDatasources;
     /** Path-opaque component diagnostics (e.g. BM25 readiness) for the search response. */
     private collectComponentDiagnostics;
-    buildSearchPrompt(query: string, options: RetrievalOptions, explorerModel?: string): string;
+    buildSearchPrompt(query: string, options: RetrievalOptions): string;
     refresh(force?: boolean, opts?: AutoRAGRefreshOptions): Promise<AutoRAGRefreshResult>;
     /**
      * Path-opaque snapshot of corpus freshness and the last refresh outcome. Runs
@@ -221,6 +208,7 @@ export declare class AutoRAGAgent {
      * so the refresh result and status remain consistent.
      */
     private scanMirrorStaleness;
+    private exactDuplicateExclusions;
     syncBM25(): Promise<BM25SyncResult | undefined>;
     syncMinSync(): Promise<MinSyncSyncResult | undefined>;
     prepareJikji(): Promise<readonly AutoRAGJikjiPrepareResult[] | undefined>;
@@ -229,12 +217,9 @@ export declare class AutoRAGAgent {
     /**
      * Provider method for the jikji_find tool. Runs JikjiClient.find over all
      * configured search roots, normalizes answer paths against planned source
-     * roots, merges per-root answer packs using least-privilege (restrictive-
-     * wins) semantics, and sets the run-scoped activeJikjiPolicy. Only mutates
-     * run state (activeJikjiPolicy / jikjiFindCallCount) when a searchDocuments
-     * run is active; direct out-of-run calls compute and return the result
-     * without persisting. When jikji is unavailable or all roots fail, returns
-     * an unavailable result and leaves the policy undefined (bash is allowed).
+     * roots, and merges per-root answer packs using least-privilege
+     * (restrictive-wins) semantics. Jikji policy metadata remains visible to the
+     * model, but it does not gate the librarian's direct file-reading tools.
      */
     findJikji(query: string, opts?: {
         readonly topK?: number;
@@ -253,20 +238,9 @@ export declare class AutoRAGAgent {
      * - stopAfterFind: OR
      * - agentShouldNotRerank: OR
      * - handoffAction: MOST RESTRICTIVE (direct_use < jikji_retry < raw_fallback_after_retry)
-     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry AND callCount>=2
+     * - rawFallbackAllowed: handoffAction===raw_fallback_after_retry
      */
     private mergePolicy;
-    /**
-     * Deny-by-default bash gate. When jikji is configured but no jikji_find has
-     * run yet this run (count===0), bash is blocked so the agent discovers local
-     * files via jikji_find first. After a find, the run-scoped activeJikjiPolicy
-     * applies: when no policy is active, bash behaves exactly as before
-     * (allowed). Under an active policy, bash is denied unless handoffAction is
-     * raw_fallback_after_retry AND rawFallbackAllowed is true (after a second
-     * jikji_find). When jikji is not configured, the find-first branch is
-     * skipped entirely (bash unchanged).
-     */
-    private bashGate;
     /**
      * Programmatic retrieval across all registered methods, merged via min-max
      * normalization + source dedup. Activates the RetrievalMethodRegistry /
@@ -300,6 +274,7 @@ export declare class AutoRAGAgent {
     /** The retrieval method registry (posix active; vector/bm25/hybrid pluggable). */
     getMethodRegistry(): RetrievalMethodRegistry;
     getSystemPrompt(): string;
+    private rerankWithMemory;
     private normalizeRetrievalOptions;
     private resolveRetrievalScope;
 }

package/dist/agent/bash-tool.d.ts

@@ -13,16 +13,6 @@ export interface BashToolOptions {
     readonly timeoutMs?: number;
     /** Maximum captured output bytes before truncation. */
     readonly maxOutputBytes?: number;
-    /**
-     * Optional deny-by-default gate. When present and returns
-     * `{ allowed: false }`, the command is NOT spawned; the tool returns the
-     * gate message with `blockedByJikjiPolicy: true`. Used by the Jikji policy
-     * to enforce stop_after_find / direct_use / forbidden bash restrictions.
-     */
-    readonly gate?: () => {
-        allowed: boolean;
-        message: string;
-    };
 }
 export interface BashToolDetails {
     readonly method: "bash";
@@ -30,7 +20,6 @@ export interface BashToolDetails {
     readonly exitCode: number | undefined;
     readonly timedOut: boolean;
     readonly truncated: boolean;
-    readonly blockedByJikjiPolicy?: boolean;
 }
 /**
  * Real shell access for the librarian agent. AutoRAG navigates and reads the

package/dist/agent/dupey-tool.d.ts

@@ -0,0 +1,15 @@
+import type { AgentTool } from "@earendil-works/pi-agent-core";
+import { Type } from "typebox";
+import type { DupeyScanResult } from "../dupey/index.js";
+export declare const SCAN_DUPLICATE_DOCUMENTS_TOOL_NAME = "scan_duplicate_documents";
+export interface ScanDuplicateDocumentsDetails {
+    readonly scans: readonly DupeyScanResult[];
+    readonly familyCount: number;
+    readonly exactDuplicateCount: number;
+}
+export interface ScanDuplicateDocumentsProvider {
+    scanDuplicateDocuments(): Promise<ScanDuplicateDocumentsDetails>;
+}
+declare const scanDuplicateDocumentsSchema: Type.TObject<{}>;
+export declare function createScanDuplicateDocumentsTool(provider: ScanDuplicateDocumentsProvider): AgentTool<typeof scanDuplicateDocumentsSchema, ScanDuplicateDocumentsDetails>;
+export {};

package/dist/agent/emit-results-tool.d.ts

@@ -27,6 +27,13 @@ declare const emitResultsSchema: Type.TObject<{
             chunkIndex: Type.TOptional<Type.TInteger>;
             lineNumber: Type.TOptional<Type.TInteger>;
             stableEvidenceId: Type.TOptional<Type.TString>;
+            retrieverMix: Type.TOptional<Type.TArray<Type.TString>>;
+            parserType: Type.TOptional<Type.TString>;
+            documentType: Type.TOptional<Type.TString>;
+            documentArea: Type.TOptional<Type.TString>;
+            evidenceType: Type.TOptional<Type.TString>;
+            evidenceLocation: Type.TOptional<Type.TString>;
+            confidence: Type.TOptional<Type.TNumber>;
         }>>>;
     }>>;
     warnings: Type.TOptional<Type.TArray<Type.TString>>;
@@ -51,6 +58,13 @@ export interface AutoRAGEvidenceRef {
     readonly chunkIndex?: number;
     readonly lineNumber?: number;
     readonly stableEvidenceId?: string;
+    readonly retrieverMix?: readonly string[];
+    readonly parserType?: string;
+    readonly documentType?: string;
+    readonly documentArea?: string;
+    readonly evidenceType?: string;
+    readonly evidenceLocation?: string;
+    readonly confidence?: number;
 }
 export interface AutoRAGMappingEntry {
     readonly number: number;

package/dist/agent/index.d.ts

@@ -1,6 +1,7 @@
 export type { AutoRAGAgentOptions, AutoRefreshOptions } from "./agent.js";
 export { AutoRAGAgent } from "./agent.js";
 export { buildDatasourceSkillsPrompt, createLoadDatasourceSkillTool, type DatasourceSkillProvider, datasourceSkillLocation, formatDatasourceSkillInvocation, LOAD_DATASOURCE_SKILL_TOOL_NAME, type LoadDatasourceSkillDetails, toDatasourceAgentSkill, } from "./datasource-skill.js";
+export { createScanDuplicateDocumentsTool, SCAN_DUPLICATE_DOCUMENTS_TOOL_NAME, type ScanDuplicateDocumentsDetails, type ScanDuplicateDocumentsProvider, } from "./dupey-tool.js";
 export type { AutoRAGEmittedResult, AutoRAGMappingEntry, AutoRAGResultsDetails, } from "./emit-results-tool.js";
 export { createEmitResultsTool, EMIT_AUTORAG_RESULTS_TOOL_NAME } from "./emit-results-tool.js";
 export { createJikjiFindTool, JIKJI_FIND_TOOL_NAME, type JikjiFindDetails, type JikjiFindPerRootPolicy, type JikjiFindProvider, type JikjiFindProviderResult, type MergedJikjiPolicy, } from "./jikji-find-tool.js";

package/dist/agent/local-model.d.ts

@@ -0,0 +1,12 @@
+import type { Model } from "@earendil-works/pi-ai";
+export interface LocalAutoRAGModel {
+    readonly provider: string;
+    readonly apiKey: string;
+    readonly model: Model<"openai-responses">;
+}
+export interface LoadLocalAutoRAGModelOptions {
+    readonly configPath?: string;
+    readonly env?: Readonly<Record<string, string | undefined>>;
+    readonly modelId?: string;
+}
+export declare function loadLocalAutoRAGModel(options?: LoadLocalAutoRAGModelOptions): LocalAutoRAGModel;

package/dist/agent/search-documents.d.ts

@@ -9,7 +9,7 @@ export type SearchDocumentDiagnosticSeverity = "info" | "warning" | "error";
  * (#21/#22) does not require another source-breaking type change; codes not yet
  * emitted are still valid members.
  */
-export type SearchDocumentDiagnosticCode = "empty-query" | "unknown-warning" | "caller-tool-dropped" | "bm25-unavailable" | "bm25-degraded-fallback" | "minsync-unavailable" | "parser-skipped" | "parser-failed" | "unsupported-file" | "stale-index" | "deleted-mirror" | "retrieval-method-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "refresh-failed" | "watch-failed" | "watch-limited";
+export type SearchDocumentDiagnosticCode = "empty-query" | "unknown-warning" | "caller-tool-dropped" | "bm25-unavailable" | "bm25-degraded-fallback" | "minsync-unavailable" | "parser-skipped" | "parser-failed" | "duplicate-excluded" | "unsupported-file" | "stale-index" | "deleted-mirror" | "retrieval-method-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "refresh-failed" | "watch-failed" | "watch-limited";
 export interface SearchDocumentDiagnostic {
     readonly code: SearchDocumentDiagnosticCode;
     readonly severity: SearchDocumentDiagnosticSeverity;

package/dist/agent/system-prompt.d.ts

@@ -2,8 +2,7 @@ import type { Skill } from "@earendil-works/pi-agent-core";
 import type { StoreManifest } from "../manifest/types.js";
 export interface SystemPromptConfig {
     toolNames: string[];
-    orchestratorModelId?: string;
-    explorerModelId?: string;
+    modelId?: string;
     memorySignalCount?: number;
     memoryEntries?: readonly unknown[];
     manifests: StoreManifest[];

package/dist/cli/commands/duplicates.d.ts

@@ -0,0 +1,4 @@
+import type { DupeyScanResult } from "../../dupey/index.js";
+import type { CommandContext } from "./types.js";
+export type DuplicateScanner = (path: string) => Promise<DupeyScanResult>;
+export declare function runDuplicates(ctx: CommandContext, scanner?: DuplicateScanner): Promise<number>;