@autorag/librarian 2.10.0 → 2.10.2

package/README.md

@@ -7,7 +7,7 @@
   <a href="https://www.npmjs.com/package/@autorag/librarian"><img src="https://img.shields.io/npm/dm/@autorag/librarian.svg?style=flat-square&color=818cf8" alt="npm downloads" /></a>
   <a href="https://github.com/Marker-Inc-Korea/AutoRAG/stargazers"><img src="https://img.shields.io/github/stars/Marker-Inc-Korea/AutoRAG?style=flat-square&logo=github&color=f59e0b" alt="GitHub stars" /></a>
   <a href="https://github.com/Marker-Inc-Korea/AutoRAG/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/Marker-Inc-Korea/AutoRAG/ci.yml?branch=main&label=CI&style=flat-square" alt="CI Status" /></a>
-  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg?style=flat-square" alt="License: MIT" /></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-green.svg?style=flat-square" alt="License: Apache-2.0" /></a>
   <img src="https://img.shields.io/badge/node-%3E%3D24.0.0-informational.svg?style=flat-square" alt="Node >= 24" />
   <img src="https://img.shields.io/badge/bun-%3E%3D1.2-black.svg?style=flat-square&logo=bun" alt="Bun >= 1.2" />
 </p>
@@ -24,6 +24,141 @@
 
 ---
 
+## 🖥️ The desktop app
+
+**A Finder for your computer that understands what's inside your files.** The
+AutoRAG desktop app puts this librarian behind a real file manager: browse your
+folders as usual, then ask a question and get an answer that cites the files,
+chats, and mail it came from. Everything it searches with is the engine in this
+repository.
+
+<p align="center">
+  <img src="assets/app/finder.webp" alt="AutoRAG desktop app: a folder in list view with a version stack expanded and the index column" width="100%" />
+</p>
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+### 🔎 Just ask
+
+Ask the way you'd ask a teammate. The librarian reads the files and writes back,
+every claim linked to the passage behind it. Drag files, folders, or links into
+a question to ask about just those.
+
+</td>
+<td width="50%" valign="top">
+
+<img src="assets/app/ask.webp" alt="An answer about the Seoul office lease with numbered citations" />
+
+</td>
+</tr>
+<tr>
+<td width="50%" valign="top">
+
+<img src="assets/app/versions.webp" alt="A folder in list view with document versions stacked under one row" />
+
+</td>
+<td width="50%" valign="top">
+
+### 🗂️ Spot the twins
+
+`proposal-final`, `proposal-v3`, `proposal (1)`: stacked into one row, newest
+on top, tagged **Latest**, **Similar**, or **Identical**. Powered by
+[dupey](https://github.com/NomaDamas/dupey), the same scan as
+`autorag duplicates`.
+
+</td>
+</tr>
+<tr>
+<td width="50%" valign="top">
+
+### ⌨️ Space to see anything
+
+Markdown, code, PDF, video, audio, Office, and HWP/HWPX previewed instantly in
+Quick Look. HWP pages are drawn locally by
+[rhwp](https://github.com/edwardkim/rhwp); Hancom Office is not required.
+
+</td>
+<td width="50%" valign="top">
+
+<img src="assets/app/quick-look.webp" alt="Quick Look showing a multi-page PDF with a page rail" />
+
+</td>
+</tr>
+<tr>
+<td width="50%" valign="top">
+
+<img src="assets/app/panes.webp" alt="Two folder panes side by side, each with its own tabs" />
+
+</td>
+<td width="50%" valign="top">
+
+### 🪟 A real file manager
+
+Tabs and split panes: drag a tab to an edge to split, and every pane keeps its
+own tabs, so a folder and a conversation sit side by side. Folders update live
+as other apps change them.
+
+</td>
+</tr>
+</table>
+
+**And more:** hybrid keyword + meaning search with a reranker · scheduled
+re-indexing that runs even while the app is closed · an *Ask* entry in the
+Finder and Explorer right-click menu · light and dark mode · Korean, English,
+Japanese, and Simplified Chinese.
+
+### Every place, one search
+
+The app connects the same [datasources](#-supported-datasources) the librarian
+does, and searches them right alongside your files:
+
+<table>
+<tr>
+<td align="center" width="25%">💬<br><b>Slack</b><br>Channels and threads</td>
+<td align="center" width="25%">💛<br><b>KakaoTalk</b><br>Chats on this computer</td>
+<td align="center" width="25%">🎮<br><b>Discord</b><br>Servers and forums</td>
+<td align="center" width="25%">✈️<br><b>Telegram</b><br>Chats and channels</td>
+</tr>
+<tr>
+<td align="center">📱<br><b>WhatsApp</b><br>Messages, read-only</td>
+<td align="center">✉️<br><b>Mail</b><br>Every inbox you sign in to</td>
+<td align="center">🐙<br><b>GitHub</b><br>Repos, issues, PRs</td>
+<td align="center">🔮<br><b>Obsidian</b><br>Your vaults</td>
+</tr>
+</table>
+
+### Private by design
+
+- **Embedded on your machine.** Document understanding runs on the local
+  [`qwen3-embedding-0.6b`](docs/model-cards/qwen3-embedding-0.6b.md) model.
+  File contents are never sent to a remote embedder.
+- **Keys stay encrypted.** Sign-in tokens and API keys live in the system's
+  encrypted store, never in a config file.
+- **Bring your own model.** Use the [hosted Dazzi Models plan](#hosted-dazzi-models-plan),
+  or connect your own provider key and pay the provider directly.
+
+> [!TIP]
+> **The app's source lives in [NomaDamas/AutoRAG-app](https://github.com/NomaDamas/AutoRAG-app)**
+> (Electron + React, AGPL-3.0). It depends on `@autorag/librarian` from this
+> repository; build, run, and file app issues there.
+
+### One engine, every surface
+
+The app is one way in. The same librarian runs everywhere else:
+
+| Surface | Use it for | Start here |
+|---|---|---|
+| **Desktop app** | Browsing and asking in one window | [NomaDamas/AutoRAG-app](https://github.com/NomaDamas/AutoRAG-app) |
+| **Terminal UI** | Interactive, steerable search sessions | [`autorag tui`](#2-search-from-cli) |
+| **CLI** | Scripts, indexing, health checks | [CLI Command Reference](#cli-command-reference) |
+| **MCP server** | Feeding retrieval to Claude Code, Codex, and other agents | [AutoRAG Lite MCP Server](#autorag-lite-mcp-server) |
+| **TypeScript API** | Embedding the librarian in your own app | [Programmatic usage](#3-programmatic-usage-typescript-api) |
+| **Agent skills** | Letting a coding agent install and drive AutoRAG | [For AI Agents](#-for-ai-agents--llms-setup--skills) |
+
+---
+
 ## What is AutoRAG Agent?
 
 Search tools dump file paths and matching lines — forcing you to open files, read context, and synthesize answers yourself.
@@ -51,7 +186,7 @@ Three principles drive every design decision in AutoRAG Agent:
 
 1. **Never migrate your data to search it.** Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data **in place**, querying CLI-native stores (`lazykatok`, `discrawl`, `slacrawl`, `mailcrawl`, `rclone`, `qmd`) where your data already lives. Results retain opaque, source-native identities (`/kakao/...`, `/slack/...`) that preserve local access control and privacy. *(See our [Competitive Landscape Study](docs/competitive-landscape-2026-09.md) on why in-place federation is the durable differentiator).*
 
-2. **Just works — no RAG degree required.** No pipeline tuning, no vector-DB maintenance, and no remote embedding keys required. AutoRAG Agent automatically manages [MinSync](docs/minsync-setup.md) for incremental Change Data Capture (CDC) chunking and provides a local [embedding gateway](docs/embedding-runtime.md) out of the box with zero external telemetry. A [remote rerank model](docs/rerank.md) (OpenRouter, `voyageai/rerank-3-lite` by default) is an opt-in extra.
+2. **Just works — no RAG degree required.** No pipeline tuning, no vector-DB maintenance, and no remote embedding keys required. AutoRAG Agent automatically manages [MinSync](docs/minsync-setup.md) for incremental Change Data Capture (CDC) chunking and provides a local [embedding gateway](docs/embedding-runtime.md) out of the box with zero external telemetry. A [remote rerank model](docs/rerank.md) (the hosted Dazzi Models reranker when signed in, else OpenRouter `voyageai/rerank-3-lite`) is an opt-in extra.
 
 3. **Fast by design.** Rather than coordinating slow multi-agent hierarchies, a single configured model owns the entire retrieval, direct-read, and curation loop. Local CDC chunks (BM25, vector, and hybrid modes) deliver rapid, low-latency turnaround across multi-turn research queries.
 
@@ -80,144 +215,9 @@ AutoRAG uses `@earendil-works/pi-coding-agent` as the runtime host for model ses
 
 The CLI TUI (`autorag tui`) is Pi's interactive mode hosted on the AutoRAG librarian: there is no separate AutoRAG renderer. Pi owns the terminal UI and its native in-session commands — `/login`/`/logout`, `/model`, `/resume`, `/new`, `/tree`, `/compact`, and `/settings` — so provider sign-in and model selection work on first launch even when no AutoRAG model is configured. AutoRAG registers its retrieval tools and runs the two-phase search in the same Pi session, where the model's progress notes, first answer, and final answer are ordinary assistant messages, so resume and two-phase search keep working together.
 
-Update notices are AutoRAG's, not Pi's: `autorag tui` suppresses Pi's "run `pi update`" banner (Pi is a bundled host, not a separate install) and instead checks the published `@autorag/librarian` release, showing a one-line notice in the session when a newer version exists. Set `AUTORAG_NO_UPDATE_CHECK=1` to skip the lookup; `AUTORAG_UPDATE_CHECK_URL` overrides the registry endpoint.
-
-
----
-
-## 🤖 For AI Agents & LLMs: Setup & Skills
-
-If you are an AI coding agent or LLM (Claude Code, Cursor, Windsurf, Codex, Senpi, Pi, etc.) tasked with installing, configuring, or querying AutoRAG Agent, use our built-in skills located in the [`skills/`](skills/) directory:
-
-| Skill | Directory | When to Use |
-|---|---|---|
-| **`autorag`** | [`skills/autorag/`](skills/autorag/SKILL.md) | Model-backed querying, searching, comparing, and summarizing with an already configured AutoRAG librarian. |
-| **`autorag-setup`** | [`skills/autorag-setup/`](skills/autorag-setup/SKILL.md) | Installing AutoRAG, configuring the model-backed librarian, adding roots/datasources, running health checks, and registering Lite MCP when needed. |
-| **`autorag-lite-setup`** | [`skills/autorag-lite-setup/`](skills/autorag-lite-setup/SKILL.md) | Installing/registering `autorag-mcp`, initializing model-free config, maintaining indexes, and verifying MCP search. |
-
-### Install the skills into your coding agent
-
-Skills are not auto-discovered — copy only the setup skill when an agent needs
-to bootstrap AutoRAG. **Routine AutoRAG Lite retrieval is provided by MCP
-tools, not by a `autorag-lite-search` skill or shell command.**
-
-```bash
-# From a clone of this repository
-mkdir -p .claude/skills
-cp -R skills/autorag-lite-setup .claude/skills/
-```
-
-```bash
-# From a global install — the npm package ships the same skills/ folder
-AUTORAG_SKILLS="$(npm root -g)/@autorag/librarian/skills"
-# Bun global installs live at ~/.bun/install/global/node_modules/@autorag/librarian/skills
-mkdir -p .claude/skills
-cp -R "$AUTORAG_SKILLS/autorag-lite-setup" .claude/skills/
-```
-
-Run the setup skill once to register the stdio server with the host. Reload the
-agent, then discover the actual tool names and schemas with MCP `tools/list`.
-The normal Lite path is `autorag.status` → `autorag.refresh` when needed →
-`autorag.search`; use `autorag.report` and `autorag.evidence` for the optional
-curation lifecycle. Copy `skills/autorag`
-and `skills/autorag-setup` only when the agent should also drive the
-model-backed librarian, and `skills/autorag-doctor` for diagnostics. Other
-agents read their own skill directories; copy the same setup folder there and
-reload the agent session so it picks up the MCP registration instructions.
-Do not copy or retain the removed `autorag-lite-search` skill.
-
-### Quick Agent Workflow
-
-1. **Install CLI:**
-   ```bash
-   command -v autorag >/dev/null || bun install -g @autorag/librarian
-   ```
-2. **Inspect & Preflight:**
-   ```bash
-   autorag status --json        # check corpus freshness and index readiness
-   autorag duplicates --json    # scan for exact or near-duplicate document families
-   ```
-3. **Perform Curated Search:**
-   ```bash
-   autorag search "your question" --json
-   ```
-4. **Safety Guidelines for Agents:**
-   - **Never delete, move, or modify original user documents.**
-   - AutoRAG writes index artifacts strictly into `<workspace>/.autorag/` and `.jikji/` caches.
-   - Never print or leak API keys, tokens, or credential values into stdout or logs.
-
----
-
-## AutoRAG Lite MCP Server
-
-Register the package's stdio server with the host; the host owns the process
-lifecycle and sends MCP tool calls:
-
-```bash
-AUTORAG_CONFIG=/absolute/path/to/.autorag/config.json autorag-mcp
-```
-
-For Claude Code and Codex registration commands, use
-[`skills/autorag-lite-setup/SKILL.md`](skills/autorag-lite-setup/SKILL.md).
-The MCP contract source of truth is [`src/mcp/server.ts`](src/mcp/server.ts)
-(tool registration, schemas, handlers) plus [`src/mcp/index.ts`](src/mcp/index.ts)
-(stdio entrypoint). Discover tools and schemas with MCP `tools/list`; do not
-hard-code a tool count.
-
-Core Lite tools include `autorag.status`, `autorag.search`,
-`autorag.search.files`, `autorag.datasources.list`, `autorag.datasources.get`,
-`autorag.refresh`, `autorag.report`, and `autorag.evidence`. Configured
-integrated datasources expose additional scoped search tools. Read-only MCP
-mode omits mutating tools such as refresh and report.
-
-## ⚡ AutoRAG Lite: Model-Free Retrieval Engine
-
-Need blazing fast local search without configuring an LLM or paying for API tokens? Use **AutoRAG Lite**.
-
-AutoRAG Lite provides the exact same high-performance indexing, BM25 ranking, and local MinSync vector/hybrid retrieval engine as the full librarian, but **runs 100% model-free**:
-
-- **Zero LLM Token Usage:** Run purely local BM25 and vector search offline.
-- **Agent Integration Ready:** Use the AutoRAG Lite MCP server to supply raw context chunks to an external model; the CLI remains a bootstrap and maintenance interface.
-- **Fast Local CLI:** The CLI remains available for terminal-only indexing and repair.
-
-```bash
-# Initialize a model-free workspace
-autorag lite init --search-paths ./documents
-
-# Index local files and datasources
-autorag lite refresh
-```
-
-Normal Lite retrieval runs through the MCP tools (`autorag.status`,
-`autorag.search`); the CLI remains the bootstrap and maintenance interface for
-indexing and repair.
-
----
-
-## 🔌 Supported Datasources
-
-AutoRAG Agent connects to external tools and communication platforms using dedicated datasource skills. Data remains in each tool's native store — AutoRAG does not copy or ingest foreign databases into a centralized store:
+Typing while a search runs works like steering one of Pi's own turns. **Enter** steers the running search: AutoRAG treats the text as a correction of a search that is going wrong, delivers it before the search's next model request (also between phases, where no turn is streaming), cuts in-flight retrieval short so the correction lands promptly, and always runs the verification phase so the answer reflects the correction. **Alt+Enter** queues a new question that runs after the current search answers. **Esc** stops the search instead of letting it continue into its next phase. Pi's registered `/` commands still run during a search; any other line starting with `/` (a path, for example) is a steer like any other text. A query typed after a search answers is a new search in the same session, as before.
 
-| Datasource | Skill Alias | Backend / Driver | Storage & Privacy Model | Search Capabilities |
-|---|---|---|---|---|
-| **Local Documents** | `local` | Native filesystem & MinSync | Workspace-local parsed mirrors (`.autorag/`) | BM25, Semantic, Hybrid |
-| **KakaoTalk** | `kakao` | [`lazykatok`](https://github.com/changeroa/lazykatok) CLI | Native KakaoTalk archive; zero direct DB access | BM25, Semantic, Hybrid |
-| **Discord** | `discord` | [`discrawl`](https://github.com/openclaw/discrawl) CLI | Native SQLite archive; token-free wiretap mode | BM25, Semantic, Hybrid |
-| **WhatsApp** | `whatsapp` | [`wacrawl`](https://github.com/openclaw/wacrawl) CLI | Local-first incremental archive + FTS5 | Lexical FTS5 |
-| **Telegram** | `telegram` | [`telecrawl`](https://github.com/openclaw/telecrawl) CLI | Local-first desktop archive + FTS5 | Lexical FTS5 |
-| **Slack** | `slack` | [`slacrawl`](https://github.com/openclaw/slacrawl) CLI | Local workspace/channel/thread archive + FTS5 | Lexical FTS5 |
-| **Notion** | `notion` | [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | Local page/database/block archive + FTS5 | Lexical FTS5 |
-| **Email Archives** | `mailcrawl` | [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | Local Gmail, IMAP, and Maildir storage | BM25, Semantic, Hybrid |
-| **Local Mail Export**| `mail-export` | Built-in `.mbox` / `.eml` parser | Local filesystem mailboxes | Lexical |
-| **Obsidian Vaults** | `obsidian` | [`qmd`](https://github.com/tobi/qmd) CLI | Direct markdown vault indexing | BM25, Semantic |
-| **GitHub** | `github` | GitHub REST API | In-memory fetched Issues and Pull Requests | Lexical, Scoped |
-| **GitHub Gists** | `github-gist` | GitHub REST API | Incremental local index of own-account public and permitted secret Gist content/metadata; token not stored | Lexical (BM25), Local Semantic, Scoped |
-| **Cloud Drives** | `cloud-drive` | [`rclone`](https://rclone.org) CLI | Google Drive (Tier-1), OneDrive, Dropbox, etc. | Incremental Mirror + BM25 |
-| **Photos & Shots** | `clawgallery` | `clawgallery` CLI | Local screenshot and photo store | Hybrid OCR/Visual Search |
-| **RSS / News** | `rss` | Native HTTP Poller | RSS 2.0 & Atom feeds (24h deduplication) | Lexical |
-| **macOS Spotlight** | `spotlight` | Native `mdfind` CLI | macOS system metadata and content index | System Native |
-
-For configuration syntax and connector details, see [docs/datasource-skills.md](docs/datasource-skills.md). Non-interactive `sync`/`index` steps get a 30-minute per-connector budget (`connector.indexTimeoutMs`) because first-run imports routinely take minutes; interactive search keeps its 60-second default.
+Update notices are AutoRAG's, not Pi's: `autorag tui` suppresses Pi's "run `pi update`" banner (Pi is a bundled host, not a separate install) and instead checks the published `@autorag/librarian` release, showing a one-line notice in the session when a newer version exists. Set `AUTORAG_NO_UPDATE_CHECK=1` to skip the lookup; `AUTORAG_UPDATE_CHECK_URL` overrides the registry endpoint.
 
 ---
 
@@ -304,6 +304,26 @@ Tesseract. Set `parserOptions.ocr.cachePath` to store the Tesseract models
 elsewhere, or `KORDOC_OFFLINE=1` to block the built-in model download. An
 explicit `parserOptions.kordoc.ocr` overrides the language-based choice.
 
+### Bounding and cancelling a parse
+
+`kordoc` (PDF, HWP/HWPX, DOCX, XLSX/XLS, images), `.pptx` and `.eml` parse in a worker
+thread during a refresh, so one corrupt or hostile file cannot hang or crash the run. A
+`Promise.race` deadline cannot stop a synchronous parse; terminating the worker can.
+
+| `parserOptions` key | Default | Effect |
+|---|---|---|
+| `fileTimeoutMs` | `120000` | Per-file deadline. A parse still running is killed with its worker. |
+| `maxHeapMb` | Node's default | V8 old-generation limit for the worker. An out-of-memory parse is caught, not a host crash. |
+
+A timeout, a heap overrun and a worker crash each become one `parser-failed` skip for that
+file; the diagnostic message says which. The rest of the corpus is still indexed. Plain-text
+files and parsers configured with a function (`parserOptions.kordoc.parse`, a custom OCR
+provider) parse in-process.
+
+```json
+{ "parserOptions": { "fileTimeoutMs": 60000, "maxHeapMb": 2048 } }
+```
+
 Photos and videos are also reachable through two other paths that are not part
 of this switch: the `clawgallery` datasource (captions and visual search over its
 own folders) and Jikji's media index (`jikji.enableMediaIndex`, file facts such as
@@ -357,80 +377,274 @@ for (const result of response.results) {
 }
 ```
 
+A caller can steer a search while it runs, the way Enter steers a TUI search, and stop it:
+
+```typescript
+const run = (async () => {
+  for await (const event of agent.searchDocumentsStream("When is the launch?")) {
+    if (event.type === "preliminary") console.log("Quick:", event.response.answer);
+    if (event.type === "complete") console.log("Final:", event.response.answer);
+  }
+})();
+
+// A correction of the running search, not a new question. false = no search to steer.
+agent.steer("Use the June plan, not the April one.");
+// agent.abort() stops the whole search, including every later phase.
+await run;
+```
+
+`steer()` reaches the search's next model request (also between phases), cuts in-flight retrieval and the baseline prefetch short, and always runs verification, whose prompt restates the correction. It returns `false` when no search is running, the search was stopped, or it is past its last model request; ask the text as a new question then.
+
+A refresh can be cancelled and observed from code:
+
+```typescript
+const controller = new AbortController();
+try {
+  await agent.refresh(false, {
+    signal: controller.signal, // controller.abort() stops it
+    onProgress: ({ phase, processed, total, currentVirtualPath }) => {
+      console.log(`${phase} ${processed}/${total}`, currentVirtualPath ?? "");
+    },
+  });
+} catch (error) {
+  if (controller.signal.aborted) console.log("refresh cancelled");
+  else throw error;
+}
+```
+
+`signal` is checked between files and between phases, kills an in-flight parse worker, and
+terminates the MinSync, Jikji and datasource child processes (and their descendants). The
+refresh then rejects with the signal's reason and is recorded as `failed`; files already
+mirrored are kept and not re-parsed on the next run. `onProgress` reports `parsed`
+(`processed`/`total` are files, `currentVirtualPath` is the file about to be processed) and
+`datasources` (`processed`/`total` are datasource skills). During the parse phase
+`.autorag/refresh-progress.json` also carries the running `sourceFiles.total` and
+`parsedCounts`. Without these options a refresh behaves as before.
+
+---
+
+## CLI Command Reference
+
+| Command | Description |
+|---|---|
+| `autorag init` | Initialize `~/.autorag/config.json` with search roots, document languages, and model settings |
+| `autorag refresh` | Refresh parsed mirrors, MinSync CDC chunks, datasources, Jikji, and the platform file-name index (Everything on Windows, FSearch on macOS/Linux) |
+| `autorag search "<query>"` | Run the librarian agent to curate structured answers |
+| `autorag status` | Inspect corpus freshness, indexing status, and vector readiness |
+| `autorag health` | Check model provider authentication, token validity, and API reachability |
+| `autorag models list` | List chat models the pi runtime can resolve (built-ins, `models.json`, custom/extension providers) with provider auth status; never prints credential values |
+| `autorag update-check` | Compare the running `autorag` against the published npm version (also runs on `autorag tui` launch) |
+| `autorag tui` | Open Pi's interactive librarian TUI (`/login`, `/model`, `/resume`, …) |
+| `autorag duplicates [DIR]` | Read-only scan for exact and near-duplicate document families with `dupey` |
+| `autorag lite ...` | CLI bootstrap, indexing repair, terminal maintenance, and fallback interface; agents use MCP for normal Lite operation |
+| `autorag memory inspect` | Render a read-only, path-opaque snapshot of retrieval memory (judged evidence, curated results, evidence chunks, insights); model-free |
+| `autorag evidence <session>` | Inspect persisted evidence behind numbered results; MCP clients normally use `autorag.evidence`, and the CLI stays available for terminal maintenance |
+| `autorag serve` | Start the P2P query server over SimpleX (opt-in) |
+| `autorag p2p ...` | Manage peer trust, query approvals, and sharing policies |
+
+---
+
+## AutoRAG Lite MCP Server
+
+Register the package's stdio server with the host; the host owns the process
+lifecycle and sends MCP tool calls:
+
+```bash
+AUTORAG_CONFIG=/absolute/path/to/.autorag/config.json autorag-mcp
+```
+
+For Claude Code and Codex registration commands, use
+[`skills/autorag-lite-setup/SKILL.md`](skills/autorag-lite-setup/SKILL.md).
+The MCP contract source of truth is [`src/mcp/server.ts`](src/mcp/server.ts)
+(tool registration, schemas, handlers) plus [`src/mcp/index.ts`](src/mcp/index.ts)
+(stdio entrypoint). Discover tools and schemas with MCP `tools/list`; do not
+hard-code a tool count.
+
+Core Lite tools include `autorag.status`, `autorag.search`,
+`autorag.search.files`, `autorag.datasources.list`, `autorag.datasources.get`,
+`autorag.refresh`, `autorag.report`, and `autorag.evidence`. Configured
+integrated datasources expose additional scoped search tools. Read-only MCP
+mode omits mutating tools such as refresh and report.
+
+---
+
+## ⚡ AutoRAG Lite: Model-Free Retrieval Engine
+
+Need blazing fast local search without configuring an LLM or paying for API tokens? Use **AutoRAG Lite**.
+
+AutoRAG Lite provides the exact same high-performance indexing, BM25 ranking, and local MinSync vector/hybrid retrieval engine as the full librarian, but **runs 100% model-free**:
+
+- **Zero LLM Token Usage:** Run purely local BM25 and vector search offline.
+- **Agent Integration Ready:** Use the AutoRAG Lite MCP server to supply raw context chunks to an external model; the CLI remains a bootstrap and maintenance interface.
+- **Fast Local CLI:** The CLI remains available for terminal-only indexing and repair.
+
+```bash
+# Initialize a model-free workspace
+autorag lite init --search-paths ./documents
+
+# Index local files and datasources
+autorag lite refresh
+```
+
+Normal Lite retrieval runs through the MCP tools (`autorag.status`,
+`autorag.search`); the CLI remains the bootstrap and maintenance interface for
+indexing and repair.
+
+---
+
+## 🤖 For AI Agents & LLMs: Setup & Skills
+
+If you are an AI coding agent or LLM (Claude Code, Cursor, Windsurf, Codex, Senpi, Pi, etc.) tasked with installing, configuring, or querying AutoRAG Agent, use our built-in skills located in the [`skills/`](skills/) directory:
+
+| Skill | Directory | When to Use |
+|---|---|---|
+| **`autorag`** | [`skills/autorag/`](skills/autorag/SKILL.md) | Model-backed querying, searching, comparing, and summarizing with an already configured AutoRAG librarian. |
+| **`autorag-setup`** | [`skills/autorag-setup/`](skills/autorag-setup/SKILL.md) | Installing AutoRAG, configuring the model-backed librarian, adding roots/datasources, running health checks, and registering Lite MCP when needed. |
+| **`autorag-lite-setup`** | [`skills/autorag-lite-setup/`](skills/autorag-lite-setup/SKILL.md) | Installing/registering `autorag-mcp`, initializing model-free config, maintaining indexes, and verifying MCP search. |
+
+### Install the skills into your coding agent
+
+Skills are not auto-discovered — copy only the setup skill when an agent needs
+to bootstrap AutoRAG. **Routine AutoRAG Lite retrieval is provided by MCP
+tools, not by a `autorag-lite-search` skill or shell command.**
+
+```bash
+# From a clone of this repository
+mkdir -p .claude/skills
+cp -R skills/autorag-lite-setup .claude/skills/
+```
+
+```bash
+# From a global install — the npm package ships the same skills/ folder
+AUTORAG_SKILLS="$(npm root -g)/@autorag/librarian/skills"
+# Bun global installs live at ~/.bun/install/global/node_modules/@autorag/librarian/skills
+mkdir -p .claude/skills
+cp -R "$AUTORAG_SKILLS/autorag-lite-setup" .claude/skills/
+```
+
+Run the setup skill once to register the stdio server with the host. Reload the
+agent, then discover the actual tool names and schemas with MCP `tools/list`.
+The normal Lite path is `autorag.status` → `autorag.refresh` when needed →
+`autorag.search`; use `autorag.report` and `autorag.evidence` for the optional
+curation lifecycle. Copy `skills/autorag`
+and `skills/autorag-setup` only when the agent should also drive the
+model-backed librarian, and `skills/autorag-doctor` for diagnostics. Other
+agents read their own skill directories; copy the same setup folder there and
+reload the agent session so it picks up the MCP registration instructions.
+Do not copy or retain the removed `autorag-lite-search` skill.
+
+### Quick Agent Workflow
+
+1. **Install CLI:**
+   ```bash
+   command -v autorag >/dev/null || bun install -g @autorag/librarian
+   ```
+2. **Inspect & Preflight:**
+   ```bash
+   autorag status --json        # check corpus freshness and index readiness
+   autorag duplicates --json    # scan for exact or near-duplicate document families
+   ```
+3. **Perform Curated Search:**
+   ```bash
+   autorag search "your question" --json
+   ```
+4. **Safety Guidelines for Agents:**
+   - **Never delete, move, or modify original user documents.**
+   - AutoRAG writes index artifacts into `<workspace>/.autorag/` and Jikji's central store (`<OS data dir>/jikji/`, override with `JIKJI_DATA_DIR`). Jikji also drops a `.jikji_agent_map.md` guide into each indexed root.
+   - Never print or leak API keys, tokens, or credential values into stdout or logs.
+
+---
+
+## 🔌 Supported Datasources
+
+AutoRAG Agent connects to external tools and communication platforms using dedicated datasource skills. Data remains in each tool's native store — AutoRAG does not copy or ingest foreign databases into a centralized store:
+
+| Datasource | Skill Alias | Backend / Driver | Storage & Privacy Model | Search Capabilities |
+|---|---|---|---|---|
+| **Local Documents** | `local` | Native filesystem & MinSync | Workspace-local parsed mirrors (`.autorag/`) | BM25, Semantic, Hybrid |
+| **KakaoTalk** | `kakao` | [`lazykatok`](https://github.com/changeroa/lazykatok) CLI | Native KakaoTalk archive; zero direct DB access | BM25, Semantic, Hybrid |
+| **Discord** | `discord` | [`discrawl`](https://github.com/openclaw/discrawl) CLI | Native SQLite archive; token-free wiretap mode | BM25, Semantic, Hybrid |
+| **WhatsApp** | `whatsapp` | [`wacrawl`](https://github.com/openclaw/wacrawl) CLI | Local-first incremental archive + FTS5 | Lexical FTS5; BM25 + Semantic + Hybrid by default via [`sidecar`](docs/datasource-skills.md#hybrid-search-for-chat-crawlers-sidecar) |
+| **Telegram** | `telegram` | [`telecrawl`](https://github.com/openclaw/telecrawl) CLI | Local-first desktop archive + FTS5 | Lexical FTS5; BM25 + Semantic + Hybrid by default via [`sidecar`](docs/datasource-skills.md#hybrid-search-for-chat-crawlers-sidecar) |
+| **Slack** | `slack` | [`slacrawl`](https://github.com/openclaw/slacrawl) CLI | Local workspace/channel/thread archive + FTS5 | Lexical FTS5; BM25 + Semantic + Hybrid by default via [`sidecar`](docs/datasource-skills.md#hybrid-search-for-chat-crawlers-sidecar) |
+| **Notion** | `notion` | [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | Local page/database/block archive + FTS5 | Lexical FTS5 |
+| **Email Archives** | `mailcrawl` | [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | Local Gmail, IMAP, and Maildir storage | BM25, Semantic, Hybrid |
+| **Local Mail Export**| `mail-export` | Built-in `.mbox` / `.eml` parser | Local filesystem mailboxes | Lexical |
+| **Obsidian Vaults** | `obsidian` | [`qmd`](https://github.com/tobi/qmd) CLI | Direct markdown vault indexing | BM25, Semantic |
+| **GitHub** | `github` | GitHub REST API | In-memory fetched Issues and Pull Requests | Lexical, Scoped |
+| **GitHub Gists** | `github-gist` | GitHub REST API | Incremental local index of own-account public and permitted secret Gist content/metadata; token not stored | Lexical (BM25), Local Semantic, Scoped |
+| **Cloud Drives** | `cloud-drive` | [`rclone`](https://rclone.org) CLI | Google Drive (Tier-1), OneDrive, Dropbox, etc. | Incremental Mirror + BM25 |
+| **Photos & Shots** | `clawgallery` | `clawgallery` CLI | Local screenshot and photo store | Hybrid OCR/Visual Search |
+| **RSS / News** | `rss` | Native HTTP Poller | RSS 2.0 & Atom feeds (24h deduplication) | Lexical |
+| **macOS Spotlight** | `spotlight` | Native `mdfind` CLI | macOS system metadata and content index | System Native |
+
+For configuration syntax and connector details, see [docs/datasource-skills.md](docs/datasource-skills.md). Non-interactive `sync`/`index` steps get a 30-minute per-connector budget (`connector.indexTimeoutMs`) because first-run imports routinely take minutes; interactive search keeps its 60-second default.
+
 ---
 
-## Hosted AutoRAG Plan (Dazzi)
+## Hosted Dazzi Models Plan
 
-AutoRAG can run on the hosted **AutoRAG** plan as a first-class model provider named
-`autorag`. The provider speaks the server's OpenAI Responses API surface (`<root>/v1`),
+AutoRAG can run on the hosted **Dazzi Models** plan as a first-class model provider named
+`dazzi`. The provider speaks the server's OpenAI Responses API surface (`<root>/v1`),
 so streaming, tool calls, and usage are handled by Pi's built-in `openai-responses`
 implementation, and the model catalog is discovered live from the server — no model id
 is baked into AutoRAG.
 
-Sign in from the interactive TUI:
+Sign in from the interactive TUI. Login is the only way to authenticate; there is no API-key
+environment variable:
 
 ```text
 autorag tui
-/login          # choose "AutoRAG"; the browser completes a PKCE login
+/login          # choose "Dazzi Models"; the browser completes a PKCE login
 ```
 
-Headless (no browser) with a long-lived API key:
-
-```bash
-export AUTORAG_API_KEY=dz_...
-```
+The provider, Jev, and rerank all use the credential `/login` stores. A login flow for
+machines with no browser is not available yet and is tracked in [#1913](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1913).
 
 On a fresh agent home, pin the model with an explicit endpoint so it resolves
-offline without a catalog snapshot — this is the headless configuration:
+offline without a catalog snapshot:
 
 ```json
 {
   "model": {
-    "provider": "autorag",
+    "provider": "dazzi",
     "id": "anthropic/claude-haiku-5.5",
     "baseUrl": "https://api.dazziapp.com/v1",
-    "api": "openai-responses",
-    "apiKeyEnv": "AUTORAG_API_KEY"
+    "api": "openai-responses"
   }
 }
 ```
 
-The `provider`/`id`-only shorthand (`{ "provider": "autorag", "id": "anthropic/claude-haiku-5.5" }`)
+The `provider`/`id`-only shorthand (`{ "provider": "dazzi", "id": "anthropic/claude-haiku-5.5" }`)
 only resolves once a TUI `/login` or a model refresh has persisted the catalog
 snapshot; with no snapshot there is nothing to restore offline, and resolution
-fails with `Unknown configured model: autorag/<id>`.
+fails with `Unknown configured model: dazzi/<id>`.
 
 | Variable | Default | Meaning |
 |---|---|---|
 | `AUTORAG_BASE_URL` | `https://api.dazziapp.com` | Server **root** (not the `/v1` path); the API base is `<root>/v1`. |
-| `AUTORAG_API_KEY` | – | Headless API key. A stored `/login` credential takes precedence. |
 
 The catalog is persisted after a `/login` (or any model refresh), so the
-`provider`/`id` shorthand `model: { provider: "autorag", id }`
+`provider`/`id` shorthand `model: { provider: "dazzi", id }`
 keeps resolving offline. When you are not signed in the provider still registers — it appears under
 `/login`, contributes no models, and makes no network call. See
-[Hosted AutoRAG provider](docs/cloud-provider.md) for the full flow.
+[Hosted Dazzi Models provider](docs/cloud-provider.md) for the full flow.
 
----
+The same sign-in can also serve the **Jev** judgment model and **rerank** stage, so
+the whole librarian loop runs on the hosted account:
 
-## CLI Command Reference
+```json
+{
+  "jev": { "backend": "dazzi", "model": "jev-1.13" },
+  "rerank": { "provider": "dazzi", "model": "qwen/qwen3-reranker-8b" }
+}
+```
 
-| Command | Description |
-|---|---|
-| `autorag init` | Initialize `~/.autorag/config.json` with search roots, document languages, and model settings |
-| `autorag refresh` | Refresh parsed mirrors, MinSync CDC chunks, datasources, Jikji, and the platform file-name index (Everything on Windows, FSearch on macOS/Linux) |
-| `autorag search "<query>"` | Run the librarian agent to curate structured answers |
-| `autorag status` | Inspect corpus freshness, indexing status, and vector readiness |
-| `autorag health` | Check model provider authentication, token validity, and API reachability |
-| `autorag models list` | List chat models the pi runtime can resolve (built-ins, `models.json`, custom/extension providers) with provider auth status; never prints credential values |
-| `autorag update-check` | Compare the running `autorag` against the published npm version (also runs on `autorag tui` launch) |
-| `autorag tui` | Open Pi's interactive librarian TUI (`/login`, `/model`, `/resume`, …) |
-| `autorag duplicates [DIR]` | Read-only scan for exact and near-duplicate document families with `dupey` |
-| `autorag lite ...` | CLI bootstrap, indexing repair, terminal maintenance, and fallback interface; agents use MCP for normal Lite operation |
-| `autorag memory inspect` | Render a read-only, path-opaque snapshot of retrieval memory (judged evidence, curated results, evidence chunks, insights); model-free |
-| `autorag evidence <session>` | Inspect persisted evidence behind numbered results; MCP clients normally use `autorag.evidence`, and the CLI stays available for terminal maintenance |
-| `autorag serve` | Start the P2P query server over SimpleX (opt-in) |
-| `autorag p2p ...` | Manage peer trust, query approvals, and sharing policies |
+`jev.backend: "dazzi"` posts Jev's native dialect to `<root>/v1/jev`
+(default model `jev-1.13`); `rerank.provider: "dazzi"` reranks against
+`<root>/v1/rerank` (default model `qwen/qwen3-reranker-8b`). Both use the stored
+`/login` credential, and an explicit `rerank.baseUrl` or `rerank.model` still overrides
+the defaults.
 
 ---
 
@@ -438,11 +652,11 @@ keeps resolving offline. When you are not signed in the provider still registers
 
 Deep dive into AutoRAG Agent's architecture, security, and integration guides:
 
-- **[MinSync Setup & Embedding QA](docs/minsync-setup.md):** Automatic binary installation, CDC chunking, and EmbeddingGemma verification.
-- **[Local Embedding Runtime & Gateway](docs/embedding-runtime.md):** AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search. Model cards: [`qwen3-embedding-0.6b`](docs/model-cards/qwen3-embedding-0.6b.md), [`embeddinggemma-300m`](docs/model-cards/embeddinggemma-300m.md).
-- **[Hosted AutoRAG Provider](docs/cloud-provider.md):** The `autorag` (Dazzi) model provider — PKCE `/login`, `AUTORAG_API_KEY`, `AUTORAG_BASE_URL`, live model discovery, and offline catalog snapshots.
+- **[MinSync Setup & Embedding QA](docs/minsync-setup.md):** Automatic binary installation, CDC chunking, and local embedding verification.
+- **[Local Embedding Runtime & Gateway](docs/embedding-runtime.md):** AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search. Model card: [`qwen3-embedding-0.6b`](docs/model-cards/qwen3-embedding-0.6b.md).
+- **[Hosted Dazzi Models Provider](docs/cloud-provider.md):** The hosted `dazzi` model provider — PKCE `/login`, `AUTORAG_BASE_URL`, live model discovery, and offline catalog snapshots.
 - **[Datasource Skills Reference](docs/datasource-skills.md):** Full configuration contracts, connection aliases, and connector options.
-- **[Jev Decisions](docs/jev-decisions.md):** Jev query pipeline, **on by default** via OpenRouter: it routes each question to a direct answer (intrinsic knowledge) or local search (everything needing more information; web search is left to the agent after the fast answer), splits multi-part questions into up to five parallel searches (`openrouter/qwen/qwen3.7-flash`), picks which registered datasources to search before the fast answer, and ends the run after the fast answer when that answer is complete. Also covers the `jev` judgment tool, backends (OpenRouter, TypeSafe, Vercel AI Gateway), and how to opt out.
+- **[Jev Decisions](docs/jev-decisions.md):** Jev query pipeline, **on by default** via Dazzi Models when signed in (OpenRouter otherwise): it routes each question to a direct answer (intrinsic knowledge) or local search (everything needing more information; web search is left to the agent after the fast answer), splits multi-part questions into up to five parallel searches (`dazzi/openai/gpt-6-luna` when signed in, else `openrouter/qwen/qwen3.7-flash`), picks which registered datasources to search before the fast answer, and ends the run after the fast answer when that answer is complete. Also covers the `jev` judgment tool, backends (OpenRouter, TypeSafe, Vercel AI Gateway), and how to opt out.
 - **[Manual QA & Datasource Test Harnesses](docs/manual-qa-datasources.md):** Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.
 - **[P2P SimpleX Sharing & Path Standard](docs/p2p-path-standard.md):** Decentralized peer query sharing with SimpleX, PII redaction, and approval queues.
 - **[Supply Chain Security & License Audits](docs/supply-chain.md):** Software bill of materials (SBOM) and dependency gate policies.
@@ -512,14 +726,17 @@ Common failures and their fix:
 | A datasource errors during refresh | `datasource-index-failed` | run that CLI's own `doctor` |
 | MinSync binary missing (search and refresh fail with `MinSync is required ...`) | — | `cargo install minsync`, or leave `minSync.autoInstall` on and re-run refresh |
 | Jikji missing | `jikji-unavailable` | check the Rust toolchain, re-run refresh |
+| Jikji prepare of a large root times out (`the Jikji prepare run timed out`) | `jikji-prepare-failed` | the refresh still finishes and the other roots are indexed. There is no fixed total wall: the per-root hang guard is `jikji.timeoutMs` (60 s base) plus `jikji.prepareTimeoutMsPerFile` ms per file under the root (default: `jikji.parseTimeout`, 5 s), uncapped, so a huge root is not cut by its size and only a run stuck beyond every file hitting its parse limit is killed. Set `prepareTimeoutMsPerFile: 0` for a flat `timeoutMs` wall. `jikji.parseTimeout` (seconds) is the per-file limit, forwarded to `jikji prepare --parse-timeout`; jikji skips a file that exceeds it. A timeout kills jikji's whole process group |
 | Windows file-name search fails during refresh | `everything-index-failed` | read the ES exit code and stderr in the message, then `autorag refresh --method everything --json` |
-| Every search does a full local search and verification, even for small talk | `query-route-fallback` | set `OPENROUTER_API_KEY` (Jev routing and decomposition are on by default) |
+| Every search does a full local search and verification, even for small talk | `query-route-fallback` | sign in to Dazzi Models with `/login`, or set `OPENROUTER_API_KEY` (Jev routing and decomposition are on by default) |
 
 Native datasource stores stay owned by their CLIs — AutoRAG never rebuilds them. Fix a broken archive with `lazykatok doctor`, `discrawl --json metadata`, `slacrawl --json doctor`, `wacrawl --json doctor`, `telecrawl --json doctor`, `notcrawl doctor`, `qmd status`, or `mailcrawl doctor`, then re-run `autorag refresh --method datasources --json`.
 
+---
+
 ## License
 
-- **AutoRAG 2.0 (AutoRAG Agent):** Released under the [MIT License](LICENSE).
+- **AutoRAG 2.0 (AutoRAG Agent):** Released under the [Apache License 2.0](LICENSE).
 - **Legacy Python AutoRAG (`legacy/`):** Released under the [Apache License 2.0](legacy/LICENSE).
 - Production third-party notices and licenses are documented in [`NOTICE`](NOTICE).
 - The bundled Windows binaries keep their own licenses: Everything (MIT, plus PCRE BSD-3-Clause) in [`licenses/everything-MIT-and-PCRE-BSD.txt`](licenses/everything-MIT-and-PCRE-BSD.txt) and ES (MIT) in [`licenses/es-MIT.txt`](licenses/es-MIT.txt). Everything's source code is not public; AutoRAG redistributes the unmodified voidtools portable ZIPs.