@autorag/librarian 2.5.1 → 2.5.3

package/README.md

@@ -1,591 +1,331 @@
-# AutoRAG
+# AutoRAG Agent
 
-**A self-evolving librarian agent for document collections.**
+**Now your agent can find anything in your computer.**
+
+<p align="left">
+  <a href="https://www.npmjs.com/package/@autorag/librarian"><img src="https://img.shields.io/npm/v/@autorag/librarian.svg?style=flat-square&color=38bdf8" alt="npm version" /></a>
+  <a href="https://www.npmjs.com/package/@autorag/librarian"><img src="https://img.shields.io/npm/dm/@autorag/librarian.svg?style=flat-square&color=818cf8" alt="npm downloads" /></a>
+  <a href="https://github.com/Marker-Inc-Korea/AutoRAG/stargazers"><img src="https://img.shields.io/github/stars/Marker-Inc-Korea/AutoRAG?style=flat-square&logo=github&color=f59e0b" alt="GitHub stars" /></a>
+  <a href="https://github.com/Marker-Inc-Korea/AutoRAG/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/Marker-Inc-Korea/AutoRAG/ci.yml?branch=main&label=CI&style=flat-square" alt="CI Status" /></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg?style=flat-square" alt="License: MIT" /></a>
+  <img src="https://img.shields.io/badge/node-%3E%3D24.0.0-informational.svg?style=flat-square" alt="Node >= 24" />
+  <img src="https://img.shields.io/badge/bun-%3E%3D1.2-black.svg?style=flat-square&logo=bun" alt="Bun >= 1.2" />
+</p>
+
+<p align="center">
+  <img src="assets/autorag-agent-social-preview.png" alt="AutoRAG Agent" width="100%" />
+</p>
 
 > [!IMPORTANT]
 > **Looking for the original AutoRAG (RAG AutoML / pipeline optimization tool)?**
 > This repository now hosts **AutoRAG 2.0**, a complete reimagining of AutoRAG as a self-evolving librarian agent. The original Python-based AutoRAG — the RAG AutoML tool for automatically finding an optimal RAG pipeline for your data — now lives in the [`legacy/`](legacy/) directory of this repository.
 >
-> **The legacy AutoRAG is NOT abandoned.** It continues to be maintained (bug fixes, dependency updates, and PyPI releases via `pip install AutoRAG`) in maintenance mode. Existing users can keep using it exactly as before — see the [legacy README](legacy/README.md) for its documentation, and file issues in this repository as usual. New feature development is focused on AutoRAG 2.0.
+> **The legacy AutoRAG is NOT abandoned.** It continues to be maintained (bug fixes, dependency updates, and PyPI releases via `pip install AutoRAG`) in maintenance mode. Existing users can keep using it exactly as before — see the [legacy README](legacy/README.md) for its documentation, and file issues in this repository as usual. New feature development is focused on AutoRAG Agent (2.0).
 
-AutoRAG searches your PDFs, wikis, notes, research papers, and knowledge bases — then curates the results into clean, numbered knowledge units. No raw grep dumps. Just answers.
+---
 
-AutoRAG is a customized [Pi](https://github.com/earendil-works/pi-mono) agent — the Pi agent loop configured into a librarian. The AutoRAG librarian retrieves candidates, reads source files directly, judges the evidence, and curates the structured answer. Its model and provider come from the user's authenticated runtime; AutoRAG does not ship a private provider default.
+## What is AutoRAG Agent?
 
-AutoRAG itself is the specialized search agent, not a coordinator for other
-model roles. You configure one model, and that model owns the complete
-retrieval, reading, judgment, and curation loop.
+Search tools dump file paths and matching lines — forcing you to open files, read context, and synthesize answers yourself.
 
-## Core values
+**AutoRAG Agent is a self-evolving librarian agent.** Built on the [Pi](https://github.com/earendil-works/pi-mono) agent framework, a single configured model searches across multiple retrieval methods, opens source documents directly via `bash` to verify ground truth, and curates answers into clean, numbered knowledge units:
 
-Three principles drive every design decision:
-
-1. **Never migrate your data to search it.** AutoRAG federates CLI-owned
-   stores (katok, discrawl, qmd, msgvault, rclone, …) in place. No forced
-   ingestion into a central index, no third-party server holding a copy of
-   your corpus. Results carry source-native identities
-   (`/kakao/<instance>/chunks/<chunk>`) with scope-checked access — see the
-   [competitive landscape study](docs/competitive-landscape-2026-09.md) for
-   why this is the durable differentiator.
-2. **Just works — no RAG degree required.** Install it and it works: minimal
-   configuration, no pipeline tuning, no vector-DB operations, no API keys.
-   The local embedder (EmbeddingGemma via Ollama) and MinSync auto-install
-   handle the "RAG plumbing" invisibly.
-3. **Fast by design.** One configured model owns the whole loop; retrieval
-   runs locally over MinSync CDC chunks (BM25 / vector / hybrid); interactive
-   search is optimized for low latency.
+```text
+You ask:  "What changes were made to the cloud infrastructure contract in Q3?"
 
-## Why AutoRAG
+AutoRAG Agent:
+[1] Enterprise Compute Discount — AWS annual discount tier increased to 28% based on commitment volume. (contracts/cloud-2026.pdf:p.4)
+[2] Regional Data Residency — Explicit compliance clause pinning customer data storage to the Seoul AWS region. (/slack/legal-ops/chunks/318)
+[3] SLA Guarantee — Guaranteed uptime threshold adjusted to 99.95% with penalty credits starting at 15 minutes downtime. (contracts/sla-addendum.md:lines 45-62)
+```
 
-### The problem with search tools
+---
 
-Every search tool gives you the same thing: a list of file paths and matching lines. Then *you* have to:
-- Open each file
-- Read the surrounding context
-- Decide what's relevant
-- Synthesize an answer
-- Remember what worked for next time
+## Core Values
 
-That's the human doing all the hard work. The tool just points.
+<p align="center">
+  <img src="assets/autorag-agent-principles.png" alt="AutoRAG Agent Core Principles" width="100%" />
+</p>
 
-### AutoRAG does the hard work
+Three principles drive every design decision in AutoRAG Agent:
 
-AutoRAG is not a search tool. It's a **librarian** — it searches, reads, thinks, and reports back:
+1. **Never migrate your data to search it.** Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data **in place**, querying CLI-native stores (`lazykatok`, `discrawl`, `slacrawl`, `mailcrawl`, `rclone`, `qmd`) where your data already lives. Results retain opaque, source-native identities (`/kakao/...`, `/slack/...`) that preserve local access control and privacy. *(See our [Competitive Landscape Study](docs/competitive-landscape-2026-09.md) on why in-place federation is the durable differentiator).*
 
-```
-You ask:  "What were the key findings in the Q3 report?"
+2. **Just works — no RAG degree required.** No pipeline tuning, no vector-DB maintenance, and no remote embedding API keys. AutoRAG Agent automatically manages [MinSync](docs/minsync-setup.md) for incremental Change Data Capture (CDC) chunking and provides a local [embedding gateway](docs/embedding-runtime.md) out of the box with zero external telemetry.
 
-AutoRAG:
-[1] Revenue grew 23% YoY to $4.2M, driven by enterprise contracts. (pages 3-5)
-[2] Three new risk factors: supply chain, regulatory, talent retention. (pages 12-14)
-[3] Headcount target missed by 12 — engineering hiring bottleneck. (page 8)
-```
+3. **Fast by design.** Rather than coordinating slow multi-agent hierarchies, a single configured model owns the entire retrieval, direct-read, and curation loop. Local CDC chunks (BM25, vector, and hybrid modes) deliver rapid, low-latency turnaround across multi-turn research queries.
 
-No file paths. No line numbers. Just curated knowledge you can act on.
+---
 
-### It gets smarter over time
+## Architecture & How It Works
 
-AutoRAG has a **self-evolving memory system**. Every search teaches it something:
+AutoRAG Agent orchestrates five integrated subsystems:
 
-- Which retrieval methods work for which types of queries
-- Which document areas are most productive
-- What the caller found useful (via explicit feedback)
+1. **Self-Evolving Memory System (`check_memory`):** Before querying, the agent consults historical outcomes stored in `~/.autorag/memory.json` to prioritize retrieval methods that have proven successful for similar queries.
+2. **Pluggable Multi-Method Retrieval:**
+   - **BM25 Lexical Search:** Fast keyword ranking via MinSync over parsed markdown mirrors.
+   - **Semantic Vector Search:** Dense vector retrieval over CDC chunks via the built-in embedding gateway.
+   - **Hybrid Search:** Combines BM25 and vector ranking via Reciprocal Rank Fusion (RRF).
+   - **Jikji Find-First Discovery:** Local CLI-backed fast discovery answer packs.
+   - **Datasource Skills:** Server-authorized federated retrieval across external applications.
+3. **Result Merger & Scoped Access Gate:** Cross-method deduplication, score normalization, and default-deny permission checks.
+4. **Direct Evidence Reading (`bash`):** The agent directly opens and inspects promising files with `cat`, `grep`, or `find` to verify facts against ground truth.
+5. **Curation & Active Feedback:** Structured findings are returned via `emit_autorag_results`. When callers provide feedback on which items were useful, AutoRAG records this to optimize future queries.
 
-A fresh AutoRAG tries everything. A seasoned one knows exactly where to look. This is not a static configuration — it's learned behavior from real usage.
+---
 
-### Multiple retrieval methods, one interface
+## 🤖 For AI Agents & LLMs: Setup & Skills
 
-Different documents need different search strategies:
+If you are an AI coding agent or LLM (Claude Code, Cursor, Windsurf, Codex, Senpi, Pi, etc.) tasked with installing, configuring, or querying AutoRAG Agent, use our built-in skills located in the [`skills/`](skills/) directory:
 
-| Your documents | Best method | Why |
+| Skill | Directory | When to Use |
 |---|---|---|
-| Plain text, config files | grep (pattern matching) | Fast, precise, literal |
-| Research papers, dense prose | Vector search (semantic) | Understands meaning, not just keywords |
-| Legal documents, specifications | BM25 (keyword ranking) | Handles domain terminology well |
-| Mixed collections | Hybrid (vector + BM25) | Combines precision and recall |
+| **`autorag`** | [`skills/autorag/`](skills/autorag/SKILL.md) | Querying, searching, comparing, and summarizing documents with an already configured AutoRAG librarian. |
+| **`autorag-setup`** | [`skills/autorag-setup/`](skills/autorag-setup/SKILL.md) | Installing AutoRAG, configuring models, adding document roots/datasources, running health checks, and repairing indexes. |
+| **`autorag-lite-setup`** | [`skills/autorag-lite-setup/`](skills/autorag-lite-setup/SKILL.md) | Initializing and maintaining the model-free AutoRAG Lite lifecycle without an LLM. |
+| **`autorag-lite-search`** | [`skills/autorag-lite-search/`](skills/autorag-lite-search/SKILL.md) | Performing model-free retrieval, reporting evidence, and recording feedback without an LLM. |
+
+### Quick Agent Workflow
+
+1. **Install CLI:**
+   ```bash
+   command -v autorag >/dev/null || bun install -g @autorag/librarian
+   ```
+2. **Inspect & Preflight:**
+   ```bash
+   autorag status --json        # check corpus freshness and index readiness
+   autorag duplicates --json    # scan for exact or near-duplicate document families
+   ```
+3. **Perform Curated Search:**
+   ```bash
+   autorag search "your question" --json
+   ```
+4. **Safety Guidelines for Agents:**
+   - **Never delete, move, or modify original user documents.**
+   - AutoRAG writes index artifacts strictly into `<workspace>/.autorag/` and `.jikji/` caches.
+   - Never print or leak API keys, tokens, or credential values into stdout or logs.
+
+---
+
+## ⚡ AutoRAG Lite: Model-Free Retrieval Engine
+
+Need blazing fast local search without configuring an LLM or paying for API tokens? Use **AutoRAG Lite**.
+
+AutoRAG Lite provides the exact same high-performance indexing, BM25 ranking, and local MinSync vector/hybrid retrieval engine as the full librarian, but **runs 100% model-free**:
+
+- **Zero LLM Token Usage:** Run purely local BM25 and vector search offline.
+- **Agent Integration Ready:** Use `autorag lite retrieve` inside your own agentic workflows to supply raw context chunks to an external model.
+- **Fast Local CLI:** Instant responses directly from your terminal.
 
-AutoRAG supports **pluggable retrieval methods**. Local lexical BM25, semantic vector, and hybrid retrieval all go through **MinSync** over one shared CDC chunk lifecycle, wired through the `RetrievalMethodRegistry`. The librarian invokes retrieval tools, reads the underlying documents directly through `bash`, and curates one unified result set after `ResultMerger` score normalization and deduplication. External datasources keep their own archive/index lifecycle.
-
-BM25, vector, and hybrid are **enabled by default** whenever MinSync is enabled. Disable local indexing with `"minSync": false`, or disable only lexical search with `"bm25": false`. MinSync auto-installs a verified release into `<workspace>/.autorag/bin` on first use (`autoInstall: true`); set `"autoInstall": false` only when managing the binary yourself. Configure `minSync.embedder` via `autorag init --embedder-*` flags for remote embedding endpoints, and set `minSync.maxChunkSize` (or `--minsync-max-chunk-size`) when a local embedder has a smaller context window. AutoRAG never forces TEI or any external embedding service.
-
-See [docs/minsync-setup.md](docs/minsync-setup.md) for automatic installation,
-managed binary paths, and the local EmbeddingGemma QA flow.
-
-### Real directory access
-
-AutoRAG reads configured source directories directly through its built-in `bash` tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured `SearchDocumentsResponse`; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. MinSync indexes parsed markdown mirrors under `.autorag` for BM25, vector, and hybrid retrieval.
-
-### Thin PDF extraction retry
+```bash
+# Initialize a model-free workspace
+autorag lite init --search-paths ./documents
 
-The default PDF parser performs a cheap quality check for multi-page PDFs.
-When local markdown is unusually sparse (fewer than 800 characters or fewer
-than 40 characters per detected page, for at least three pages), it retries
-through OpenDataLoader's `docling-fast` hybrid backend with `hybridMode:
-"auto"` and a 30-second timeout. Dense PDFs are not retried, and hybrid is
-never used for single-page PDFs or as the first path for images.
+# Index local files and datasources
+autorag lite refresh
 
-The gate is parser-owned and can be tuned through trusted programmatic
-`parserOptions`:
+# Retrieve ranked document chunks (returns JSON candidates with scores)
+autorag lite retrieve "compliance policy exception process" --top-k 5 --json
 
-```typescript
-new AutoRAGAgent({
-  searchPaths: ["/path/to/documents"],
-  parserOptions: {
-    thinExtract: {
-      minPages: 3,
-      minChars: 800,
-      minCharsPerPage: 40,
-      timeoutMs: 30_000,
-      hybrid: "docling-fast",
-      hybridMode: "auto",
-    },
-  },
-});
+# Check index status and freshness
+autorag lite status
 ```
 
-If the hybrid sidecar is missing, times out, or fails, AutoRAG keeps the local
-markdown and emits `pdf-extract-thin` plus `pdf-hybrid-unavailable`
-diagnostics; refresh remains successful.
+---
 
-### Default Jikji discovery and indexing
+## 🔌 Supported Datasources
 
-AutoRAG uses [Jikji](https://github.com/NomaDamas/jikji) by default as a local CLI-backed **find-first discovery and indexing** layer. Jikji is not registered as a retrieval backend: it supplies bounded discovery answer packs while the librarian still reads original files directly.
+AutoRAG Agent connects to external tools and communication platforms using dedicated datasource skills. Data remains in each tool's native store — AutoRAG does not copy or ingest foreign databases into a centralized store:
 
-The default agent calls `jikji find ROOT "query" --json` via the `jikji_find` tool. The tool parses and validates the upstream answer pack and exposes its `handoff_action`, `tool_call_policy`, and `agent_should_not_rerank` to the librarian. Direct file reading remains available for source verification. `prepare`/`refresh` remain for indexing only and do not answer queries directly. If the binary or Rust toolchain is unavailable, Jikji reports a diagnostic and the normal filesystem/retrieval paths continue.
+| Datasource | Skill Alias | Backend / Driver | Storage & Privacy Model | Search Capabilities |
+|---|---|---|---|---|
+| **Local Documents** | `local` | Native filesystem & MinSync | Workspace-local parsed mirrors (`.autorag/`) | BM25, Semantic, Hybrid |
+| **KakaoTalk** | `kakao` | [`lazykatok`](https://github.com/changeroa/lazykatok) CLI | Native KakaoTalk archive; zero direct DB access | BM25, Semantic, Hybrid |
+| **Discord** | `discord` | [`discrawl`](https://github.com/openclaw/discrawl) CLI | Native SQLite archive; token-free wiretap mode | BM25, Semantic, Hybrid |
+| **WhatsApp** | `whatsapp` | [`wacrawl`](https://github.com/openclaw/wacrawl) CLI | Local-first incremental archive + FTS5 | Lexical FTS5 |
+| **Telegram** | `telegram` | [`telecrawl`](https://github.com/openclaw/telecrawl) CLI | Local-first desktop archive + FTS5 | Lexical FTS5 |
+| **Slack** | `slack` | [`slacrawl`](https://github.com/openclaw/slacrawl) CLI | Local workspace/channel/thread archive + FTS5 | Lexical FTS5 |
+| **Notion** | `notion` | [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | Local page/database/block archive + FTS5 | Lexical FTS5 |
+| **Email Archives** | `mailcrawl` | [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | Local Gmail, IMAP, and Maildir storage | BM25, Semantic, Hybrid |
+| **Local Mail Export**| `mail-export` | Built-in `.mbox` / `.eml` parser | Local filesystem mailboxes | Lexical |
+| **Obsidian Vaults** | `obsidian` | [`qmd`](https://github.com/tobi/qmd) CLI | Direct markdown vault indexing | BM25, Semantic |
+| **GitHub** | `github` | GitHub REST API | In-memory fetched Issues and Pull Requests | Lexical, Scoped |
+| **Cloud Drives** | `cloud-drive` | [`rclone`](https://rclone.org) CLI | Google Drive (Tier-1), OneDrive, Dropbox, etc. | Incremental Mirror + BM25 |
+| **Photos & Shots** | `clawgallery` | `clawgallery` CLI | Local screenshot and photo store | Hybrid OCR/Visual Search |
+| **RSS / News** | `rss` | Native HTTP Poller | RSS 2.0 & Atom feeds (24h deduplication) | Lexical |
+| **macOS Spotlight** | `spotlight` | Native `mdfind` CLI | macOS system metadata and content index | System Native |
 
-New `autorag init` configs include `"jikji": {}`. To opt out, set `"jikji": false` in `config.json`; programmatic callers can pass `jikji: false`.
+For configuration syntax and connector details, see [docs/datasource-skills.md](docs/datasource-skills.md).
 
-Programmatic use:
-
-```typescript
-const agent = new AutoRAGAgent({
-  searchPaths: ["/path/to/documents"],
-});
-await agent.prepareJikji();
-```
+---
 
-### Duplicate document management with dupey
+## Installation & Setup
 
-AutoRAG can use the external [`dupey`](https://github.com/NomaDamas/dupey) CLI
-to detect exact, near, and containment document families.
+AutoRAG Agent is published as `@autorag/librarian` (requires Node.js ≥ 24 or Bun):
 
 ```bash
-autorag duplicates /path/to/documents
-autorag duplicates --json
-```
-
-The command is read-only: it reports exact duplicate groups and review
-guidance, but never moves or deletes source files. The `scan_duplicate_documents`
-Agent tool exposes the same read-only scan to the orchestrator.
-
-Exact duplicate exclusion is enabled by default during parsed-mirror refresh.
-For each exact canonical-text hash, the newest filesystem copy is indexed and
-older copies are omitted. Disable it in `config.json` when both copies must be
-searchable:
+# Install CLI globally
+bun install -g @autorag/librarian
+# or with npm:
+npm install -g @autorag/librarian
 
-```json
-{
-  "dupey": { "enabled": true },
-  "excludeExactDuplicates": false
-}
-```
-
-If `dupey` is not installed or fails, refresh continues without exclusion and
-reports no destructive action; install it with `cargo install dupey`.
-
-The same configuration shape customizes Jikji when needed:
-
-```json
-{
-  "enabled": true,
-  "binaryPath": "jikji",
-  "timeoutMs": 10000,
-  "maxBufferBytes": 1048576,
-  "includeHidden": false,
-  "includeSensitive": false,
-  "maxFiles": 0,
-  "writeAgentRules": false,
-  "enableMediaIndex": false,
-  "exclude": []
-}
-```
-
-Call `agent.prepareJikji()` (or `agent.refresh()`) to prepare configured source roots. Hidden files, sensitive files, and media indexing are disabled by default; AutoRAG does not pass `--include-hidden`, `--include-sensitive`, or `--enable-media-index` unless the corresponding option is `true`. AutoRAG-managed prepare runs with `--no-agent-rules` by default, so it never rewrites the consumer repo's `AGENTS.md`/`CLAUDE.md`/`.cursorrules`; an explicit `writeAgentRules: true` opt-in re-enables upstream routing-block injection. AutoRAG passes `--enable-media-index` only when `enableMediaIndex: true`.
-
-The upstream Rust `PrepareArgs` defines reference defaults that AutoRAG does not override unless explicitly configured: parse timeout `5.0`, max hash bytes `512 MiB`, doc text max chars `2,000,000`, doc text chunk chars `1,000,000`, and media index max MB `25.0`. AutoRAG emits `--parse-timeout`, `--max-hash-bytes`, `--doc-text-max-chars`, `--doc-text-chunk-chars`, and `--media-index-max-mb` only when the matching option is set, so the upstream defaults apply otherwise. AutoRAG answers queries through `jikji find` (find-first) plus the Pi agent loop and its registered retrieval methods; `prepare`/`refresh` are indexing-only.
-
-### Datasource skills
-
-Datasource skills let AutoRAG search external, server-configured data sources through the same retrieval pipeline as local documents. A skill describes what it indexes, how it should be refreshed, what source instances exist, and which permission tags/scopes bound access. Retrieval still flows through `RetrievalMethodRegistry` → `ParallelRetriever` → datasource result filtering → `ResultMerger`; datasource skills do not create a parallel search path.
-
-Every datasource can be registered multiple times through a connection alias:
-use the config key as the unique name and set `type` to the reusable backend
-(`mailcrawl`, `github`, `slack`, `discord`, `kakao`, `cloud-drive`, and so on).
-Each alias becomes an independently loadable agent skill with its own source
-scope and workspace namespace. Chat aliases search all channels by default;
-trusted `channels.ids` / `channels.names` allowlists can expose a particular
-channel or group chat as its own datasource.
-
-Security defaults are intentionally strict:
-
-- datasource access is default-deny unless trusted server/API configuration supplies `datasourceAccess.allowedTags`; `allowedScopes` applies only to datasource methods that advertise the `scoped` capability;
-- model/tool arguments never grant datasource tags or scopes;
-- `search_datasource_documents` accepts only `{ query, topK?, scope? }`, and `scope` can only narrow trusted access.
-
-#### Supported datasources
-
-| Datasource | Skill | Connects via | Notes |
-|---|---|---|---|
-| KakaoTalk | `katok` | external [`katok`](https://github.com/NomaDamas/katok) CLI | first datasource skill; AutoRAG never reads KakaoTalk databases directly |
-| WhatsApp | `whatsapp` | external [`wacrawl`](https://github.com/openclaw/wacrawl) CLI | local-first incremental archive + FTS5 search; live desktop ingestion is macOS-only |
-| Telegram | `telegram` | external [`telecrawl`](https://github.com/openclaw/telecrawl) CLI | local-first archive + FTS5 search; live desktop ingestion is macOS-only |
-| Slack | `slack` | external [`slacrawl`](https://github.com/openclaw/slacrawl) CLI | local-first workspace/channel/thread archive + FTS5 search |
-| Discord | `discord` | external [`discrawl`](https://github.com/openclaw/discrawl) CLI | guild/channel/thread/DM archive; FTS5 + semantic + hybrid retrieval, incremental sync |
-| Notion | `notion` | external [`notcrawl`](https://github.com/openclaw/notcrawl) CLI | local-first page/database/block archive + FTS5 search |
-| GitHub Issues/PRs | `github` | GitHub REST (token optional) | issues + PR bodies per `owner/repo`; public repos work unauthenticated |
-| Cloud drives | `cloud-drive` | **[`rclone`](https://rclone.org) CLI** | Incremental Google Drive Tier-1; OneDrive/network remotes; iCloud experimental |
-| Local mail exports | `mail-export` | filesystem (`.mbox` / `.eml`) | classic `From_` splitting, mailparser-based; count-only warnings |
-| Mail archives | `mailcrawl` | external [`mailcrawl`](https://github.com/NomaDamas/mailcrawl) CLI | local email sync plus BM25, semantic, and hybrid retrieval |
-| Obsidian vault | `obsidian` | external [`qmd`](https://github.com/tobi/qmd) CLI | incremental `qmd update`, BM25 `qmd search`, semantic `qmd vsearch`; vault path via `connector.vaultPath` |
-| RSS / news | `rss` | HTTP feed polling | RSS 2.0 + Atom, feed/category hierarchy, 24h dedupe window |
-| ClawGallery | `clawgallery` | external `clawgallery` CLI (`cargo install clawgallery`) | local screenshots and photos; incremental bootstrap + hybrid search |
-| macOS Spotlight | `spotlight` | built-in `mdfind` | macOS only; no extra install; Full Disk Access for protected files |
-
-Connector-backed skills fetch documents into AutoRAG's local chunk store. External-crawler skills such as KakaoTalk, WhatsApp, Telegram, Slack, and Notion leave incremental archive and FTS ownership with their CLI and map query results into the same retrieval pipeline. Obsidian uses the external `qmd` CLI (incremental update + BM25 + semantic). Tokens are referenced by environment variable name only, never stored in config. Process/API failures surface as path/PII-opaque diagnostics. See [docs/manual-qa-datasources.md](docs/manual-qa-datasources.md) for the QA harnesses.
-
-Configure them in `config.json` (CLI) or pass `datasourceSkills` programmatically:
-
-```jsonc
-{
-  "datasources": {
-    "whatsapp": { "instanceId": "personal", "connector": { "binaryPath": "wacrawl" } },
-    "telegram": { "instanceId": "personal", "connector": { "binaryPath": "telecrawl" } },
-    "slack":    { "connector": { "binaryPath": "slacrawl", "configPath": "/path/to/slacrawl.yaml", "syncSource": "primary" } },
-    "notion":   { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } },
-    "github":   { "connector": { "repos": ["owner/repo"] } },
-    "mailcrawl": { "instanceId": "personal", "connector": { "account": "personal", "mailbox": "INBOX", "binaryPath": "mailcrawl" } },
-    "personal-google-drive": { "type": "cloud-drive", "instanceId": "personal", "connector": { "provider": "google-drive", "remote": "personal-gdrive:", "include": ["**/*.md"] } },
-    "company-onedrive": { "type": "cloud-drive", "instanceId": "work", "connector": { "provider": "onedrive", "remote": "company-onedrive:Documents" } },
-    "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
-    "rss":      { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
-  },
-  "datasourceAccess": {
-    "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "mailcrawl", "cloud-drive", "obsidian", "rss"],
-    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/mailcrawl/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"]
-  }
-}
+# Add as a TypeScript/JavaScript library
+bun add @autorag/librarian
 ```
 
-Install wacrawl with `brew install openclaw/tap/wacrawl`. AutoRAG invokes `wacrawl sync` during datasource refresh and `wacrawl --json --sync never search` during retrieval. Optional trusted connector fields are `binaryPath`, `databasePath`, and `sourcePath`. The child process receives only a restricted environment; unrelated model/provider secrets are not forwarded. Live WhatsApp Desktop discovery requires macOS and the permissions documented by wacrawl, while an existing portable archive can be queried on other supported platforms.
+### System Prerequisites
 
-Install telecrawl with `brew install openclaw/tap/telecrawl`. AutoRAG invokes `telecrawl import` during datasource refresh and `telecrawl --json search` during retrieval. It uses the same optional trusted connector fields and restricted child environment as wacrawl. Live Telegram Desktop discovery requires macOS and the permissions documented by telecrawl, while an existing portable archive can be queried on other supported platforms.
+- **Java 11 or newer:** Required for PDF parsing via `@opendataloader/pdf`. Check with `java -version`.
+- **Rust Toolchain (Optional):** Automatically compiles Jikji (`jikji-cli`) if installed.
+- **MinSync:** Automatically downloaded and installed into `<workspace>/.autorag/bin` on first run.
 
-Install slacrawl with `brew install openclaw/tap/slacrawl`. AutoRAG invokes `slacrawl sync` during datasource refresh and `slacrawl --json search` during retrieval. Optional trusted connector fields are `binaryPath`, `configPath`, and `syncSource`. Slack credentials and source definitions remain in slacrawl's own configuration rather than AutoRAG.
+---
 
-Install notcrawl with `brew install openclaw/tap/notcrawl`. AutoRAG invokes `notcrawl sync` during datasource refresh and `notcrawl search` during retrieval (notcrawl has no `--json` flag through v0.6.2; AutoRAG parses its tab-separated hits). Optional trusted connector fields are `binaryPath` and `configPath`. Notion credentials and workspace definitions remain in notcrawl's own configuration rather than AutoRAG.
-
-Install and configure [`mailcrawl`](https://github.com/NomaDamas/mailcrawl)
-`@nomadamas/mailcrawl@0.1.6` or newer separately (0.1.4 is the minimum). AutoRAG invokes
-`mailcrawl sync` followed by `mailcrawl index` during datasource refresh, then
-calls `mailcrawl search` in BM25, semantic, or hybrid mode. The archive remains
-in mailcrawl's native store unless the operator explicitly sets
-`connector.dataDir`; Himalaya credentials and provider configuration remain
-owned by mailcrawl. 0.1.3 and earlier fail a repeated
-`index` after a no-op sync.
-Use `mailcrawl` for Gmail, IMAP, and Maildir retrieval. The former Gmail REST
-datasource and `gmail` + `backend: "himalaya"` connector are removed.
-
-Install and authenticate rclone separately (`brew install rclone && rclone
-config` on macOS), then configure the provider-neutral `cloud-drive` skill.
-`cloud-drive` is a reusable `type`: each datasource config key is a connection
-alias and becomes a separate agent skill and scope. Multiple Google accounts,
-or Google Drive plus OneDrive/iCloud, can therefore be loaded and searched
-independently.
-AutoRAG inventories with `rclone lsjson`, keeps a workspace-local manifest and
-managed mirror, and downloads only added or changed indexable files. Google
-Drive is Tier-1. OneDrive, Dropbox, SMB/SFTP/WebDAV, and mounted drives share
-the same manifest contract. iCloud Drive is experimental because rclone marks
-that backend Tier 4 and Apple ID/2FA sessions periodically require
-reauthentication. See [docs/datasource-skills.md](docs/datasource-skills.md)
-for filtering, size, concurrency, bandwidth, dry-run, and agent tool-calling
-details.
-
-#### KakaoTalk (katok)
-
-KakaoTalk was the first datasource skill. It uses the external [`katok`](https://github.com/NomaDamas/katok) CLI only — AutoRAG never reads KakaoTalk databases directly. `katok` failures return diagnostics instead of throwing, and remote embedding egress configuration is rejected before the CLI is spawned.
+## Quick Start
 
-```typescript
-import { AutoRAGAgent, KatokSkill } from "@autorag/librarian";
+### 1. Initialize and Index
 
-const kakao = new KatokSkill({
-  instanceId: "personal",
-  tags: ["kakaotalk", "personal", "pii"],
-  // Optional: client: new KatokClient({ binaryPath: "katok" })
-});
+```bash
+# Initialize configuration for your documents folder
+autorag init --search-paths ~/Documents/research
 
-const agent = new AutoRAGAgent({
-  searchPaths: ["/path/to/documents"],
-  datasourceSkills: [kakao],
-  datasourceAccess: {
-    allowedTags: ["kakaotalk"],
-  },
-});
+# Index documents (parses PDFs/Markdown, builds BM25 and MinSync vectors)
+autorag refresh
 
-await agent.refresh(); // refreshes parsed mirrors, BM25/MinSync, and datasource indexes
-const results = await agent.searchDatasourceDocuments("meeting with Mina", { topK: 5 });
+# Check indexing health
+autorag status
 ```
 
-#### Discord (discrawl)
-
-Discord uses the external [`discrawl`](https://github.com/openclaw/discrawl) CLI, which owns the SQLite archive, the FTS5 index, and the message vectors. AutoRAG never calls the Discord API itself.
+### 2. Search from CLI
 
 ```bash
-brew install openclaw/tap/discrawl
+# Perform a curated search (uses your configured reasoning model)
+autorag search "What are our primary Q3 deliverables?"
+
+# Launch the interactive Terminal UI (beta)
+autorag tui
+
+# Open the loopback web management dashboard (127.0.0.1)
+autorag ui
 ```
 
-AutoRAG reads discrawl's own local archive: `discrawl wiretap` imports the Discord Desktop cache and needs **no token at all**, and every search, embed, and sync step runs against that native store. AutoRAG carries no Discord credential — no bot token, no credential env var, no credential check in `autorag setup` — and it refuses to spawn the CLI when a Discord *user* token is present in the environment, because automating a user account violates Discord's Community Guidelines and can get the account terminated. A bot-backed archive is the CLI's own concern: configure and sync it with `discrawl` itself.
+### 3. Programmatic Usage (TypeScript API)
 
 ```typescript
-import { AutoRAGAgent, DiscrawlClient, DiscrawlSkill } from "@autorag/librarian";
-
-const discord = new DiscrawlSkill({
-  client: new DiscrawlClient({ root: process.cwd() }),
-  instanceId: "community",
-});
+import { AutoRAGAgent } from "@autorag/librarian";
 
+// Initialize librarian agent
 const agent = new AutoRAGAgent({
   searchPaths: ["/path/to/documents"],
-  datasourceSkills: [discord],
-  datasourceAccess: {
-    allowedTags: ["discord"],
-  },
 });
-```
-
-Or through the trusted config factory:
-
-```json
-{
-  "datasources": {
-    "discord": {
-      "instanceId": "community",
-      "connector": {
-        "embeddingProvider": "ollama",
-        "embeddingModel": "embeddinggemma",
-        "defaultMode": "hybrid"
-      }
-    }
-  }
-}
-```
-
-Two defaults are deliberate and worth keeping:
-
-- **`defaultMode: "hybrid"`** — discrawl's FTS index strips newlines without substituting a space, welding words across line breaks into a single unsearchable token (measured at ~47% of post-newline words on a real archive). Semantic recall covers that gap. See [#1413](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1413).
-- **`embeddingProvider: "ollama"` + `embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). Configure these values in discrawl's own config; AutoRAG uses discrawl's native store unless an explicit `connector.configPath` is supplied. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources can share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
-
-A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as `/kakao/personal/chunks/<chunk-id>`, and permission tags that match your server-side access policy.
 
-### Primary target: document collections
+// Run curated search loop
+const response = await agent.searchDocuments("Summarize recent compliance updates");
 
-AutoRAG is built for **non-code document retrieval**: manuals, legal docs, internal wikis, meeting notes, research literature, knowledge bases, PDFs.
-
-Code repositories work too (direct `grep` is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
-
-## Configuration and state
-
-The default home state is kept outside the workspace:
-
-```text
-~/.autorag/
-├── config.json
-├── memory.json
-└── logs/
-    └── runs.jsonl
-```
-
-`config.json` selects sources, the workspace, memory path, retrieval settings, and the agent model. Provider and model IDs must refer to a model available in the user's authenticated runtime:
-
-```json
-{
-  "searchPaths": ["/path/to/documents"],
-  "workspacePath": "/path/to/workspace",
-  "memoryPath": "/Users/you/.autorag/memory.json",
-  "model": { "provider": "provider-name", "id": "reasoning-model" }
+console.log("Answer:", response.answer);
+for (const result of response.results) {
+  console.log(`[${result.number}] ${result.title} (${result.source})`);
+  console.log(`    ${result.summary}`);
 }
-```
-
-`autorag init` leaves `model` unset when no model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure the model explicitly.
-
-For fast interactive search, prefer a model with reliable tool calling, high
-output TPS, and low first-token latency. A query can require several short
-model turns while AutoRAG alternates between retrieval tools and direct source
-reading, so model throughput has a visible effect on end-to-end response time.
-It does not accelerate BM25, MinSync, Jikji, filesystem access, or indexing
-itself. Larger reasoning models remain useful for difficult synthesis,
-conflicting evidence, and specialized domain judgment, but they are not a
-requirement for ordinary retrieval.
 
-Config path precedence is `--config` > `AUTORAG_CONFIG` > `~/.autorag/config.json`. When the home config is absent and `<cwd>/autorag.config.json` exists, AutoRAG copies the legacy file to `~/.autorag/config.json` without deleting or modifying the legacy file. The legacy cwd file is a migration source, not the default location.
+// Record feedback: Result [1] was useful, [2] was not
+agent.recordFeedbackByNumbers(response.sessionId, [1], [2]);
+```
 
-`memory.json` stores retrieval memory and `logs/runs.jsonl` records run events. Model authentication remains with the user's configured provider or authenticated local runtime. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under `<workspace>/.autorag`.
+---
 
-`autorag refresh` and `autorag index reset|rebuild` accept `--method <csv>` (e.g. `--method minsync,parsed` or `--method datasources,jikji`) to scope which indexing methods run or which index directories are removed. Valid values are `parsed`, `minsync`, `datasources`, `jikji`, and `all`. BM25 is a MinSync retrieval mode, not a `--method` name. When omitted, all methods run. `autorag init` accepts `--embedder-*` flags to configure the MinSync embedder endpoint in the config file.
+## CLI Command Reference
 
-`autorag refresh --json` reports the MinSync outcome in a `minsync` block (`ok`, `synced`, and a `reason`/`diagnostics` on failure), and the envelope's `ok` is `false` whenever MinSync, a datasource index, or any error-severity diagnostic failed — a green envelope never hides a semantic index that did not update.
+| Command | Description |
+|---|---|
+| `autorag init` | Initialize `~/.autorag/config.json` with search roots and model settings |
+| `autorag refresh` | Refresh parsed mirrors, MinSync CDC chunks, datasources, and Jikji |
+| `autorag search "<query>"` | Run the librarian agent to curate structured answers |
+| `autorag status` | Inspect corpus freshness, indexing status, and vector readiness |
+| `autorag health` | Check model provider authentication, token validity, and API reachability |
+| `autorag tui` | Open the interactive librarian terminal UI |
+| `autorag ui` | Open the local loopback web dashboard to configure datasources visually |
+| `autorag duplicates [DIR]` | Read-only scan for exact and near-duplicate document families with `dupey` |
+| `autorag lite ...` | Model-free indexing, retrieval, report generation, and status |
+| `autorag feedback <session>` | Record useful / not-useful feedback by item number |
+| `autorag evidence <session>` | Inspect exact underlying document chunks and sources for a past query |
+| `autorag serve` | Start the P2P query server over SimpleX (opt-in) |
+| `autorag p2p ...` | Manage peer trust, query approvals, and sharing policies |
 
-`autorag health` checks model/provider auth before a search — it resolves the model, verifies credential presence, and optionally probes one completion call. Use it to diagnose model, provider, auth, or timeout failures. `autorag status` remains the model-free index-health command (corpus freshness and BM25/MinSync readiness). When `autorag search` fails for a model/provider reason, the error output includes a hint pointing to `autorag health`.
+---
 
-`autorag ui` opens a loopback-only page (`127.0.0.1`) to connect local folders and datasource skills without editing JSON. It writes the same trusted `datasources` / `datasourceAccess` fields as a hand-edited config, stores env-var *names* rather than secrets, and refuses non-loopback binds. Use `--no-open` to print the URL without launching a browser.
+## Documentation Links
 
-For a deliberately deployed UI, opt in explicitly in `config.json`. Keep the
-session token in the environment, set the public URL used by the browser, and
-allow only the frontend origins that should make credentialed API requests:
+Deep dive into AutoRAG Agent's architecture, security, and integration guides:
 
-```json
-{
-  "ui": {
-    "host": "0.0.0.0",
-    "port": 8787,
-    "allowRemote": true,
-    "publicOrigin": "https://autorag.example.com",
-    "corsOrigins": ["https://autorag.example.com"],
-    "tokenEnv": "AUTORAG_UI_TOKEN"
-  }
-}
-```
+- **[MinSync Setup & Embedding QA](docs/minsync-setup.md):** Automatic binary installation, CDC chunking, and EmbeddingGemma verification.
+- **[Local Embedding Runtime & Gateway](docs/embedding-runtime.md):** AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search. Model cards: [`qwen3-embedding-0.6b`](docs/model-cards/qwen3-embedding-0.6b.md), [`embeddinggemma-300m`](docs/model-cards/embeddinggemma-300m.md).
+- **[Datasource Skills Reference](docs/datasource-skills.md):** Full configuration contracts, connection aliases, and connector options.
+- **[Manual QA & Datasource Test Harnesses](docs/manual-qa-datasources.md):** Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.
+- **[P2P SimpleX Sharing & Path Standard](docs/p2p-path-standard.md):** Decentralized peer query sharing with SimpleX, PII redaction, and approval queues.
+- **[Supply Chain Security & License Audits](docs/supply-chain.md):** Software bill of materials (SBOM) and dependency gate policies.
+- **[Competitive Landscape Study](docs/competitive-landscape-2026-09.md):** In-depth analysis of why in-place federation outperforms centralized RAG.
+- **[Legacy AutoRAG (AutoML)](legacy/README.md):** Documentation for the legacy Python AutoML pipeline optimizer (`pip install AutoRAG`).
 
-Start it with `AUTORAG_UI_TOKEN` set to a random value of at least 16
-characters. Local use remains the safe default: omit `ui` (or leave
-`allowRemote` false) and AutoRAG binds to loopback, including a working
-`localhost` URL on systems that resolve it to IPv6.
+---
 
-### CLI commands
+## Contributors & Community
 
-```bash
-autorag init                    # write ~/.autorag/config.json
-autorag refresh                 # parsed + MinSync + datasources + Jikji
-autorag watch [--once]          # filesystem watch, or one cron/poll tick
-autorag status                  # corpus freshness (model-free)
-autorag search "your question"  # curated answer (requires a configured model)
-autorag feedback <session>      # numbered useful / not-useful feedback
-autorag evidence <session>      # persisted source/chunk evidence
-autorag memory inspect          # retrieval memory snapshot
-autorag index reset|rebuild     # remove or rebuild workspace indexes
-autorag health                  # model/provider auth (no index check)
-autorag duplicates [DIR]        # read-only duplicate families; never deletes
-autorag tui                     # interactive librarian terminal UI (beta)
-autorag ui                      # loopback page to connect datasources
-autorag lite …                  # model-free setup, retrieve, report, status
-autorag setup                   # install and configure local runtime assets
-autorag models prefetch|import|verify
-                                # manage local embedding model profiles
-autorag gateway status|stop     # inspect or stop the local embedding gateway
-autorag serve                   # P2P peer query server over SimpleX (opt-in)
-autorag p2p                     # peer trust, approvals, and sharing policy
-```
+AutoRAG Agent is an open-source project built by the community. We welcome contributions, bug reports, datasource connectors, and ideas!
 
-`autorag --help` is the command list of record; `autorag <command> --help`
-prints that command's flags and subcommands, and `autorag --version` prints the
-installed version. See
-[docs/embedding-runtime.md](docs/embedding-runtime.md) for the local embedding
-runtime and gateway setup.
+- **Contributing:** Feel free to open an issue or pull request. Start with [CONTRIBUTING.md](CONTRIBUTING.md) — development setup, the `make ci` check we ask for, the `Signed-off-by` trailer every commit needs, and when to run the live end-to-end lanes. [MAINTAINERS](MAINTAINERS) lists who reviews which area. AI tools are allowed; unverified dumps are not — see [AI_POLICY.md](AI_POLICY.md).
+- **Security:** Private vulnerability reports and the response SLA live in [SECURITY.md](SECURITY.md).
+- **GitHub Contributors:** [View all contributors](https://github.com/Marker-Inc-Korea/AutoRAG/graphs/contributors) on GitHub.
 
-### P2P sharing (SimpleX)
+---
 
-Peer query sharing is **disabled by default**. Set `p2p.enabled` to `true` in
-`config.json`, or pass `--force` to start the server once. Peers reach this
-installation through SimpleX; the local control socket binds to loopback
-(default port 5225).
+## Acknowledgements
 
-```bash
-autorag serve                          # start the peer query server
-autorag p2p peers                      # list trusted peers
-autorag p2p peers --add alice --contact-id 1
-autorag p2p requests                   # pending responses held for approval
-autorag p2p requests approve <id>
-autorag p2p policy list                # merged sharing rules (virtual-path keys)
-autorag p2p policy set '/docs/**' peers
-autorag p2p policy unset '/docs/**'
-```
+AutoRAG Agent stands on the shoulders of fantastic open-source projects:
 
-Inbound queries are screened for injection. Outbound answers are filtered by
-sharing policy (`private` / `never` / `always` / `peers`) and PII redaction,
-and held until the operator approves them. The librarian can call
-`recommend_peer_targets` to suggest local peer personas without contacting
-them.
+- **[Pi Framework](https://github.com/earendil-works/pi-mono)** by [@earendil-works](https://github.com/earendil-works) — The foundational agent framework powering AutoRAG's reasoning loop.
+- **[MinSync](https://github.com/Marker-Inc-Korea/minsync)** — Ultra-fast incremental Change Data Capture (CDC) chunking and local BM25/vector indexing.
+- **[Jikji](https://github.com/NomaDamas/jikji)** by [NomaDamas](https://github.com/NomaDamas) — High-performance find-first local document discovery.
+- **[dupey](https://github.com/NomaDamas/dupey)** by [NomaDamas](https://github.com/NomaDamas) — Fast duplicate and near-duplicate document family detection.
+- **Federated CLI Authors:** External datasource tools [`lazykatok`](https://github.com/changeroa/lazykatok), [`discrawl`](https://github.com/openclaw/discrawl), [`mailcrawl`](https://github.com/NomaDamas/mailcrawl), [`wacrawl`](https://github.com/openclaw/wacrawl), [`telecrawl`](https://github.com/openclaw/telecrawl), [`slacrawl`](https://github.com/openclaw/slacrawl), [`notcrawl`](https://github.com/openclaw/notcrawl), [`qmd`](https://github.com/tobi/qmd), and [`rclone`](https://rclone.org).
+- **[OpenDataLoader](https://github.com/opendataloader/opendataloader-pdf)** / **Docling** — Robust PDF and document parsing runtimes.
 
-## Installation
+---
 
-Published as `@autorag/librarian` (dist bundled with Bun, runtime Node ≥ 24 or Bun):
+## Troubleshooting
 
-```bash
-bun add @autorag/librarian          # library
-bun install -g @autorag/librarian   # autorag CLI
-# or run directly from the repo:
-bun add github:Marker-Inc-Korea/AutoRAG
-```
+When search returns nothing, a datasource disappears from results, refresh hangs, or the gateway will not start, run the **`autorag-doctor`** agent skill (`skills/autorag-doctor/SKILL.md`). Point your coding agent at it and say *"check AutoRAG"* — it walks the full diagnose-and-repair procedure and ends with a per-datasource status table showing what is indexed and what actually returns hits.
 
-PDF parsing requires **Java 11 or newer** because the bundled
-`@opendataloader/pdf` runtime is compiled for Java 11. This applies to
-Windows, macOS, and Linux. Verify the runtime that will be selected from
-`PATH` before indexing PDFs:
+The first three commands cover most of it:
 
 ```bash
-java -version
+autorag status --json          # freshness, per-component state, diagnostics
+autorag health --json          # model resolution + one live completion probe
+autorag gateway status --format json   # on-demand embedding runtime
 ```
 
-On Windows, ensure a Java 11+ installation appears before older Java
-installations in `PATH` (or set `JAVA_HOME` to the Java 11+ installation and
-start a new shell). Java 8 is not supported by the PDF parser.
-
-Git-based installs build `dist/` via the `prepare` script and require Bun on the installing machine.
-External tool binaries auto-install on first use into `<workspace>/.autorag/bin`: **MinSync** installs from crates.io via `cargo install minsync` first (on by default; verified GitHub release assets are the fallback when cargo is unavailable; `minSync.autoInstall: false` to opt out), and **Jikji** compiles the [`jikji-cli`](https://crates.io/crates/jikji-cli) crate via cargo (requires the [Rust toolchain](https://rustup.rs); `jikji.autoInstall: false` to opt out). New `autorag init` configs enable Jikji find-first discovery by default. KakaoTalk (`katok`) and Discord (`discrawl`) stay manual, optional installs. All of them degrade gracefully when missing — core BM25 search works without any of them.
-
-### TUI (beta)
-
-`autorag tui` opens an interactive Pi-powered librarian terminal UI using the
-same config as `autorag search`:
+Indexing is not the same thing as searchability, so always confirm retrieval itself — this needs no model:
 
 ```bash
-autorag tui
+autorag lite retrieve 'a word that certainly appears' --top-k 3 --json
+autorag lite retrieve 'recent topic' --tags discord --top-k 3 --json
 ```
 
-Inside the TUI, use `/resume` to browse saved sessions. Selecting a session
-replaces the current view with that session instead of mixing the two
-conversation histories. The TUI is beta and may still change.
+Common failures and their fix:
 
-## Quick Start
-
-```typescript
-import { AutoRAGAgent } from "@autorag/librarian";
-
-const agent = new AutoRAGAgent({
-  searchPaths: ["/path/to/documents"],
-});
-
-const response = await agent.searchDocuments("summarize the compliance requirements");
-console.log(response.answer);
-for (const result of response.results) {
-  console.log(`[${result.number}] ${result.title} — ${result.summary}`);
-}
-
-// Mark which results were useful — AutoRAG remembers for next time
-agent.recordFeedbackByNumbers(response.sessionId, [1, 3], [2]);
-```
-
-`searchDocuments()` runs the Pi agent loop — it searches, reads, consults memory, curates, and finalizes through the `emit_autorag_results` structured tool — then returns a typed `SearchDocumentsResponse`. The caller consumes the structured payload directly; no assistant text parsing.
-
-## How It Works
+| Symptom | Diagnostic code | Fix |
+|---|---|---|
+| Results are missing recent files | `stale-index` | `autorag refresh --method parsed,minsync --json` |
+| Semantic search returns nothing after changing the embedder | `embedding-identity-mismatch` | `autorag index rebuild --method minsync` |
+| Gateway will not start, a previous run was killed | `lock-conflict` | `autorag gateway stop`, then retry |
+| A datasource is healthy in its own CLI but absent from results | — | add its tag/scope to `datasourceAccess` |
+| A datasource errors during refresh | `datasource-index-failed` | run that CLI's own `doctor` |
+| MinSync or Jikji missing | `minsync-unavailable`, `jikji-unavailable` | check the Rust toolchain, re-run refresh |
 
-```
-    You ask a question
-           │
-           ▼
-    ┌──────────────┐
-    │ Plan + search│ ← check_memory, Jikji, and retrieval tools
-    └──────┬───────┘
-           ▼
-    ┌──────────────┐
-    │ Direct read  │ ← bash find/grep/cat
-    └──────┬──────┘
-           ▼
-    ┌─────────────┐
-    │   Curate     │ ← Extract key insights, not raw lines
-    └──────┬──────┘
-           ▼
-    [1] Finding A — summary (location)
-    [2] Finding B — summary (location)
-
-           │
-    Caller says "1 was useful, 2 wasn't"
-           │
-           ▼
-    ┌─────────────┐
-    │   Memory     │ ← Remembers what worked, adapts next time
-    └─────────────┘
-```
+Native datasource stores stay owned by their CLIs — AutoRAG never rebuilds them. Fix a broken archive with `lazykatok doctor`, `discrawl --json metadata`, `slacrawl --json doctor`, `wacrawl --json doctor`, `telecrawl --json doctor`, `notcrawl doctor`, `qmd status`, or `mailcrawl doctor`, then re-run `autorag refresh --method datasources --json`.
 
 ## License
 
-MIT for AutoRAG 2.0 (`LICENSE`). The legacy Python package in `legacy/` is Apache-2.0 (`legacy/LICENSE`). Production third-party licenses are listed in `NOTICE`. Supply-chain gates are documented in `docs/supply-chain.md`.
+- **AutoRAG 2.0 (AutoRAG Agent):** Released under the [MIT License](LICENSE).
+- **Legacy Python AutoRAG (`legacy/`):** Released under the [Apache License 2.0](legacy/LICENSE).
+- Production third-party notices and licenses are documented in [`NOTICE`](NOTICE).