@autorag/librarian 2.8.0 → 2.10.0

package/NOTICE

@@ -6,16 +6,16 @@ legacy/ package license: Apache-2.0
 
 Third-party production dependencies:
 
-@earendil-works/pi-agent-core@0.86.1
+@earendil-works/pi-agent-core@1.1.0
 License: MIT
 
-@earendil-works/pi-ai@0.86.1
+@earendil-works/pi-ai@1.1.0
 License: MIT
 
-@earendil-works/pi-coding-agent@0.86.1
+@earendil-works/pi-coding-agent@1.1.0
 License: MIT
 
-@earendil-works/pi-tui@0.86.1
+@earendil-works/pi-tui@1.1.0
 License: MIT
 
 @modelcontextprotocol/server@2.3.0

package/README.md

@@ -61,7 +61,7 @@ Three principles drive every design decision in AutoRAG Agent:
 
 AutoRAG Agent orchestrates five integrated subsystems:
 
-1. **Self-Evolving Memory System (`check_memory`):** Before querying, the agent consults historical outcomes stored in `~/.autorag/memory.json` to prioritize retrieval methods that have proven successful for similar queries.
+1. **Retrieval Memory (`check_memory`):** After each answer, cited evidence Jev judges to genuinely support it is stored in `~/.autorag/memory.json`. Later searches receive this conversation's evidence, similar past questions, and long-term insights as **advisory reference** — it never reorders results or overrides current evidence. See [docs/retrieval-memory.md](docs/retrieval-memory.md).
 2. **Pluggable Multi-Method Retrieval:**
    - **BM25 Lexical Search:** Fast keyword ranking via MinSync over parsed markdown mirrors.
    - **Semantic Vector Search:** Dense vector retrieval over CDC chunks via the built-in embedding gateway.
@@ -69,16 +69,16 @@ AutoRAG Agent orchestrates five integrated subsystems:
    - **Jikji Find-First Discovery:** Local CLI-backed fast discovery answer packs.
    - **Everything File-Name Search (Windows):** The bundled [voidtools Everything](https://www.voidtools.com/) indexes file and folder names under your search roots so the agent finds files by name, extension, path, size, or date instantly (`everything_search`).
    - **FSearch File-Name Search (macOS/Linux):** [fsearch-cli](https://github.com/NomaDamas/fsearch-mac) (FSearch) keeps a live, per-workspace name index of your search roots so the agent finds files by name, extension, path, size, or date instantly (`fsearch_search`); without fsearch-cli installed it degrades to a slow filesystem walk.
-   - **Datasource Skills:** Server-authorized federated retrieval across external applications.
-3. **Result Merger & Scoped Access Gate:** Cross-method deduplication, score normalization, and default-deny permission checks.
+   - **Datasource Skills:** Federated retrieval across every configured external datasource.
+3. **Result Merger & Scope Narrowing:** Cross-method deduplication, score normalization, and ordinary query-scope narrowing.
 4. **Direct Evidence Reading (`bash`):** The agent directly opens and inspects promising files with `cat`, `grep`, or `find` to verify facts against ground truth.
-5. **Curation & Active Feedback:** Structured findings are returned via `emit_autorag_results`. When callers provide feedback on which items were useful, AutoRAG records this to optimize future queries.
+5. **Curation:** The final answer is the model's plain reply; the harness resolves its inline `[eN]` citations into numbered, source-backed results; cited evidence that Jev judges to genuinely support the answer is remembered as reference context for later searches ([docs/retrieval-memory.md](docs/retrieval-memory.md)).
 
 ### Pi host boundary
 
-AutoRAG uses `@earendil-works/pi-coding-agent` as the runtime host for model sessions. Pi owns provider credentials and OAuth storage, model-runtime dispatch, session JSONL persistence/resume, extension loading, lifecycle events, and the built-in `read`/`bash`/`edit`/`write`/`grep`/`find`/`ls` tools. AutoRAG registers only its domain tools and keeps orchestration outside the host: datasource authorization, MinSync/Jikji preparation, memory hints, fast-to-verification two-phase search, structured result emission, and remote-session filtering.
+AutoRAG uses `@earendil-works/pi-coding-agent` as the runtime host for model sessions. Pi owns provider credentials and OAuth storage, model-runtime dispatch, session JSONL persistence/resume, extension loading, lifecycle events, and the built-in `read`/`bash`/`edit`/`write`/`grep`/`find`/`ls` tools. AutoRAG registers only its domain tools and keeps orchestration outside the host: datasource orchestration, MinSync/Jikji preparation, memory hints, fast-to-verification two-phase search, structured result emission, and remote-session filtering.
 
-The CLI TUI (`autorag tui`) is Pi's interactive mode hosted on the AutoRAG librarian: there is no separate AutoRAG renderer. Pi owns the terminal UI and its native in-session commands — `/login`/`/logout`, `/model`, `/resume`, `/new`, `/tree`, `/compact`, and `/settings` — so provider sign-in and model selection work on first launch even when no AutoRAG model is configured. AutoRAG registers its retrieval tools and streams progress, preliminary answers, and final results into the same Pi session, so resume and two-phase search keep working together.
+The CLI TUI (`autorag tui`) is Pi's interactive mode hosted on the AutoRAG librarian: there is no separate AutoRAG renderer. Pi owns the terminal UI and its native in-session commands — `/login`/`/logout`, `/model`, `/resume`, `/new`, `/tree`, `/compact`, and `/settings` — so provider sign-in and model selection work on first launch even when no AutoRAG model is configured. AutoRAG registers its retrieval tools and runs the two-phase search in the same Pi session, where the model's progress notes, first answer, and final answer are ordinary assistant messages, so resume and two-phase search keep working together.
 
 Update notices are AutoRAG's, not Pi's: `autorag tui` suppresses Pi's "run `pi update`" banner (Pi is a bundled host, not a separate install) and instead checks the published `@autorag/librarian` release, showing a one-line notice in the session when a newer version exists. Set `AUTORAG_NO_UPDATE_CHECK=1` to skip the lookup; `AUTORAG_UPDATE_CHECK_URL` overrides the registry endpoint.
 
@@ -118,8 +118,8 @@ cp -R "$AUTORAG_SKILLS/autorag-lite-setup" .claude/skills/
 Run the setup skill once to register the stdio server with the host. Reload the
 agent, then discover the actual tool names and schemas with MCP `tools/list`.
 The normal Lite path is `autorag.status` → `autorag.refresh` when needed →
-`autorag.search`; use `autorag.report`, `autorag.evidence`, and
-`autorag.feedback` for the optional curation lifecycle. Copy `skills/autorag`
+`autorag.search`; use `autorag.report` and `autorag.evidence` for the optional
+curation lifecycle. Copy `skills/autorag`
 and `skills/autorag-setup` only when the agent should also drive the
 model-backed librarian, and `skills/autorag-doctor` for diagnostics. Other
 agents read their own skill directories; copy the same setup folder there and
@@ -166,10 +166,9 @@ hard-code a tool count.
 
 Core Lite tools include `autorag.status`, `autorag.search`,
 `autorag.search.files`, `autorag.datasources.list`, `autorag.datasources.get`,
-`autorag.refresh`, `autorag.report`, `autorag.evidence`, and
-`autorag.feedback`. Configured integrated datasources expose additional scoped
-search tools. Read-only MCP mode omits mutating tools such as refresh, report,
-and feedback.
+`autorag.refresh`, `autorag.report`, and `autorag.evidence`. Configured
+integrated datasources expose additional scoped search tools. Read-only MCP
+mode omits mutating tools such as refresh and report.
 
 ## ⚡ AutoRAG Lite: Model-Free Retrieval Engine
 
@@ -277,15 +276,59 @@ falling back to `["ko", "en"]`. Supported tags: `ko`, `en`, `ja`, `zh-hans`,
 | `.pptx` | built-in PPTX reader |
 | `.eml` | built-in mail reader |
 | `.txt` `.text` `.md` `.markdown` | plain text (CP949/EUC-KR aware) |
-| `.png` `.jpg` `.jpeg` `.bmp` `.tiff` `.webp` | image OCR (opt-in) |
+| `.png` `.jpg` `.jpeg` `.webp` | `kordoc` image OCR (only when `parserOptions.ocr.enabled` is `true`) |
 
-OCR is opt-in and never runs unless enabled, so indexing downloads no model by
-default. When enabled, `languages` selects the recognition languages
-(`ja` → `jpn`, `zh-hans` → `chi_sim`, …) for both standalone images and scanned
-PDF pages.
+`.bmp` and `.tiff` are not parsed: kordoc rejects both as unsupported.
+
+### Making photos and scans searchable
+
+Image files and scanned PDF pages become searchable through one switch,
+`parserOptions.ocr.enabled` (default `false`). It is off by default so indexing
+downloads no recognition model and never runs OCR over a photo library. With it
+off, image files are not parsed at all and scanned pages stay unrecognized.
+
+```json
+{ "languages": ["ko", "en"], "parserOptions": { "ocr": { "enabled": true } } }
+```
+
+`kordoc` owns both inputs. The configured `languages` pick the engine:
+
+| `languages` | Engine | First-use download |
+|---|---|---|
+| only `ko` and/or `en` | kordoc built-in PP-OCRv5 (local CPU) | ~18 MB from Hugging Face, cached in `~/.cache/kordoc/models` |
+| any other tag (`ja`, `zh-hans`, `fr`, `de`, …) | Tesseract, all configured languages at once (`kor+jpn`) | traineddata per language from the jsDelivr CDN, cached in `<workspace>/.autorag/models/tessdata` |
+
+The built-in model has no kana, han, Cyrillic, Thai, Arabic or Devanagari
+characters and drops diacritics in accented Latin text, so those languages need
+Tesseract. Set `parserOptions.ocr.cachePath` to store the Tesseract models
+elsewhere, or `KORDOC_OFFLINE=1` to block the built-in model download. An
+explicit `parserOptions.kordoc.ocr` overrides the language-based choice.
+
+Photos and videos are also reachable through two other paths that are not part
+of this switch: the `clawgallery` datasource (captions and visual search over its
+own folders) and Jikji's media index (`jikji.enableMediaIndex`, file facts such as
+pixel size and format only, no content). Video content is not searchable yet.
 
 ### 2. Search from CLI
 
+`autorag search` needs one chat model. `autorag init` does not pick one, so on
+first run either sign in and choose a model inside the TUI:
+
+```bash
+autorag tui      # then /login (OAuth or API key) and /model
+```
+
+or add the `model` entry to your existing AutoRAG config, preserving its other
+settings. Find provider/model ids with `autorag models list --available`:
+
+```json
+"model": { "provider": "<provider>", "id": "<id>" }
+```
+
+Then run `autorag health`. Do not re-run `autorag init --force` just to select
+a model: it replaces the existing config, including datasource and parser settings.
+Configure the provider's credentials through `/login` or its supported environment variables.
+
 ```bash
 # Perform a curated search (uses your configured reasoning model)
 autorag search "What are our primary Q3 deliverables?"
@@ -312,11 +355,62 @@ for (const result of response.results) {
   console.log(`[${result.number}] ${result.title} (${result.source})`);
   console.log(`    ${result.summary}`);
 }
+```
 
-// Record feedback: Result [1] was useful, [2] was not
-agent.recordFeedbackByNumbers(response.sessionId, [1], [2]);
+---
+
+## Hosted AutoRAG Plan (Dazzi)
+
+AutoRAG can run on the hosted **AutoRAG** plan as a first-class model provider named
+`autorag`. The provider speaks the server's OpenAI Responses API surface (`<root>/v1`),
+so streaming, tool calls, and usage are handled by Pi's built-in `openai-responses`
+implementation, and the model catalog is discovered live from the server — no model id
+is baked into AutoRAG.
+
+Sign in from the interactive TUI:
+
+```text
+autorag tui
+/login          # choose "AutoRAG"; the browser completes a PKCE login
+```
+
+Headless (no browser) with a long-lived API key:
+
+```bash
+export AUTORAG_API_KEY=dz_...
+```
+
+On a fresh agent home, pin the model with an explicit endpoint so it resolves
+offline without a catalog snapshot — this is the headless configuration:
+
+```json
+{
+  "model": {
+    "provider": "autorag",
+    "id": "anthropic/claude-haiku-5.5",
+    "baseUrl": "https://api.dazziapp.com/v1",
+    "api": "openai-responses",
+    "apiKeyEnv": "AUTORAG_API_KEY"
+  }
+}
 ```
 
+The `provider`/`id`-only shorthand (`{ "provider": "autorag", "id": "anthropic/claude-haiku-5.5" }`)
+only resolves once a TUI `/login` or a model refresh has persisted the catalog
+snapshot; with no snapshot there is nothing to restore offline, and resolution
+fails with `Unknown configured model: autorag/<id>`.
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `AUTORAG_BASE_URL` | `https://api.dazziapp.com` | Server **root** (not the `/v1` path); the API base is `<root>/v1`. |
+| `AUTORAG_API_KEY` | – | Headless API key. A stored `/login` credential takes precedence. |
+
+The catalog is persisted after a `/login` (or any model refresh), so the
+`provider`/`id` shorthand `model: { provider: "autorag", id }`
+keeps resolving offline. When you are not signed in the provider still registers — it appears under
+`/login`, contributes no models, and makes no network call. See
+[Hosted AutoRAG provider](docs/cloud-provider.md) for the full flow.
+
 ---
 
 ## CLI Command Reference
@@ -333,7 +427,7 @@ agent.recordFeedbackByNumbers(response.sessionId, [1], [2]);
 | `autorag tui` | Open Pi's interactive librarian TUI (`/login`, `/model`, `/resume`, …) |
 | `autorag duplicates [DIR]` | Read-only scan for exact and near-duplicate document families with `dupey` |
 | `autorag lite ...` | CLI bootstrap, indexing repair, terminal maintenance, and fallback interface; agents use MCP for normal Lite operation |
-| `autorag feedback <session>` | Record numbered feedback; MCP clients normally use `autorag.feedback`, and the CLI stays available for terminal maintenance |
+| `autorag memory inspect` | Render a read-only, path-opaque snapshot of retrieval memory (judged evidence, curated results, evidence chunks, insights); model-free |
 | `autorag evidence <session>` | Inspect persisted evidence behind numbered results; MCP clients normally use `autorag.evidence`, and the CLI stays available for terminal maintenance |
 | `autorag serve` | Start the P2P query server over SimpleX (opt-in) |
 | `autorag p2p ...` | Manage peer trust, query approvals, and sharing policies |
@@ -346,8 +440,9 @@ Deep dive into AutoRAG Agent's architecture, security, and integration guides:
 
 - **[MinSync Setup & Embedding QA](docs/minsync-setup.md):** Automatic binary installation, CDC chunking, and EmbeddingGemma verification.
 - **[Local Embedding Runtime & Gateway](docs/embedding-runtime.md):** AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search. Model cards: [`qwen3-embedding-0.6b`](docs/model-cards/qwen3-embedding-0.6b.md), [`embeddinggemma-300m`](docs/model-cards/embeddinggemma-300m.md).
+- **[Hosted AutoRAG Provider](docs/cloud-provider.md):** The `autorag` (Dazzi) model provider — PKCE `/login`, `AUTORAG_API_KEY`, `AUTORAG_BASE_URL`, live model discovery, and offline catalog snapshots.
 - **[Datasource Skills Reference](docs/datasource-skills.md):** Full configuration contracts, connection aliases, and connector options.
-- **[Jev Decisions](docs/jev-decisions.md):** Jev query pipeline, **on by default** via OpenRouter: it routes each question to local search, web search, or a direct answer, splits multi-part questions into up to five parallel searches (`openrouter/qwen/qwen3.7-flash`), and ends the run after the fast answer when that answer is complete. Also covers the `jev` judgment tool, backends (OpenRouter, TypeSafe, Vercel AI Gateway), and how to opt out.
+- **[Jev Decisions](docs/jev-decisions.md):** Jev query pipeline, **on by default** via OpenRouter: it routes each question to a direct answer (intrinsic knowledge) or local search (everything needing more information; web search is left to the agent after the fast answer), splits multi-part questions into up to five parallel searches (`openrouter/qwen/qwen3.7-flash`), picks which registered datasources to search before the fast answer, and ends the run after the fast answer when that answer is complete. Also covers the `jev` judgment tool, backends (OpenRouter, TypeSafe, Vercel AI Gateway), and how to opt out.
 - **[Manual QA & Datasource Test Harnesses](docs/manual-qa-datasources.md):** Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.
 - **[P2P SimpleX Sharing & Path Standard](docs/p2p-path-standard.md):** Decentralized peer query sharing with SimpleX, PII redaction, and approval queues.
 - **[Supply Chain Security & License Audits](docs/supply-chain.md):** Software bill of materials (SBOM) and dependency gate policies.
@@ -400,7 +495,7 @@ itself through the connected Lite MCP server:
 autorag.status {}
 autorag.refresh {}
 autorag.search {"query":"a word that certainly appears","topK":3}
-autorag.search {"query":"recent topic","tags":["discord"],"topK":3}
+autorag.search {"query":"recent topic","scope":"/discord/local/**","topK":3}
 ```
 
 The CLI equivalents remain available for terminal repair, but do not use them
@@ -413,9 +508,10 @@ Common failures and their fix:
 | Results are missing recent files | `stale-index` | `autorag refresh --method parsed,minsync --json` |
 | Semantic search returns nothing after changing the embedder | `embedding-identity-mismatch` | `autorag index rebuild --method minsync` |
 | Gateway will not start, a previous run was killed | `lock-conflict` | `autorag gateway stop`, then retry |
-| A datasource is healthy in its own CLI but absent from results | — | add its tag/scope to `datasourceAccess` |
+| A datasource is healthy in its own CLI but absent from results | — | confirm it is configured under `datasources` and connected through its native CLI |
 | A datasource errors during refresh | `datasource-index-failed` | run that CLI's own `doctor` |
-| MinSync or Jikji missing | `minsync-unavailable`, `jikji-unavailable` | check the Rust toolchain, re-run refresh |
+| MinSync binary missing (search and refresh fail with `MinSync is required ...`) | — | `cargo install minsync`, or leave `minSync.autoInstall` on and re-run refresh |
+| Jikji missing | `jikji-unavailable` | check the Rust toolchain, re-run refresh |
 | Windows file-name search fails during refresh | `everything-index-failed` | read the ES exit code and stderr in the message, then `autorag refresh --method everything --json` |
 | Every search does a full local search and verification, even for small talk | `query-route-fallback` | set `OPENROUTER_API_KEY` (Jev routing and decomposition are on by default) |