@autorag/librarian 2.10.3 → 2.10.5

package/NOTICE

@@ -18,6 +18,9 @@ License: MIT
 @earendil-works/pi-tui@1.2.0
 License: MIT
 
+@llblab/pi-telegram@0.57.2
+License: MIT
+
 @modelcontextprotocol/server@2.3.1
 License: Apache-2.0
 

package/README.md

@@ -140,8 +140,9 @@ does, and searches them right alongside your files:
   or connect your own provider key and pay the provider directly.
 
 > [!TIP]
-> **The app's source lives in [NomaDamas/AutoRAG-app](https://github.com/NomaDamas/AutoRAG-app)**
-> (Electron + React, AGPL-3.0). It depends on `@autorag/librarian` from this
+> **The Dazzi desktop app's source lives in [NomaDamas/AutoRAG-app](https://github.com/NomaDamas/AutoRAG-app)**
+> (Electron + React, [Fair Source](https://fair.io/) under `FSL-1.1-MIT`). It
+> depends on the open-source `@autorag/librarian` engine (Apache-2.0) from this
 > repository; build, run, and file app issues there.
 
 ### One engine, every surface
@@ -150,8 +151,9 @@ The app is one way in. The same librarian runs everywhere else:
 
 | Surface | Use it for | Start here |
 |---|---|---|
-| **Desktop app** | Browsing and asking in one window | [NomaDamas/AutoRAG-app](https://github.com/NomaDamas/AutoRAG-app) |
+| **Desktop app (Dazzi)** | Browsing and asking in one window | [NomaDamas/AutoRAG-app](https://github.com/NomaDamas/AutoRAG-app) (Fair Source, FSL-1.1-MIT) |
 | **Terminal UI** | Interactive, steerable search sessions | [`autorag tui`](#2-search-from-cli) |
+| **Messenger** | Asking the librarian from a chat app | [Telegram](#messenger-connections) |
 | **CLI** | Scripts, indexing, health checks | [CLI Command Reference](#cli-command-reference) |
 | **MCP server** | Feeding retrieval to Claude Code, Codex, and other agents | [AutoRAG Lite MCP Server](#autorag-lite-mcp-server) |
 | **TypeScript API** | Embedding the librarian in your own app | [Programmatic usage](#3-programmatic-usage-typescript-api) |
@@ -198,7 +200,7 @@ AutoRAG Agent orchestrates five integrated subsystems:
 
 1. **Retrieval Memory (`check_memory`):** After each answer, cited evidence Jev judges to genuinely support it is stored in `~/.autorag/memory.json`. Later searches receive this conversation's evidence, similar past questions, and long-term insights as **advisory reference** — it never reorders results or overrides current evidence. See [docs/retrieval-memory.md](docs/retrieval-memory.md).
 2. **Pluggable Multi-Method Retrieval:**
-   - **BM25 Lexical Search:** Fast keyword ranking via MinSync over parsed markdown mirrors.
+   - **BM25 Lexical Search:** Fast keyword ranking via MinSync over parsed markdown mirrors. AutoRAG always uses MinSync's `multilingual` tokenizer, which keeps each original word and adds Korean morphemes (via the [Kiwi](https://github.com/bab2min/Kiwi) library), so both `연구목표` and `인증서를` → `인증서` match; without Kiwi AutoRAG warns and tokenizes on whitespace. See [docs/minsync-setup.md](docs/minsync-setup.md#bm25-tokenizer).
    - **Semantic Vector Search:** Dense vector retrieval over CDC chunks via the built-in embedding gateway.
    - **Hybrid Search:** Combines BM25 and vector ranking via Reciprocal Rank Fusion (RRF).
    - **Jikji Find-First Discovery:** Local CLI-backed fast discovery answer packs.
@@ -221,6 +223,18 @@ Update notices are AutoRAG's, not Pi's: `autorag tui` suppresses Pi's "run `pi u
 
 ---
 
+## Messenger Connections
+
+| Messenger | Status | Start here |
+|---|---|---|
+| **Telegram** | Supported | [`autorag tui --telegram`](docs/telegram.md) |
+| **WhatsApp** | Supported (unofficial client) | [`autorag tui --whatsapp`](docs/whatsapp.md) |
+| Discord, Slack, others | Not supported yet | |
+
+Telegram runs through the [`@llblab/pi-telegram`](https://github.com/llblab/pi-telegram) bridge inside a running `autorag tui` session: questions you send the bot get the same routing and two-phase search as the terminal. Setup, commands, and keeping it running are in [docs/telegram.md](docs/telegram.md).
+
+WhatsApp runs through the pinned [`whatsapp-pi`](https://www.npmjs.com/package/whatsapp-pi) extension the same way; see [docs/whatsapp.md](docs/whatsapp.md).
+
 ## Installation & Setup
 
 AutoRAG Agent is published as `@autorag/librarian` (requires Node.js ≥ 24 or Bun):
@@ -263,6 +277,25 @@ autorag refresh
 autorag status
 ```
 
+### Excluding files and folders
+
+Leave generated workspaces or archives out when initializing:
+
+```bash
+autorag init --search-paths "$HOME/Documents" \
+  --exclude "$HOME/Documents/Build output" \
+  --exclude "$HOME/Documents/Archive, old"
+```
+
+Repeat `--exclude` for each file or folder. Each argument is one complete path,
+including spaces and commas; relative paths resolve from the current directory.
+Excluding a folder also excludes its descendants. Paths are literal, not globs.
+
+For an existing configuration, edit its `excludePaths` array and run
+`autorag refresh` to update the indexes. File-name search applies exclusions to
+native fsearch and Everything results as well as the filesystem walk. See the
+[exclusion configuration examples](docs/minsync-setup.md#excluding-files-and-folders).
+
 ### Document languages and parsers
 
 One global `languages` setting describes the corpus. It is resolved from
@@ -434,7 +467,7 @@ mirrored are kept and not re-parsed on the next run. `onProgress` reports `parse
 | `autorag health` | Check model provider authentication, token validity, and API reachability |
 | `autorag models list` | List chat models the pi runtime can resolve (built-ins, `models.json`, custom/extension providers) with provider auth status; never prints credential values |
 | `autorag update-check` | Compare the running `autorag` against the published npm version (also runs on `autorag tui` launch) |
-| `autorag tui` | Open Pi's interactive librarian TUI (`/login`, `/model`, `/resume`, …) |
+| `autorag tui` | Open Pi's interactive librarian TUI (`/login`, `/model`, `/resume`, …); `--telegram` / `--whatsapp` add the Telegram / WhatsApp remote-control bridges |
 | `autorag duplicates [DIR]` | Read-only scan for exact and near-duplicate document families with `dupey` |
 | `autorag lite ...` | CLI bootstrap, indexing repair, terminal maintenance, and fallback interface; agents use MCP for normal Lite operation |
 | `autorag memory inspect` | Render a read-only, path-opaque snapshot of retrieval memory (judged evidence, curated results, evidence chunks, insights); model-free |
@@ -655,6 +688,7 @@ Deep dive into AutoRAG Agent's architecture, security, and integration guides:
 - **[MinSync Setup & Embedding QA](docs/minsync-setup.md):** Automatic binary installation, CDC chunking, and local embedding verification.
 - **[Local Embedding Runtime & Gateway](docs/embedding-runtime.md):** AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search. Model card: [`qwen3-embedding-0.6b`](docs/model-cards/qwen3-embedding-0.6b.md).
 - **[Hosted Dazzi Models Provider](docs/cloud-provider.md):** The hosted `dazzi` model provider — PKCE `/login`, `AUTORAG_BASE_URL`, live model discovery, and offline catalog snapshots.
+- **[Telegram Remote Control](docs/telegram.md):** Ask the librarian from a Telegram chat through a running `autorag tui --telegram` session: setup, commands, and keeping it alive.
 - **[Datasource Skills Reference](docs/datasource-skills.md):** Full configuration contracts, connection aliases, and connector options.
 - **[Jev Decisions](docs/jev-decisions.md):** Jev query pipeline, **on by default** via Dazzi Models when signed in (OpenRouter otherwise): it routes each question to a direct answer (intrinsic knowledge) or local search (everything needing more information; web search is left to the agent after the fast answer), splits multi-part questions into up to five parallel searches (`dazzi/openai/gpt-6-luna` when signed in, else `openrouter/qwen/qwen3.7-flash`), picks which registered datasources to search before the fast answer, and ends the run after the fast answer when that answer is complete. Also covers the `jev` judgment tool, backends (OpenRouter, TypeSafe, Vercel AI Gateway), and how to opt out.
 - **[Manual QA & Datasource Test Harnesses](docs/manual-qa-datasources.md):** Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.

package/dist/agent/agent.d.ts

@@ -247,7 +247,7 @@ export interface AutoRAGAgentOptions {
      * `searchPaths`). Enabled by default on Windows; ignored elsewhere.
      * `false` disables it. The remaining fields are test seams.
      */
-    everything?: Omit<EverythingClientOptions, "root" | "folders"> | false;
+    everything?: Omit<EverythingClientOptions, "root" | "folders" | "excludeFolders"> | false;
     /**
      * macOS/Linux-only instant file/folder name search through the user's
      * fsearch-cli (FSearch, GPL — spawned as a separate process, never
@@ -263,9 +263,10 @@ export interface AutoRAGAgentOptions {
      * provider-chain web module. Default enabled and credential-free: the
      * chain leads with providers that need no user-issued key — model-native
      * search reusing the agent's own model credentials (Gemini grounding,
-     * Anthropic/OpenAI/xAI web_search), the anonymous Perplexity ask
-     * endpoint, and Parallel's keyless MCP — then scraped engines with
-     * headless-browser escalation for bot challenges. A self-hosted
+     * Anthropic/OpenAI/xAI web_search), Parallel's keyless MCP, and the
+     * DuckDuckGo HTML form — then the anonymous Perplexity ask endpoint and
+     * the other scraped engines with headless-browser escalation for bot
+     * challenges. A self-hosted
      * SEARXNG_ENDPOINT is the only env-gated option. `false` disables both
      * tools. The `fetch` sub-option tunes or disables `web_fetch` alone.
      * Web tools are always omitted for remote P2P sessions.
@@ -366,6 +367,17 @@ export interface AutoRAGAgentOptions {
      * a launch.
      */
     updateNotice?: () => Promise<string | undefined>;
+    /** Extension entry files (installed pi packages) loaded into the interactive session. */
+    piExtensionPaths?: readonly string[];
+    /** Flag values for the flags those extensions register. */
+    piExtensionFlags?: Readonly<Record<string, string | boolean>>;
+    /**
+     * Load the `@llblab/pi-telegram` bridge into the interactive session so a
+     * paired Telegram chat can start, steer, and read answers from it. The
+     * bridge adds `/telegram-setup` and `/telegram-connect`; its bot token and
+     * owner pairing live in pi's `telegram.json`, never in AutoRAG config.
+     */
+    telegram?: boolean;
 }
 /** Post-merge reranking options. Mirrors the CLI `RerankConfig` (secrets via env). */
 export interface RerankAgentOptions {
@@ -483,6 +495,9 @@ export declare class AutoRAGAgent {
     private readonly piSessionDir;
     private readonly persistPiSessions;
     private readonly updateNotice;
+    private readonly piExtensionPaths;
+    private readonly piExtensionFlags;
+    private readonly telegram;
     private boundPiRuntime;
     /** Embeds past questions for similar-question search; undefined keeps memory search keyword-only. */
     private readonly memoryEmbedder;

package/dist/agent/answer-guidelines.d.ts

@@ -17,3 +17,20 @@ export declare const ANSWER_IMAGE_DELTA_RULE = "An image already embedded in the
  * caller only ever sees numbered citations backed by recorded evidence.
  */
 export declare const ANSWER_CITATION_RULE = "**Citations**: cite supporting evidence inline with the evidence id shown next to each retrieved result, in square brackets (e.g. [e3], or [e3][e7] for two). Copy ids exactly; never invent one. For a local file you opened yourself with `bash`, cite `[file:<absolute path>]`. Cite the evidence that backs a sentence at the end of that sentence. Do not cite numbers, paths, or anything else.";
+/**
+ * Deliverable/artifact rule (issue #1832). The 5-bullet cap and direct-answer
+ * format remain the default unless the user explicitly asks otherwise; when the
+ * caller requests a deliverable (a reply email, letter, legal opinion, summary
+ * document, insertable case citation block, checklist, report, or workflow
+ * document), the agent produces the complete finished artifact as a file in a
+ * format aligned with the context and domain, and keeps the answer itself
+ * concise by stating the created file's location and briefly summarizing what
+ * was generated (citing only the underlying source evidence).
+ */
+export declare const ANSWER_DELIVERABLE_RULE = "**Deliverables and artifacts**: if the user asks you to draft, write, generate, or prepare a deliverable or artifact (a reply email, letter, legal opinion, summary document, standard-format case citation block, checklist, report, or workflow document \u2014 including requests phrased as \"how should I reply...\" to a specific message), produce the finished, ready-to-use artifact as a file (using `write`, `edit`, or `bash` in local sessions, without overwriting existing source files) in a file format, extension, and structure aligned with the context and domain (e.g. `.eml` or `.md`/`.txt` with headers for an email reply; `.md`, `.txt`, `.docx`, `.hwp`, or `.pdf` matching workspace conventions for a legal opinion, insertable case-citation block, or summary document; `.csv`, `.xlsx`, or `.json` for structured tabular data), containing the complete unabridged artifact (including required headers, precedent passages, pinpoint paragraphs, and standard-format citations) inside the file instead of merely describing how to produce it. Do NOT dump the full artifact text into your reply: unless the user explicitly requests otherwise, keep your reply concise within the at-most-5-bullet direct-answer format, clearly stating the created file's exact path/location and briefly summarizing what was generated with inline citations to the underlying source evidence (`[eN]` or `[file:<source path>]`); never cite the newly created deliverable file itself as `[file:...]`.";
+/**
+ * Verification-phase deliverable rule: create the context-aligned deliverable
+ * file (in local sessions) and report its location and concise summary if the
+ * fast-phase draft could not write the file yet or only gave advice about it.
+ */
+export declare const ANSWER_DELIVERABLE_DELTA_RULE = "If a deliverable or artifact was requested and the first answer could not write the file yet (the fast phase runs without tool calls) or only gave advice about it, create the complete context-aligned deliverable file now during verification and report the created file's location and a brief summary concisely.";

package/dist/agent/pi-session.d.ts

@@ -25,6 +25,10 @@ export interface AutoRAGPiSessionOptions {
      * extension. Each factory registers its tools, commands, and events with pi.
      */
     readonly extensionFactories?: readonly ExtensionFactory[];
+    /** Extension entry files (e.g. an installed pi package's extension) loaded into the interactive session. */
+    readonly extensionPaths?: readonly string[];
+    /** CLI-style flag values for the flags those extensions register (`--name` / `--name value`). */
+    readonly extensionFlags?: Readonly<Record<string, string | boolean>>;
     /**
      * Tool names registered by {@link extensionFactories}. pi's `tools`
      * allow-list keeps only named tools active, so every extension tool that
@@ -86,6 +90,23 @@ export interface AutoRAGPiSession {
 }
 /** Adapts a pi-agent-core {@link AgentTool} to pi's extension {@link ToolDefinition}. */
 export declare function toToolDefinition(tool: AgentTool): ToolDefinition;
+/**
+ * Prefix `@llblab/pi-telegram` puts on every prompt it injects from a Telegram
+ * chat. The bridge hands them to pi through `pi.sendUserMessage`, so pi tags
+ * them `source: "extension"`, the same source AutoRAG's own phase prompts use.
+ * The prefix is what tells a person typing on a phone from AutoRAG talking to
+ * itself.
+ */
+export declare const TELEGRAM_INPUT_PREFIX = "[telegram]";
+/** `whatsapp-pi` frames every inbound chat message as `Message from <name> (<number>)`. */
+export declare const WHATSAPP_INPUT_PREFIX = "Message from ";
+/**
+ * The user's own words from a bridged Telegram prompt: the bridge's `[telegram]`
+ * prefix and its `[time]` line are envelope, not the question, so a Telegram
+ * question reaches the librarian exactly as the same text typed in the TUI.
+ * Attachment blocks stay: they are part of what the user sent.
+ */
+export declare function stripTelegramEnvelope(text: string): string;
 /** Creates a pi session with AutoRAG's tools and prompt. */
 export declare function createAutoRAGPiSession(options: AutoRAGPiSessionOptions): Promise<AutoRAGPiSession>;
 export interface AutoRAGPiInteractiveRuntime {

package/dist/agent/search-documents.d.ts

@@ -9,7 +9,7 @@ export type SearchDocumentDiagnosticSeverity = "info" | "warning" | "error";
  * (#21/#22) does not require another source-breaking type change; codes not yet
  * emitted are still valid members.
  */
-export type SearchDocumentDiagnosticCode = "empty-query" | "no-verified-results" | "unknown-warning" | "caller-tool-dropped" | "minsync-staging-excluded" | "minsync-file-skipped" | "minsync-sync-failed" | "minsync-lock-held" | "embedder-unavailable" | "embedding-identity-mismatch" | "parser-skipped" | "parser-failed" | "parser-crashed" | "source-vanished" | "source-unreadable" | "source-read-failed" | "mirror-write-failed" | "parser-warning" | "duplicate-excluded" | "unsupported-file" | "stale-index" | "deleted-mirror" | "search-root-unavailable" | "mass-delete-suppressed" | "walk-truncated" | "directory-unreadable" | "pdf-extract-thin" | "retrieval-method-failed" | "rerank-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "everything-index-failed" | "fsearch-index-failed" | "fsearch-binary-missing" | "refresh-failed" | "refresh-interrupted" | "watch-failed" | "watch-limited" | "unknown-datasource-skill" | "no-final-answer" | "model-request-failed" | "tool-call-burst" | "query-routed" | "query-route-fallback" | "query-decomposition-failed" | "follow-up-skipped" | "follow-up-check-fallback" | "datasources-selected" | "datasource-selection-fallback" | "citation-without-result" | "evidence-judged" | "evidence-judgment-fallback" | "memory-search-fallback" | "self-config-unavailable" | "self-config-rolled-back";
+export type SearchDocumentDiagnosticCode = "empty-query" | "no-verified-results" | "unknown-warning" | "caller-tool-dropped" | "minsync-staging-excluded" | "minsync-file-skipped" | "minsync-sync-failed" | "minsync-lock-held" | "embedder-unavailable" | "embedding-identity-mismatch" | "lexical-tokenizer-unavailable" | "parser-skipped" | "parser-failed" | "parser-crashed" | "source-vanished" | "source-unreadable" | "source-read-failed" | "mirror-write-failed" | "parser-warning" | "duplicate-excluded" | "unsupported-file" | "stale-index" | "deleted-mirror" | "search-root-unavailable" | "mass-delete-suppressed" | "walk-truncated" | "directory-unreadable" | "pdf-extract-thin" | "retrieval-method-failed" | "rerank-failed" | "jikji-unavailable" | "jikji-prepare-failed" | "jikji-find-failed" | "everything-index-failed" | "fsearch-index-failed" | "fsearch-binary-missing" | "refresh-failed" | "refresh-interrupted" | "watch-failed" | "watch-limited" | "unknown-datasource-skill" | "no-final-answer" | "model-request-failed" | "tool-call-burst" | "query-routed" | "query-route-fallback" | "query-decomposition-failed" | "follow-up-skipped" | "follow-up-check-fallback" | "datasources-selected" | "datasource-selection-fallback" | "citation-without-result" | "evidence-judged" | "evidence-judgment-fallback" | "memory-search-fallback" | "self-config-unavailable" | "self-config-rolled-back";
 export interface SearchDocumentDiagnostic {
     readonly code: SearchDocumentDiagnosticCode;
     readonly severity: SearchDocumentDiagnosticSeverity;

package/dist/agent/web-search-tool.d.ts

@@ -3,8 +3,8 @@
  *
  * LLM-facing wrapper around the oh-my-pi-style provider chain in
  * `src/web/search/`: credential-free end to end (model-native search on the
- * agent's own model credential, the anonymous Perplexity ask endpoint,
- * Parallel's keyless MCP, then the scraped engines), where quota/auth/bot
+ * agent's own model credential, Parallel's keyless MCP, DuckDuckGo, then
+ * the anonymous Perplexity ask endpoint and the other scraped engines), where quota/auth/bot
  * failures automatically fall back down the chain. The model only supplies
  * `query` plus optional hints; provider selection and credentials are never
  * exposed through tool arguments.

package/dist/cli/commands/types.d.ts

@@ -1,6 +1,8 @@
 export interface CommandContext {
     positionals: string[];
     flags: Record<string, string | boolean | undefined>;
+    /** All values of repeatable flags, in command-line order. */
+    repeatedFlags?: Readonly<Record<string, readonly string[]>>;
     json: boolean;
     debug: boolean;
     cwd: string;

package/dist/cli/index.d.ts

@@ -5,6 +5,7 @@ export type { CommandName };
 interface ParsedArgs {
     positionals: string[];
     flags: Record<string, string | boolean>;
+    repeatedFlags?: Readonly<Record<string, readonly string[]>>;
 }
 export declare function parseArgs(argv: readonly string[]): ParsedArgs | {
     error: string;