@autorag/librarian 2.5.2 → 2.5.3

package/README.md

@@ -49,7 +49,7 @@ AutoRAG Agent:
 
 Three principles drive every design decision in AutoRAG Agent:
 
-1. **Never migrate your data to search it.** Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data **in place**, querying CLI-native stores (`katok`, `discrawl`, `slacrawl`, `mailcrawl`, `rclone`, `qmd`) where your data already lives. Results retain opaque, source-native identities (`/kakao/...`, `/slack/...`) that preserve local access control and privacy. *(See our [Competitive Landscape Study](docs/competitive-landscape-2026-09.md) on why in-place federation is the durable differentiator).*
+1. **Never migrate your data to search it.** Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data **in place**, querying CLI-native stores (`lazykatok`, `discrawl`, `slacrawl`, `mailcrawl`, `rclone`, `qmd`) where your data already lives. Results retain opaque, source-native identities (`/kakao/...`, `/slack/...`) that preserve local access control and privacy. *(See our [Competitive Landscape Study](docs/competitive-landscape-2026-09.md) on why in-place federation is the durable differentiator).*
 
 2. **Just works — no RAG degree required.** No pipeline tuning, no vector-DB maintenance, and no remote embedding API keys. AutoRAG Agent automatically manages [MinSync](docs/minsync-setup.md) for incremental Change Data Capture (CDC) chunking and provides a local [embedding gateway](docs/embedding-runtime.md) out of the box with zero external telemetry.
 
@@ -140,7 +140,7 @@ AutoRAG Agent connects to external tools and communication platforms using dedic
 | Datasource | Skill Alias | Backend / Driver | Storage & Privacy Model | Search Capabilities |
 |---|---|---|---|---|
 | **Local Documents** | `local` | Native filesystem & MinSync | Workspace-local parsed mirrors (`.autorag/`) | BM25, Semantic, Hybrid |
-| **KakaoTalk** | `katok` | [`katok`](https://github.com/NomaDamas/katok) CLI | Native KakaoTalk archive; zero direct DB access | BM25, Semantic, Hybrid |
+| **KakaoTalk** | `kakao` | [`lazykatok`](https://github.com/changeroa/lazykatok) CLI | Native KakaoTalk archive; zero direct DB access | BM25, Semantic, Hybrid |
 | **Discord** | `discord` | [`discrawl`](https://github.com/openclaw/discrawl) CLI | Native SQLite archive; token-free wiretap mode | BM25, Semantic, Hybrid |
 | **WhatsApp** | `whatsapp` | [`wacrawl`](https://github.com/openclaw/wacrawl) CLI | Local-first incremental archive + FTS5 | Lexical FTS5 |
 | **Telegram** | `telegram` | [`telecrawl`](https://github.com/openclaw/telecrawl) CLI | Local-first desktop archive + FTS5 | Lexical FTS5 |
@@ -259,7 +259,7 @@ agent.recordFeedbackByNumbers(response.sessionId, [1], [2]);
 Deep dive into AutoRAG Agent's architecture, security, and integration guides:
 
 - **[MinSync Setup & Embedding QA](docs/minsync-setup.md):** Automatic binary installation, CDC chunking, and EmbeddingGemma verification.
-- **[Local Embedding Runtime & Gateway](docs/embedding-runtime.md):** AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search.
+- **[Local Embedding Runtime & Gateway](docs/embedding-runtime.md):** AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search. Model cards: [`qwen3-embedding-0.6b`](docs/model-cards/qwen3-embedding-0.6b.md), [`embeddinggemma-300m`](docs/model-cards/embeddinggemma-300m.md).
 - **[Datasource Skills Reference](docs/datasource-skills.md):** Full configuration contracts, connection aliases, and connector options.
 - **[Manual QA & Datasource Test Harnesses](docs/manual-qa-datasources.md):** Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.
 - **[P2P SimpleX Sharing & Path Standard](docs/p2p-path-standard.md):** Decentralized peer query sharing with SimpleX, PII redaction, and approval queues.
@@ -273,7 +273,8 @@ Deep dive into AutoRAG Agent's architecture, security, and integration guides:
 
 AutoRAG Agent is an open-source project built by the community. We welcome contributions, bug reports, datasource connectors, and ideas!
 
-- **Contributing:** Feel free to open an issue or pull request. Please check our [Git Workflow in AGENTS.md](AGENTS.md) and run `bun test` before submitting.
+- **Contributing:** Feel free to open an issue or pull request. Start with [CONTRIBUTING.md](CONTRIBUTING.md) — development setup, the `make ci` check we ask for, the `Signed-off-by` trailer every commit needs, and when to run the live end-to-end lanes. [MAINTAINERS](MAINTAINERS) lists who reviews which area. AI tools are allowed; unverified dumps are not — see [AI_POLICY.md](AI_POLICY.md).
+- **Security:** Private vulnerability reports and the response SLA live in [SECURITY.md](SECURITY.md).
 - **GitHub Contributors:** [View all contributors](https://github.com/Marker-Inc-Korea/AutoRAG/graphs/contributors) on GitHub.
 
 ---
@@ -286,7 +287,7 @@ AutoRAG Agent stands on the shoulders of fantastic open-source projects:
 - **[MinSync](https://github.com/Marker-Inc-Korea/minsync)** — Ultra-fast incremental Change Data Capture (CDC) chunking and local BM25/vector indexing.
 - **[Jikji](https://github.com/NomaDamas/jikji)** by [NomaDamas](https://github.com/NomaDamas) — High-performance find-first local document discovery.
 - **[dupey](https://github.com/NomaDamas/dupey)** by [NomaDamas](https://github.com/NomaDamas) — Fast duplicate and near-duplicate document family detection.
-- **Federated CLI Authors:** External datasource tools [`katok`](https://github.com/NomaDamas/katok), [`discrawl`](https://github.com/openclaw/discrawl), [`mailcrawl`](https://github.com/NomaDamas/mailcrawl), [`wacrawl`](https://github.com/openclaw/wacrawl), [`telecrawl`](https://github.com/openclaw/telecrawl), [`slacrawl`](https://github.com/openclaw/slacrawl), [`notcrawl`](https://github.com/openclaw/notcrawl), [`qmd`](https://github.com/tobi/qmd), and [`rclone`](https://rclone.org).
+- **Federated CLI Authors:** External datasource tools [`lazykatok`](https://github.com/changeroa/lazykatok), [`discrawl`](https://github.com/openclaw/discrawl), [`mailcrawl`](https://github.com/NomaDamas/mailcrawl), [`wacrawl`](https://github.com/openclaw/wacrawl), [`telecrawl`](https://github.com/openclaw/telecrawl), [`slacrawl`](https://github.com/openclaw/slacrawl), [`notcrawl`](https://github.com/openclaw/notcrawl), [`qmd`](https://github.com/tobi/qmd), and [`rclone`](https://rclone.org).
 - **[OpenDataLoader](https://github.com/opendataloader/opendataloader-pdf)** / **Docling** — Robust PDF and document parsing runtimes.
 
 ---
@@ -321,7 +322,7 @@ Common failures and their fix:
 | A datasource errors during refresh | `datasource-index-failed` | run that CLI's own `doctor` |
 | MinSync or Jikji missing | `minsync-unavailable`, `jikji-unavailable` | check the Rust toolchain, re-run refresh |
 
-Native datasource stores stay owned by their CLIs — AutoRAG never rebuilds them. Fix a broken archive with `katok doctor`, `discrawl --json metadata`, `slacrawl --json doctor`, `wacrawl --json doctor`, `telecrawl --json doctor`, `notcrawl doctor`, `qmd status`, or `mailcrawl doctor`, then re-run `autorag refresh --method datasources --json`.
+Native datasource stores stay owned by their CLIs — AutoRAG never rebuilds them. Fix a broken archive with `lazykatok doctor`, `discrawl --json metadata`, `slacrawl --json doctor`, `wacrawl --json doctor`, `telecrawl --json doctor`, `notcrawl doctor`, `qmd status`, or `mailcrawl doctor`, then re-run `autorag refresh --method datasources --json`.
 
 ## License
 

package/dist/agent/agent.d.ts

@@ -394,21 +394,16 @@ export declare class AutoRAGAgent {
         results: RetrievalResult[];
         diagnostics: RetrievalDiagnostic[];
     }>;
-    searchDatasourceDocuments(query: string, options?: {
-        readonly topK?: number;
-        readonly scope?: string;
-    }): Promise<{
-        results: RetrievalResult[];
-        diagnostics: RetrievalDiagnostic[];
-    }>;
     /**
-     * Search one datasource connection only. Unlike
-     * {@link searchDatasourceDocuments}, which fans out to every authorized
-     * datasource and filters afterwards, this registers only the target
-     * connection's retrieval methods with the retriever, so no other
-     * datasource CLI is spawned at all. Access is still gated by the trusted
-     * datasource context: an unknown or unauthorized `datasourceId` yields an
-     * empty result set.
+     * Search one datasource connection only. Only the target connection's
+     * retrieval methods are registered with the retriever, so no other
+     * datasource CLI is spawned at all and only that connection's hits are
+     * returned. Access is still gated by the trusted datasource context: an
+     * unknown or unauthorized `datasourceId` yields an empty result set.
+     *
+     * This backs the generated `search_datasource_<id>` tools; every authorized
+     * connection has one. Cross-datasource fan-out is
+     * {@link searchAllDocuments}, which spans every retrieval method.
      */
     searchSingleDatasourceDocuments(datasourceId: string, query: string, options?: {
         readonly topK?: number;

package/dist/agent/index.d.ts

@@ -6,8 +6,7 @@ export type { AutoRAGEmittedResult, AutoRAGMappingEntry, AutoRAGResultsDetails,
 export { createEmitResultsTool, EMIT_AUTORAG_RESULTS_TOOL_NAME, emitResultsSchema, } from "./emit-results-tool.js";
 export { createJikjiFindTool, JIKJI_FIND_TOOL_NAME, type JikjiFindDetails, type JikjiFindPerRootPolicy, type JikjiFindProvider, type JikjiFindProviderResult, type MergedJikjiPolicy, } from "./jikji-find-tool.js";
 export { createSearchAllDocumentsTool, SEARCH_ALL_DOCUMENTS_TOOL_NAME, type SearchAllDocumentsDetails, type SearchAllDocumentsProvider, type SearchAllDocumentsResult, } from "./search-all-tool.js";
-export { createSearchDatasourceDocumentsTool, type DatasourceSearchProvider, SEARCH_DATASOURCE_DOCUMENTS_TOOL_NAME, type SearchDatasourceDocumentsDetails, } from "./search-datasource-tool.js";
 export type { SearchDocumentDiagnostic, SearchDocumentDiagnosticCode, SearchDocumentDiagnosticSeverity, SearchDocumentEvidence, SearchDocumentResult, SearchDocumentsResponse, SearchDocumentsStreamEvent, SearchDocumentWarning, } from "./search-documents.js";
 export { createSearchMinSyncDocumentsTool, SEARCH_MINSYNC_DOCUMENTS_TOOL_NAME, type SearchMinSyncDocumentsDetails, } from "./search-minsync-tool.js";
-export { createSingleDatasourceSearchTools, SEARCH_SINGLE_DATASOURCE_TOOL_PREFIX, type SearchSingleDatasourceDetails, type SingleDatasourceSearchProvider, type SingleDatasourceToolSpec, singleDatasourceToolName, } from "./search-single-datasource-tool.js";
+export { createSingleDatasourceSearchTools, formatDatasourceResults, SEARCH_SINGLE_DATASOURCE_TOOL_PREFIX, type SearchSingleDatasourceDetails, type SingleDatasourceSearchProvider, type SingleDatasourceToolSpec, singleDatasourceToolName, } from "./search-single-datasource-tool.js";
 export { buildSystemPrompt, type SystemPromptConfig } from "./system-prompt.js";

package/dist/agent/search-single-datasource-tool.d.ts

@@ -5,14 +5,18 @@ import { type SearchDocumentRetrievalTraceResult } from "./search-documents.js";
 /**
  * Per-datasource search tools.
  *
- * `search_datasource_documents` always fans out to every authorized datasource:
- * the retriever spawns every datasource CLI and only then filters results, so
- * an agent that already knows which connection holds the answer (from memory or
- * from the question itself) still pays for — and receives — every other
- * datasource's hits. These generated tools fix that: one tool per authorized
- * datasource connection (e.g. `search_datasource_discord`,
- * `search_datasource_kakao_work` for a `kakao-work` account alias) whose
- * execution spawns only that connection's retrieval methods.
+ * One tool is generated per authorized datasource connection (e.g.
+ * `search_datasource_discord`, or `search_datasource_kakao_work` for a
+ * `kakao-work` account alias). Its execution registers only that connection's
+ * retrieval methods with the retriever, so no other datasource CLI is spawned
+ * and the result set carries only that connection's hits.
+ *
+ * These generated tools are the only model-facing datasource retrieval
+ * surface. Cross-datasource fan-out belongs to `search_all_documents`, which
+ * already spans every configured retrieval method — including datasources — so
+ * a datasource-only fan-out tool would be redundant. Every authorized
+ * connection must therefore appear here; a datasource with no generated tool
+ * would become unreachable except through the full fan-out.
  *
  * Tools are generated from the configured, access-authorized datasource skills
  * at agent construction, so a datasource that is disabled in config or denied
@@ -54,4 +58,6 @@ declare const searchSingleDatasourceSchema: Type.TObject<{
     topK: Type.TOptional<Type.TInteger>;
     scope: Type.TOptional<Type.TString>;
 }>;
+/** Shared rendering for datasource search tool results (fan-out removed). */
+export declare function formatDatasourceResults(results: readonly RetrievalResult[], diagnostics: readonly RetrievalDiagnostic[]): string;
 export {};