@autorag/librarian 2.3.0 → 2.4.0

package/skills/autorag/SKILL.md

@@ -1,13 +1,14 @@
 ---
 name: autorag
-description: Use an already configured AutoRAG librarian agent to search, summarize, compare, and answer questions from local document collections. Use autorag-setup for configuration or indexing changes.
+description: Search, summarize, compare, and answer questions from an already configured AutoRAG librarian over local documents and authorized datasources. Use when the user asks AutoRAG to search PDFs, wikis, notes, or a knowledge base. Use autorag-setup for install, model, roots, indexing, or datasource changes.
+license: MIT
 ---
 
 # AutoRAG Librarian Skill
 
 Use this skill when AutoRAG is already configured and the user asks to search,
 summarize, compare, or answer questions from local PDFs, wikis, notes, research
-papers, or knowledge bases.
+papers, knowledge bases, or authorized datasources.
 
 AutoRAG is the specialized librarian agent. One configured model plans the
 search, calls BM25, MinSync, Jikji, datasource, and filesystem tools, reads
@@ -20,14 +21,14 @@ move, rename, edit, or delete source files.
 
 ## Preflight
 
-Confirm `~/.autorag/config.json`, an explicit `--config`, or `AUTORAG_CONFIG`
-exists. Inspect only non-secret `searchPaths` and `model` metadata, then run:
+Confirm a config exists at `--config`, `AUTORAG_CONFIG`,
+`$AUTORAG_HOME/config.json`, or `~/.autorag/config.json`. Inspect only
+non-secret `searchPaths` and `model` metadata, then run:
 
 ```bash
-autorag duplicates              # read-only duplicate-family review
-autorag duplicates --json       # machine-readable cleanup planning input
-autorag status
-autorag health
+autorag duplicates --json
+autorag status --json
+autorag health --json
 ```
 
 ### Duplicate-file review
@@ -45,59 +46,65 @@ checks credential presence, and normally probes one live completion. If the
 model, authentication, configuration, or indexes are unhealthy, use
 `autorag-setup` rather than guessing private provider details.
 
-BM25, MinSync, and Jikji should normally be healthy. If they are missing or
-stale, run a full `autorag refresh` or return to setup rather than silently
-degrading to lexical-only search.
+BM25, MinSync, and Jikji should normally be healthy. MinSync and Jikji
+auto-install on first use by default. If they are missing or stale, run a full
+`autorag refresh` or return to setup rather than silently degrading to
+lexical-only search.
 
 ## Search
 
+Prefer `--json --debug` when another agent will consume the result or record
+feedback. `--json` alone omits `sessionId`. `--debug` adds session/diagnostics
+fields and does not print filesystem paths.
+
 ```bash
-autorag search "what were the key findings in the Q3 report" --top-k 5 --json
+autorag search "what were the key findings in the Q3 report" --top-k 5 --json --debug
 ```
 
-The response contains a `sessionId`, numbered curated results, source mapping,
-and an answer grounded in those results.
+`--json --debug` includes `answer`, numbered `results` (`number`, `title`,
+`summary`, optional `source`), and `sessionId`. Use that `sessionId` for
+feedback. `--json` without `--debug` is only `answer` plus `results`.
 
-- `--scope` narrows to a configured virtual sub-path.
-- `--tags` narrows already-authorized datasource results and never grants new
-  access.
-- `--json` is preferred for programmatic consumption.
-- `--debug` is for diagnostics only.
+- `--scope` narrows datasource retrieval to a requested sub-path; it cannot
+  grant access.
+- `--tags` further narrows already-authorized datasource results and never
+  grants new access.
+- `--json` is required for programmatic consumption.
+- `--debug` is required for `sessionId`, evidence, and diagnostics.
 
 Do not bypass the librarian with ad hoc raw search when the user requested
 AutoRAG. The search loop can use Jikji, BM25, MinSync, datasource retrieval, and
 direct source reading as appropriate. If search fails because of model,
-provider, auth, or timeout problems, diagnose with `autorag health`.
+provider, auth, or timeout problems, diagnose with `autorag health --json`.
 
-Record feedback so retrieval memory can learn:
+Record feedback so retrieval memory can learn. Numbers refer to the returned
+knowledge units. Supply at least one feedback list:
 
 ```bash
-autorag feedback <sessionId> --useful 1,3 --not-useful 2
+autorag feedback <sessionId> --useful 1,3 --not-useful 2 --json
 ```
 
-Supply at least one feedback list. Numbers refer to the returned knowledge
-units.
-
 ## Maintenance
 
 ```bash
-autorag status
-autorag health
-autorag refresh
-autorag refresh --method bm25,minsync,jikji
-autorag watch --once
+autorag status --json
+autorag health --json
+autorag refresh --json
+autorag refresh --method bm25,minsync,jikji --json
+autorag watch --once --json
 autorag watch
-autorag refresh --force
-autorag index rebuild --yes
-autorag index reset --method bm25 --yes
-autorag memory inspect
+autorag refresh --force --json
+autorag index rebuild --yes --json
+autorag index reset --method bm25 --yes --json
+autorag memory inspect --json
 ```
 
 Prefer a full refresh so parsed mirrors, BM25, MinSync, Jikji, and configured
-datasources stay aligned. Use `--method` only for deliberate narrowing.
-Scheduled maintenance should use non-daemon `autorag watch --once`, typically
-every 15–30 minutes, with the same config used by search and no overlapping
-runs.
+datasources stay aligned. `--method` accepts
+`parsed,bm25,minsync,datasources,jikji,all`. Use it only for deliberate
+narrowing. Scheduled maintenance should use non-daemon `autorag watch --once`,
+typically every 15–30 minutes, with the same config used by search and no
+overlapping runs.
 
 Reset and rebuild commands remove only selected workspace `.autorag` indexes.
 Never target source documents. `memory inspect` is read-only and path-opaque.
@@ -106,7 +113,8 @@ Never target source documents. `memory inspect` is read-only and path-opaque.
 
 - Use only configured and approved search paths.
 - Never expose provider credentials or authentication payloads.
-- Never invent or reveal private provider names or model ids.
+- Never invent provider identities or model ids.
 - A Pi-usable subscription is valid; a subscription Pi cannot invoke is not.
 - Preserve real source mapping and numbered feedback identifiers.
-- Prefer `--json` when another agent consumes the response.
+- Prefer `--json --debug` when another agent consumes search output or will
+  call `autorag feedback`.

package/skills/autorag-setup/SKILL.md

@@ -1,12 +1,14 @@
 ---
 name: autorag-setup
-description: Configure AutoRAG for first use or repair its single-agent model, approved document roots, retrieval indexes, datasource skills, and health checks without exposing credentials.
+description: Install and configure AutoRAG, or repair its single search model, approved document roots, retrieval indexes, datasource skills, and health checks without exposing credentials. Use when autorag is missing, init/refresh/health fails, indexes are stale, or the user wants to add folders or datasources.
+license: MIT
 ---
 
 # AutoRAG setup
 
-Use this skill when AutoRAG is unconfigured, model resolution fails, indexes are
-missing or stale, or the user wants to change the document collection.
+Use this skill when AutoRAG is unconfigured, the `autorag` CLI is missing, model
+resolution fails, indexes are missing or stale, or the user wants to change the
+document collection or datasources.
 
 ## Safety
 
@@ -19,15 +21,26 @@ missing or stale, or the user wants to change the document collection.
   `node_modules`, `.git`, `dist`, `build`, `target`, `.cache`, `.autorag`, or
   `.jikji`.
 
+## Install the CLI if needed
+
+The CLI is `@autorag/librarian` (`autorag`). Runtime is Node.js ≥ 24 or Bun.
+
+```bash
+command -v autorag >/dev/null || bun install -g @autorag/librarian
+autorag --help
+```
+
+If Bun is unavailable, `npm install -g @autorag/librarian` is acceptable.
+
 ## Inspect existing configuration
 
-Check `~/.autorag/config.json`, an explicit `--config` path, or
-`AUTORAG_CONFIG`. Relevant fields are:
+Check `--config`, `AUTORAG_CONFIG`, `$AUTORAG_HOME/config.json`, or
+`~/.autorag/config.json`. Relevant fields are:
 
 - `searchPaths`, `workspacePath`, and `memoryPath`
 - `model.provider`, `model.id`, `model.api`, `model.baseUrl`, `model.apiKeyEnv`
 - `bm25`, `minSync`, and `jikji`
-- `datasources` and `datasourceAccess`
+- `datasources`, `datasourceAccess`, and `ui`
 
 Preserve explicit user choices and a working config unless the user asks to
 replace them or health checks fail.
@@ -73,9 +86,10 @@ approval before indexing:
 | Linux | XDG Documents/Downloads/Desktop or their `~/` defaults | `~/Notes`, `~/Sync`, Nextcloud/Syncthing roots |
 | Windows | Documents, Downloads, Desktop shell folders | OneDrive document roots, user-named project docs |
 
-Supported index formats include `md`, `markdown`, `txt`, `text`, `pdf`, `docx`,
-`pptx`, `xlsx`, `hwpx`, and `eml`; OCR image formats are optional. Do not
-present legacy `doc`, `xls`, or `hwp` as fully supported parsed formats.
+Supported parsed formats are `md`, `markdown`, `txt`, `text`, `pdf`, `docx`,
+`pptx`, `xlsx`, `xls`, `hwp`, `hwpx`, and `eml`. OCR for `jpg`, `jpeg`, `png`, `bmp`,
+and `tiff` is optional (`parserOptions.ocr.enabled`). Do not present legacy
+`.doc` as a supported parsed format.
 
 Keep the first-run set small, usually one to three roots. Present a concrete
 proposal and require `yes`, a narrowed keep-list, a custom list, or `skip`
@@ -114,9 +128,11 @@ Use `--force` only when intentionally replacing an existing config. Legacy cwd
 ### Retrieval defaults
 
 BM25, MinSync, and Jikji are enabled by default. Leave them enabled unless the
-user explicitly asks otherwise. MinSync auto-install is off by default and its
-binary must already be installed or on `PATH`; Jikji can install `jikji-cli`
-through cargo when enabled and allowed.
+user explicitly asks otherwise. MinSync auto-installs a verified GitHub release
+into `<workspace>/.autorag/bin` on first use (`minSync.autoInstall` defaults to
+true). Set `"autoInstall": false` only when managing the binary yourself. Jikji
+auto-installs `jikji-cli` through cargo when enabled (`jikji.autoInstall`
+defaults to true; requires the Rust toolchain).
 
 Exact duplicate exclusion is enabled by default. AutoRAG invokes the external
 `dupey` CLI before parsed-mirror indexing, keeps the newest filesystem copy for
@@ -140,42 +156,55 @@ size must be positive integers.
 
 ## Configure datasource skills when requested
 
-Datasource skills belong in trusted config and remain default-deny. Common
-entries include WhatsApp, Telegram, Slack, Discord, Notion, GitHub, Google
-Drive, a generic cloud-drive/rclone source, Gmail, local mail exports,
-Obsidian, RSS/news, and Spotlight.
+Prefer `autorag ui --no-open` to connect datasources. It writes the same trusted
+`datasources` / `datasourceAccess` fields as a hand-edited config, stores
+env-var names rather than secrets, and prints a loopback URL (`127.0.0.1`).
+Do not bind non-loopback hosts unless the user explicitly set `ui.allowRemote`.
+
+Datasource skills belong in trusted config and remain default-deny. Builtin
+template names are `kakao`, `whatsapp`, `telegram`, `slack`, `discord`,
+`clawgallery`, `notion`, `github`, `cloud-drive`, `gmail`, `mail-export`,
+`mailcrawl`, `obsidian`, `rss`, and `spotlight`. Config keys may be connection
+aliases with `"type": "<template>"`. Unknown names are skipped with an
+`unknown-datasource-skill` warning; they do not fail config resolution.
+`scope` and tags can narrow trusted access but cannot grant it.
 
 ```jsonc
 {
   "datasources": {
     "github": { "connector": { "repos": ["owner/repo"], "tokenEnv": "GITHUB_TOKEN" } },
-    "gdrive": { "connector": { "backend": "rclone", "remote": "gdrive:" } },
-    "cloud-drive": { "connector": { "backend": "rclone", "remote": "archive:" } },
+    "google-drive": { "type": "cloud-drive", "connector": { "provider": "google-drive", "remote": "gdrive:" } },
+    "archive-drive": { "type": "cloud-drive", "connector": { "remote": "archive:" } },
     "gmail": { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
+    "mailcrawl": { "instanceId": "personal", "connector": { "account": "personal", "mailbox": "INBOX", "binaryPath": "mailcrawl" } },
     "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
     "rss": { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
   },
   "datasourceAccess": {
-    "allowedTags": ["github", "gdrive", "cloud-drive", "gmail", "obsidian", "rss"],
-    "allowedScopes": ["/github/**", "/gdrive/**", "/cloud-drive/**", "/gmail/**", "/obsidian/**", "/rss/**"]
+    "allowedTags": ["github", "cloud-drive", "gmail", "mailcrawl", "obsidian", "rss"],
+    "allowedScopes": ["/github/**", "/google-drive/**", "/archive-drive/**", "/gmail/**", "/mailcrawl/**", "/obsidian/**", "/rss/**"]
   }
 }
 ```
 
 Tokens are environment-variable names, not raw secrets. CLI-backed connectors
-keep authentication in their external tool configuration. Unknown skill names
-fail config resolution. `scope` and tags can narrow trusted access but cannot
-grant it.
+keep authentication in their external tool configuration.
+
+Mailcrawl must be installed separately (`@nomadamas/mailcrawl@0.1.4` or newer)
+and configured through its own Himalaya account. AutoRAG runs its local `sync`
+and `index` lifecycle, then uses the mailcrawl CLI for BM25, semantic, or
+hybrid search. Do not use 0.1.3 or earlier: a no-op sync followed by `index`
+fails with `text array must be non-empty`.
 
 ## Verify and build indexes
 
 Configuration alone is not a successful setup:
 
 ```bash
-autorag status
-autorag health
-autorag refresh
-autorag search "summarize the collection" --top-k 3 --json
+autorag status --json
+autorag health --json
+autorag refresh --json
+autorag search "summarize the collection" --top-k 3 --json --debug
 ```
 
 - `status` is model-free and path-opaque.
@@ -184,7 +213,8 @@ autorag search "summarize the collection" --top-k 3 --json
 - `health --skip-probes` is only for intentionally offline validation and does
   not prove live provider access.
 - `refresh` syncs parsed mirrors, BM25, MinSync, Jikji, and authorized
-  datasources. `--method <csv>` may deliberately narrow it.
+  datasources. `--method <csv>` may deliberately narrow it
+  (`parsed,bm25,minsync,datasources,jikji,all`).
 - Use `refresh --force` for a full resync only when incremental refresh is not
   enough. Keep destructive reset/rebuild operations scoped to workspace
   `.autorag` indexes, never source documents.
@@ -192,13 +222,14 @@ autorag search "summarize the collection" --top-k 3 --json
 ## Keep indexes fresh
 
 For continuous freshness, create or verify an OS-appropriate scheduled
-`autorag watch --once` job, typically every 30 minutes. Prefer cron or launchd
-on macOS, cron or a user systemd timer on Linux, and Task Scheduler on Windows.
-Use the same config as search, avoid overlapping runs, and keep logs outside
-source trees.
+`autorag watch --once` job, typically every 15–30 minutes. Prefer cron or
+launchd on macOS, cron or a user systemd timer on Linux, and Task Scheduler on
+Windows. Use the same config as search, avoid overlapping runs, and keep logs
+outside source trees.
 
 ## Environment overrides
 
+- `AUTORAG_HOME`
 - `AUTORAG_CONFIG`
 - `AUTORAG_SEARCH_PATHS`
 - `AUTORAG_WORKSPACE`
@@ -208,7 +239,7 @@ source trees.
 
 ## Completion condition
 
-Setup is complete only when roots are approved, a non-secret single-model
-config is written, `status` is acceptable, live `health` passes, `refresh`
-builds the requested indexes, one real structured search succeeds, and any
-requested ongoing schedule is installed or verified.
+Setup is complete only when the CLI is installed, roots are approved, a
+non-secret single-model config is written, `status` is acceptable, live `health`
+passes, `refresh` builds the requested indexes, one real structured search
+succeeds, and any requested ongoing schedule is installed or verified.

package/dist/datasource/skills/discrawl/config.d.ts

@@ -1,9 +0,0 @@
-import type { DiscrawlOptions } from "./types.js";
-/**
- * Creates or updates AutoRAG's workspace-local discrawl config.
- *
- * Only the fields AutoRAG owns are changed: the workspace database path and
- * embedding enablement/provider/model. Existing operator settings such as
- * guilds, source filters, sync policy, and search mode are preserved.
- */
-export declare function ensureManagedDiscrawlConfig(configPath: string, databasePath: string, options: DiscrawlOptions): void;

package/dist/datasource/skills/gdrive/connector.d.ts

@@ -1,30 +0,0 @@
-/**
- * Google Drive connector (issue #1301).
- *
- * Lists files through the Drive v3 API with a trusted, server-configured
- * OAuth access token, exporting Google Docs/Sheets as plain text/CSV and
- * downloading plain-text files. Binary formats are skipped. Per-file export
- * failures degrade to warnings. Never throws; messages stay path/PII-opaque.
- */
-import type { ConnectorFetchResult, DatasourceConnector } from "../../connector.js";
-export interface GDriveConnectorOptions {
-    /** Drive v3 REST base; override to point at a mock server in tests. */
-    readonly baseUrl?: string;
-    /** OAuth access token; explicit value wins over {@link tokenEnv}. */
-    readonly token?: string;
-    /** Env var name holding the access token. Default `GDRIVE_ACCESS_TOKEN`. */
-    readonly tokenEnv?: string;
-    /** Restrict to files whose parent is this folder id. */
-    readonly folderId?: string;
-    /** Include shared-drive items in listings. */
-    readonly includeSharedDrives?: boolean;
-    readonly timeoutMs?: number;
-    readonly fetchImpl?: typeof fetch;
-    readonly maxPages?: number;
-    readonly maxDocuments?: number;
-}
-export declare class GDriveConnector implements DatasourceConnector {
-    private readonly options;
-    constructor(options?: GDriveConnectorOptions);
-    fetch(signal?: AbortSignal): Promise<ConnectorFetchResult>;
-}

package/dist/datasource/skills/gdrive/index.d.ts

@@ -1,3 +0,0 @@
-export { GDriveConnector, type GDriveConnectorOptions } from "./connector.js";
-export { RcloneConnector, type RcloneConnectorOptions, type RcloneRunner, type RcloneRunResult, } from "./rclone-connector.js";
-export { GDRIVE_SKILL_DEFINITION, GDriveSkill, type GDriveSkillOptions } from "./skill.js";

package/dist/datasource/skills/gdrive/skill.d.ts

@@ -1,24 +0,0 @@
-/**
- * Google Drive datasource skill (issue #1301).
- *
- * Thin composition of GDriveConnector with the shared
- * {@link ConnectorDatasourceSkill} base: descriptor, polling metadata,
- * ok/fail indexing with path/PII-opaque diagnostics, lexical retrieval via
- * the shared pipeline, opaque `/gdrive/<instance>/…` sources, and a
- * progressive-disclosure agent-skill manifest.
- */
-import type { DatasourceConnector } from "../../connector.js";
-import { ConnectorDatasourceSkill, type ConnectorSkillDefinition, type ConnectorSkillOptions } from "../../connector-skill.js";
-import { type GDriveConnectorOptions } from "./connector.js";
-export declare const GDRIVE_SKILL_DEFINITION: ConnectorSkillDefinition;
-export interface GDriveSkillOptions extends Omit<ConnectorSkillOptions, "connector"> {
-    /**
-     * Trusted pre-built connector; wins over {@link connectorOptions}. Accepts
-     * any drive-backed connector (Drive REST or the rclone CLI bridge).
-     */
-    readonly connector?: DatasourceConnector;
-    readonly connectorOptions?: GDriveConnectorOptions;
-}
-export declare class GDriveSkill extends ConnectorDatasourceSkill {
-    constructor(options?: GDriveSkillOptions);
-}