@autorag/librarian 2.2.0 → 2.4.0

package/skills/autorag-setup/SKILL.md

@@ -1,213 +1,149 @@
 ---
 name: autorag-setup
-description: Configure AutoRAG for first use or repair its setup. Detect the current agent's usable subscription-backed runtime or API provider without exposing private provider identities, select orchestrator and explorer models, propose OS-aware document folders for approval, initialize configuration, and build indexes (parsed mirrors, BM25, MinSync vectors, optional Jikji maps).
+description: Install and configure AutoRAG, or repair its single search model, approved document roots, retrieval indexes, datasource skills, and health checks without exposing credentials. Use when autorag is missing, init/refresh/health fails, indexes are stale, or the user wants to add folders or datasources.
+license: MIT
 ---
 
-# AutoRAG Setup Skill
+# AutoRAG setup
 
-Use this skill for first-time setup, missing or broken model configuration,
-provider/authentication discovery, changing indexed folders, or rebuilding the
-initial indexes. After setup succeeds, use the separate `autorag` skill for
-normal searches and feedback.
+Use this skill when AutoRAG is unconfigured, the `autorag` CLI is missing, model
+resolution fails, indexes are missing or stale, or the user wants to change the
+document collection or datasources.
 
-## Safety boundaries
+## Safety
 
 - Inspect only non-secret provider/model metadata and credential availability.
-- Never print, copy, migrate, compare, or persist credential values.
-- Never expose private provider aliases or internal model catalogs in generated
-  configuration, logs, or user-facing explanations.
+- Never print, copy, migrate, compare, or persist credential values. Store only
+  environment-variable names such as `apiKeyEnv`.
 - Do not scan the whole filesystem or home directory without explicit approval.
 - Never move, rename, edit, or delete source documents.
-- Recommend document-dense folders only. Do not index system trees, app bundles,
-  caches, or credential stores.
+- Do not index system trees, app bundles, caches, credential stores,
+  `node_modules`, `.git`, `dist`, `build`, `target`, `.cache`, `.autorag`, or
+  `.jikji`.
 
-## Detect an authenticated model runtime
+## Install the CLI if needed
 
-Do not begin by asking the user to manually name a provider. Determine the best
-usable configuration from available evidence.
+The CLI is `@autorag/librarian` (`autorag`). Runtime is Node.js ≥ 24 or Bun.
 
-1. Preserve explicit user choices and a working `~/.autorag/config.json`.
-2. Inspect the current agent's exposed provider/model capabilities.
-3. Inspect compatible non-secret local metadata, including configured model
-   registries and `~/.autorag/pi-agent/models.json`. Provider-specific local
-   configuration may be used only to determine endpoint compatibility, model
-   IDs, credential environment-variable names, and whether credentials exist.
-4. Check Pi authentication entries by provider identity without reading or
-   showing their secret payloads.
-5. Treat ChatGPT, Claude, Gemini, and other consumer subscriptions as usable
-   only when the active runtime can demonstrably delegate that authenticated
-   session to AutoRAG or compatible authentication already exists in AutoRAG's
-   Pi state. A subscription is not automatically an API entitlement.
-6. Do not infer usability from an installed CLI, a config filename, or an
-   environment-variable name alone. Authentication and protocol compatibility
-   must both be established.
-
-If no compatible authenticated runtime exists, report the exact missing public
-provider/authentication requirement. Do not write a configuration that cannot
-run. Ask one concise question only when multiple equally suitable public
-providers remain and runtime evidence cannot choose between them.
-
-## Select role models
-
-Select only models actually advertised by the authenticated runtime:
+```bash
+command -v autorag >/dev/null || bun install -g @autorag/librarian
+autorag --help
+```
 
-- `agents.orchestrator`: strongest reliable reasoning and high-context model.
-- `agents.explorer`: faster, cheaper high-recall model with sufficient context.
-- If only one usable model exists, configure it for both roles.
-- Preserve an existing working explicit pair.
-- Never invent model IDs or write a private provider alias into distributed or
-  user-facing configuration.
+If Bun is unavailable, `npm install -g @autorag/librarian` is acceptable.
 
-Provider and model ID must always be supplied together for each configured role.
-If role models are intentionally omitted, leave them unset so search-time
-resolution can still pick up an authenticated local runtime when available.
+## Inspect existing configuration
 
-## Discover and approve document folders
+Check `--config`, `AUTORAG_CONFIG`, `$AUTORAG_HOME/config.json`, or
+`~/.autorag/config.json`. Relevant fields are:
 
-Propose a **short list of recommended folders** tailored to the user's OS, then
-let the user accept, drop, or replace entries before any indexing. Explicit user
-paths always win. Reuse previously approved folders without re-asking.
+- `searchPaths`, `workspacePath`, and `memoryPath`
+- `model.provider`, `model.id`, `model.api`, `model.baseUrl`, `model.apiKeyEnv`
+- `bm25`, `minSync`, and `jikji`
+- `datasources`, `datasourceAccess`, and `ui`
 
-### 1. Detect OS and home roots
+Preserve explicit user choices and a working config unless the user asks to
+replace them or health checks fail.
 
-Resolve the user home once:
+## Configure one search model
 
-| OS | User home |
-|---|---|
-| macOS | `$HOME` (e.g. `/Users/<name>`) |
-| Linux | `$HOME` (e.g. `/home/<name>`; honor `XDG_*` when set) |
-| Windows | `%USERPROFILE%` / `$HOME` under Git Bash (e.g. `C:\Users\<name>`) |
+AutoRAG is the specialized librarian agent. One configured model plans the
+search, calls retrieval and filesystem tools, reads sources, judges evidence,
+and curates the answer in one loop. There are no orchestrator/explorer roles.
 
-Do not walk the entire home tree. Check only the recommended candidates below
-for existence and approximate document density (counts of supported extensions),
-then present the findings.
+Prefer a model with reliable tool calling and structured output, enough context
+for source excerpts, high output TPS, and low first-token latency. Use a larger
+reasoning model only when difficult synthesis or domain judgment matters more
+than latency.
 
-### 2. Recommend document-dense anchors per OS
+Start from a provider/model the current runtime can actually call. A Pi-usable
+ChatGPT, Claude, Gemini, or other authenticated subscription is valid; an
+installed CLI or subscription that Pi cannot invoke is not. For custom
+OpenAI-compatible endpoints, record the real wire API, base URL, and credential
+environment-variable name.
 
-Offer candidates that commonly hold PDFs, Office docs, notes, and downloads.
-Mark each as **recommended** or **optional**, and never enable a path that does
-not exist.
+Allowed `api` values:
 
-**macOS**
+- `openai-completions`
+- `openai-responses`
+- `anthropic-messages`
+- `openai-codex-responses`
+- `azure-openai-responses`
 
-| Priority | Path | Why |
-|---|---|---|
-| Recommended | `~/Documents` | Default document library |
-| Recommended | `~/Downloads` | Fresh PDFs, reports, attachments |
-| Recommended | `~/Desktop` | Dropped working files |
-| Optional | `~/Notes`, `~/Obsidian`, `~/iCloud Drive/Documents` when present | Personal knowledge bases |
-| Optional | Current project docs roots the user already named | Repo/wiki collections |
+If no callable setup can be established, ask the user for the provider, model
+id, API protocol, base URL when custom, and credential environment-variable
+name. Do not invent provider identities or model ids.
 
-Prefer user-visible folders over iCloud container UUIDs or Library internals.
-Skip `~/Library`, `~/Applications`, and Time Machine volumes unless the user
-explicitly points there.
+## Propose and approve document roots
 
-**Linux**
+Explicit user paths win. Otherwise inspect only these likely document-dense
+candidates for existence and approximate supported-file counts, then ask for
+approval before indexing:
 
-| Priority | Path | Why |
+| OS | Recommended | Optional |
 |---|---|---|
-| Recommended | `~/Documents` or `$XDG_DOCUMENTS_DIR` | Document library |
-| Recommended | `~/Downloads` or `$XDG_DOWNLOAD_DIR` | Incoming files |
-| Recommended | `~/Desktop` or `$XDG_DESKTOP_DIR` | Working files |
-| Optional | `~/Notes`, `~/Sync`, Nextcloud/Syncthing folders when present | Synced knowledge |
-| Optional | Current project docs roots the user already named | Repo/wiki collections |
-
-When XDG user-dirs are configured, prefer those resolved paths over bare
-`~/Documents`-style guesses.
-
-**Windows**
-
-| Priority | Path | Why |
-|---|---|---|
-| Recommended | `%USERPROFILE%\Documents` | Document library (also OneDrive Documents if that is the real shell folder) |
-| Recommended | `%USERPROFILE%\Downloads` | Incoming PDFs and exports |
-| Recommended | `%USERPROFILE%\Desktop` | Working files |
-| Optional | OneDrive `Documents` / `Desktop` when they differ from the local shell folders | Cloud-backed corp libraries |
-| Optional | Current project docs roots the user already named | Repo/wiki collections |
-
-Prefer the shell-known special folders. Do not crawl `AppData`, `Program Files`,
-or system roots.
-
-### 3. Density and format filter
-
-Prefer folders dense with formats AutoRAG parses into `.autorag/parsed` for
-BM25/MinSync:
-
-- text notes: `md`, `markdown`, `txt`, `text`
-- documents: `pdf`, `docx`, `pptx`, `xlsx`, `hwpx`, `eml`
-- optional OCR images only when enabled: `jpg`, `jpeg`, `png`, `bmp`, `tiff`
-
-Do not present legacy binary shells that currently fail pure-JS parsing as fully
-supported: bare `doc`, `xls`, and legacy `hwp` (use `docx` / `xlsx` / `hwpx`
-instead). Explorers can still open any readable text under approved paths with
-`read`/`grep`/`find`/`ls`, but those files will not contribute to BM25/MinSync
-indexes.
+| macOS | `~/Documents`, `~/Downloads`, `~/Desktop` | `~/Notes`, `~/Obsidian`, user-named project docs |
+| Linux | XDG Documents/Downloads/Desktop or their `~/` defaults | `~/Notes`, `~/Sync`, Nextcloud/Syncthing roots |
+| Windows | Documents, Downloads, Desktop shell folders | OneDrive document roots, user-named project docs |
 
-Skip generated/vendor directories including `node_modules`, `.git`, `dist`,
-`build`, `target`, `.cache`, `.autorag`, and `.jikji`.
+Supported parsed formats are `md`, `markdown`, `txt`, `text`, `pdf`, `docx`,
+`pptx`, `xlsx`, `xls`, `hwp`, `hwpx`, and `eml`. OCR for `jpg`, `jpeg`, `png`, `bmp`,
+and `tiff` is optional (`parserOptions.ocr.enabled`). Do not present legacy
+`.doc` as a supported parsed format.
 
-### 4. Present a proposal, then require approval
-
-Summarize a concrete proposal, for example:
-
-```text
-Recommended index roots (macOS):
-  [R] ~/Documents      (~120 pdf/md/docx)
-  [R] ~/Downloads      (~45 pdf/pptx)
-  [R] ~/Desktop        (~12 md/pdf)
-  [O] ~/Notes          (~80 md) — optional personal vault
-
-Reply with: accept all / keep only Documents+Downloads / custom list
-```
-
-Rules:
-
-1. Prefer an explicit directory already named by the user over any suggestion.
-2. Do not silently index large or sensitive trees.
-3. Keep the first-run set small (typically 1–3 approved roots). Users can add
-   more after the initial index build.
-4. Wallpaper/background folders are rarely useful; only suggest Desktop itself
-   (where people leave docs), not OS wallpaper asset caches.
+Keep the first-run set small, usually one to three roots. Present a concrete
+proposal and require `yes`, a narrowed keep-list, a custom list, or `skip`
+before running `refresh`.
 
 ## Initialize
 
-Write the approved paths and selected role models:
-
 ```bash
 autorag init \
-  --search-paths "/path/to/docs,/path/to/notes" \
-  --orchestrator-model-provider PROVIDER \
-  --orchestrator-model-id ORCHESTRATOR_MODEL \
-  --explorer-model-provider PROVIDER \
-  --explorer-model-id EXPLORER_MODEL
+  --search-paths "/path/to/documents,/path/to/notes" \
+  --workspace "/path/to/workspace" \
+  --model-provider PROVIDER \
+  --model-id MODEL
 ```
 
-This writes `~/.autorag/config.json` by default. Use `--config PATH` (or
-`AUTORAG_CONFIG`) for an explicit location. Optional:
+If the authenticated local runtime already supplies the intended model, model
+flags may be omitted. For a custom endpoint, add `api`, `baseUrl`, and
+`apiKeyEnv` to the single `model` object in the trusted config:
 
-- `--workspace DIR` for the workspace that owns `.autorag` indexes
-- `--memory-path FILE` for retrieval-memory storage
-- `--force` only when intentionally replacing an existing config
+```json
+{
+  "searchPaths": ["/path/to/documents"],
+  "model": {
+    "provider": "openrouter",
+    "id": "anthropic/claude-sonnet-4",
+    "api": "openai-completions",
+    "baseUrl": "https://openrouter.ai/api/v1",
+    "apiKeyEnv": "OPENROUTER_API_KEY"
+  }
+}
+```
 
-### Indexing method defaults
+Use `--force` only when intentionally replacing an existing config. Legacy cwd
+`autorag.config.json` is a migration source only and is never deleted by init.
 
-BM25 and MinSync are **enabled by default**. `autorag init` writes
-`bm25: { enabled: true }` and `minSync: { enabled: true, autoInstall: false }`
-into the config even when no method flags are supplied. To disable a method,
-set it to `false` in the config file (`"bm25": false` or `"minSync": false`).
+### Retrieval defaults
 
-MinSync auto-install is off by default (`autoInstall: false`); the binary must
-be pre-installed or available on `PATH`. AutoRAG never forces TEI or any
-external embedding service.
+BM25, MinSync, and Jikji are enabled by default. Leave them enabled unless the
+user explicitly asks otherwise. MinSync auto-installs a verified GitHub release
+into `<workspace>/.autorag/bin` on first use (`minSync.autoInstall` defaults to
+true). Set `"autoInstall": false` only when managing the binary yourself. Jikji
+auto-installs `jikji-cli` through cargo when enabled (`jikji.autoInstall`
+defaults to true; requires the Rust toolchain).
 
-### MinSync embedder flags
+Exact duplicate exclusion is enabled by default. AutoRAG invokes the external
+`dupey` CLI before parsed-mirror indexing, keeps the newest filesystem copy for
+each exact canonical-text hash, and excludes older copies from the mirror. Set
+`"excludeExactDuplicates": false` to index every copy. Missing dupey is
+non-fatal; refresh continues without this optimization.
 
-`autorag init` accepts non-secret embedder configuration flags that are written
-into `minSync.embedder` in the config:
+Optional MinSync embedder settings:
 
 ```bash
 autorag init \
-  --search-paths "/path/to/docs" \
   --embedder-id "text-embedding-3-small" \
   --embedder-base-url "https://api.openai.com/v1" \
   --embedder-api-key-env "OPENAI_API_KEY" \
@@ -215,256 +151,95 @@ autorag init \
   --embedder-batch-size 64
 ```
 
-All fields are optional; only provided fields are written. `--embedder-api-key-env`
-accepts the **environment variable name** (e.g. `OPENAI_API_KEY`), never the key
-value itself. `--embedder-dimension` and `--embedder-batch-size` must be positive
-integers.
-
-If role models are intentionally omitted, `autorag init` must leave `agents`
-unset; it must never inject a private provider default. Legacy cwd
-`autorag.config.json` is a migration source only and is never deleted by init.
-
-Per-run overrides (flags and env take precedence over the file) are available
-via the role-specific flags or:
+Only store the environment-variable name, never its value. Dimension and batch
+size must be positive integers.
 
-- `AUTORAG_ORCHESTRATOR_MODEL_PROVIDER` / `AUTORAG_ORCHESTRATOR_MODEL_ID`
-- `AUTORAG_EXPLORER_MODEL_PROVIDER` / `AUTORAG_EXPLORER_MODEL_ID`
-- legacy single-model aliases: `AUTORAG_MODEL_PROVIDER` / `AUTORAG_MODEL_ID`
-- `AUTORAG_SEARCH_PATHS`, `AUTORAG_WORKSPACE`, `AUTORAG_MEMORY_PATH`
+## Configure datasource skills when requested
 
-## Configure datasource skills (optional)
+Prefer `autorag ui --no-open` to connect datasources. It writes the same trusted
+`datasources` / `datasourceAccess` fields as a hand-edited config, stores
+env-var names rather than secrets, and prints a loopback URL (`127.0.0.1`).
+Do not bind non-loopback hosts unless the user explicitly set `ui.allowRemote`.
 
-External datasources (WhatsApp, Telegram, Slack, Discord, Notion, GitHub Issues/PRs, Google
-Drive, Gmail, local mail exports, Obsidian vaults, RSS/news) are configured
-as **datasource skills** in the trusted config file — never via model/tool
-arguments. Add a `datasources` section (skill name → config) plus a trusted
-`datasourceAccess` allow-list to `config.json`:
+Datasource skills belong in trusted config and remain default-deny. Builtin
+template names are `kakao`, `whatsapp`, `telegram`, `slack`, `discord`,
+`clawgallery`, `notion`, `github`, `cloud-drive`, `gmail`, `mail-export`,
+`mailcrawl`, `obsidian`, `rss`, and `spotlight`. Config keys may be connection
+aliases with `"type": "<template>"`. Unknown names are skipped with an
+`unknown-datasource-skill` warning; they do not fail config resolution.
+`scope` and tags can narrow trusted access but cannot grant it.
 
 ```jsonc
 {
   "datasources": {
-    "whatsapp": { "instanceId": "personal", "connector": { "binaryPath": "wacrawl" } },
-    "telegram": { "instanceId": "personal", "connector": { "binaryPath": "telecrawl" } },
-    "slack":    { "connector": { "binaryPath": "slacrawl", "configPath": "/path/to/slacrawl.yaml", "syncSource": "primary" } },
-    "discord":  { "connector": { "tokenEnv": "DISCORD_BOT_TOKEN", "guildId": "..." } },
-    "notion":   { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } },
-    "github":   { "connector": { "repos": ["owner/repo"], "tokenEnv": "GITHUB_TOKEN" } },
-    "gdrive":   { "connector": { "backend": "rclone", "remote": "gdrive:" } },
-    "gmail":    { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
-    "mail-export": { "connector": { "paths": ["/path/to/exports"] } },
+    "github": { "connector": { "repos": ["owner/repo"], "tokenEnv": "GITHUB_TOKEN" } },
+    "google-drive": { "type": "cloud-drive", "connector": { "provider": "google-drive", "remote": "gdrive:" } },
+    "archive-drive": { "type": "cloud-drive", "connector": { "remote": "archive:" } },
+    "gmail": { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } },
+    "mailcrawl": { "instanceId": "personal", "connector": { "account": "personal", "mailbox": "INBOX", "binaryPath": "mailcrawl" } },
     "obsidian": { "connector": { "vaultPath": "/path/to/vault" } },
-    "rss":      { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
+    "rss": { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } }
   },
   "datasourceAccess": {
-    "allowedTags": ["whatsapp", "telegram", "slack", "github", "rss"],
-    "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/github/**", "/rss/**"]
+    "allowedTags": ["github", "cloud-drive", "gmail", "mailcrawl", "obsidian", "rss"],
+    "allowedScopes": ["/github/**", "/google-drive/**", "/archive-drive/**", "/gmail/**", "/mailcrawl/**", "/obsidian/**", "/rss/**"]
   }
 }
 ```
 
-Rules:
-
-- REST connector tokens are configured as **environment variable names** (`tokenEnv`), never
-  raw secrets in the file. Each skill has a default env var
-  (`DISCORD_BOT_TOKEN`, `GITHUB_TOKEN`,
-  `GDRIVE_ACCESS_TOKEN`, `GMAIL_ACCESS_TOKEN`).
-- **CLI-bridge backends** (recommended where available): `gmail` accepts
-  `"backend": "himalaya"` to index any IMAP/Maildir account the external
-  [himalaya](https://pimalaya.org) CLI has configured (no OAuth plumbing);
-  `gdrive` accepts `"backend": "rclone"` to index a Google Drive remote (or
-  any of rclone's 70+ backends) configured via `rclone config` — Docs/Sheets
-  are exported as text through `--drive-export-formats`. Auth lives entirely
-  in the external tool's own config, matching the katok pattern.
-- `whatsapp` requires the external
-  [wacrawl](https://github.com/openclaw/wacrawl) binary
-  (`brew install openclaw/tap/wacrawl`). Optional trusted connector fields are
-  `binaryPath`, `databasePath`, and `sourcePath`. Live desktop ingestion is
-  macOS-only; the crawler owns Full Disk Access and archive setup.
-- `telegram` requires the external
-  [telecrawl](https://github.com/openclaw/telecrawl) binary
-  (`brew install openclaw/tap/telecrawl`). Optional trusted connector fields
-  are `binaryPath`, `databasePath`, and `sourcePath`. Live desktop ingestion
-  is macOS-only; the crawler owns Full Disk Access and archive setup.
-- `slack` requires the external
-  [slacrawl](https://github.com/openclaw/slacrawl) binary
-  (`brew install openclaw/tap/slacrawl`). Optional trusted connector fields
-  are `binaryPath`, `configPath`, and `syncSource`; Slack credentials and
-  remote source definitions stay in slacrawl's own configuration.
-- `notion` requires the external
-  [notcrawl](https://github.com/openclaw/notcrawl) binary
-  (`brew install openclaw/tap/notcrawl`). Optional trusted connector fields
-  are `binaryPath` and `configPath`; Notion credentials and workspace
-  definitions stay in notcrawl's own configuration.
-- Access is **default-deny**: without `datasourceAccess.allowedTags`, no
-  datasource skill is announced or searchable, even when configured.
-  `allowedScopes` are opaque slash-hierarchical roots like `/slack/<instance>/**`.
-- Per-skill options: `instanceId`, `pollingIntervalMs`, `tags`, and the
-  connector-specific `connector` object. `false` or `{ "enabled": false }`
-  disables an entry. Unknown skill names fail config resolution.
-- `autorag refresh --method datasources` indexes them; chunks persist under
-  `<workspace>/.autorag/datasources/<skill>/<instance>/`.
-- Verify with the manual QA harness: `bun scripts/manual-qa/run-qa.ts`
-  (see `docs/manual-qa-datasources.md`).
-
-## Build indexes after install
-
-Installation + `init` alone does **not** make search useful. Immediately after
-the approved config is written, build the local indexes before handing off to
-the `autorag` skill.
-
-### What `autorag refresh` builds
-
-`autorag refresh` is the post-setup indexing step. It is model-free and:
-
-1. **Parses** approved source files into workspace-local `.autorag/parsed`
-   markdown mirrors.
-2. **Prepares BM25** lexical indexes over those mirrors.
-3. **Embeds into the MinSync vector DB** (semantic index) over the same mirrors
-   when MinSync is configured/available.
-4. **Prepares Jikji maps/caches** under each approved source's `.jikji/` when
-   Jikji is configured (indexing only — find answers come later via
-   `jikji_find` at search time).
-5. **Indexes authorized datasources** when datasource skills are configured.
-
-Order of operations for first-time setup:
+Tokens are environment-variable names, not raw secrets. CLI-backed connectors
+keep authentication in their external tool configuration.
 
-```bash
-# 1) write approved folders + role models
-autorag init \
-  --search-paths "$HOME/Documents,$HOME/Downloads,$HOME/Desktop" \
-  --orchestrator-model-provider PROVIDER \
-  --orchestrator-model-id ORCHESTRATOR_MODEL \
-  --explorer-model-provider PROVIDER \
-  --explorer-model-id EXPLORER_MODEL
-
-# 2) verify model/provider auth and explorer subagent preflight
-autorag health
-
-# 3) parse sources, build BM25, embed MinSync, prepare Jikji maps
-autorag refresh
-
-# 4) verify corpus freshness / index health (path-opaque)
-autorag status
-```
-
-Interpret results:
-
-- `refresh` prints parse counts and BM25 readiness (and datasource rows when
-  present). Re-run with `autorag refresh --force` only when a dirty/partial
-  index needs a full rebuild path and a lighter incremental refresh is not
-  enough.
-- `--method <csv>` restricts which methods refresh runs:
-  `bm25,minsync,parsed,datasources,jikji,all`. When omitted, all methods run.
-  Parsed mirrors are always synced when BM25 or MinSync is selected (they index
-  over the parsed mirrors). Example: `autorag refresh --method bm25,minsync`.
-- `status` is model-free and path-opaque: inspect freshness and component
-  health only; do not expect absolute source paths in the output.
-- `health` checks model/provider auth and explorer subagent setup: it resolves
-  both role models, verifies credential presence, and (unless `--skip-probes`)
-  probes a completion call per role. Use it to confirm both role models resolve
-  from the authenticated runtime or written config without displaying
-  credentials or private provider details.
-- Do not claim setup succeeded when authentication, indexes, role-model
-  resolution, or subagent dispatch remain unverified.
-- Prefer bounded `refresh` over destructive resets. Reserve
-  `autorag index rebuild --yes` for a full wipe+reindex of workspace
-  `.autorag` parsed/BM25/MinSync dirs only — never against source documents.
-  `index reset` and `index rebuild` also accept `--method` to scope which
-  indexes are removed/rebuilt (e.g. `autorag index reset --method bm25 --yes`
-  removes only the BM25 index).
-
-Large first-time corpora can take several minutes for parse + embed. Stay on
-the approximate status of the running refresh rather than starting concurrent
-index jobs.
-
-## Customize and extend indexed folders
-
-Users can change the indexed set after the initial install. Always get explicit
-approval before adding paths, then re-init (or rewrite config) and refresh.
-
-### Add or replace folders
-
-1. Collect the desired absolute paths (user paste, or another short recommended
-   proposal using the OS tables above).
-2. Prefer merging with the existing approved list rather than silently dropping
-   working roots, unless the user asks to replace everything.
-3. Write the new list:
-
-```bash
-# Replace/extend the configured search roots, preserve role models already set
-autorag init \
-  --force \
-  --search-paths "/existing/docs,/new/research,/path/to/notes"
-```
-
-If role models must be preserved and you are rewriting via `init --force`, pass
-the same orchestrator/explorer provider+id pair already in
-`~/.autorag/config.json`. Alternatively edit `searchPaths` in that config when
-the agent can safely update JSON without touching credentials/private fields.
-
-4. Rebuild indexes for the new roots:
-
-```bash
-autorag refresh
-autorag status
-```
-
-### Narrow temporarily without re-init
-
-For a one-off search scope, prefer `autorag search "…" --scope <virtual-subpath>`
-through the `autorag` skill after setup. Scope narrows retrieval; it does not
-add new roots to the permanent index set.
-
-### Env/flag overrides for automation
-
-Machine-driven agents may point at a custom list without rewriting the default
-home config:
-
-```bash
-AUTORAG_SEARCH_PATHS="/docs,/notes" autorag refresh
-# or
-autorag refresh --search-paths "/docs,/notes"
-```
+Mailcrawl must be installed separately (`@nomadamas/mailcrawl@0.1.4` or newer)
+and configured through its own Himalaya account. AutoRAG runs its local `sync`
+and `index` lifecycle, then uses the mailcrawl CLI for BM25, semantic, or
+hybrid search. Do not use 0.1.3 or earlier: a no-op sync followed by `index`
+fails with `text array must be non-empty`.
 
-Remember that CLI flags / env vars take precedence for that invocation only;
-persist planned long-term folders with `autorag init` so later searches and
-refreshes share the same set.
+## Verify and build indexes
 
-### Ongoing maintenance
+Configuration alone is not a successful setup:
 
 ```bash
-autorag status                 # corpus freshness / index health
-autorag refresh                # resync after documents change
-autorag watch --once           # preferred single tick for cron / Task Scheduler / systemd timer
-autorag watch                  # long-running fs watch for interactive sessions
-autorag refresh --force        # force a full re-sync when needed
-autorag index rebuild --yes    # wipe parsed/BM25/MinSync under .autorag, then forced refresh
-autorag memory inspect         # read-only retrieval memory snapshot
+autorag status --json
+autorag health --json
+autorag refresh --json
+autorag search "summarize the collection" --top-k 3 --json --debug
 ```
 
-After setup, install an **OS-appropriate periodic job** so indexes stay fresh without a permanent daemon. Agents MUST create or verify this schedule before claiming continuous indexing works.
-
-| OS | Scheduler | Suggested tick (every 30 min) |
-|---|---|---|
-| macOS | `cron` or LaunchAgent in `~/Library/LaunchAgents` | `*/30 * * * * PATH=... NODE_OPTIONS=--max-old-space-size=16384 autorag watch --once` |
-| Linux | `cron` or `systemd --user` timer | same crontab pattern, or a oneshot unit + timer |
-| Windows | Task Scheduler | repeating task every 30 minutes running `autorag watch --once` |
-
-Prefer `watch --once`/`refresh` for scheduled ticks. Reserve long-running `watch` for interactive foreground use. Log under `~/.autorag/` (or user temp), never into source document trees. Avoid overlapping concurrent refreshes.
-
-Never run reset/rebuild against source document trees. Indexes live under the
-configured workspace's `.autorag/`; Jikji prepare caches live under
-per-source `.jikji/` when enabled.
-
-## Hand off
-
-After models authenticate, `autorag health` confirms model/provider auth and
-explorer subagent dispatch, folders are approved, `init` has written the config,
-`refresh` has built parsed + BM25 + MinSync (+ optional Jikji/datasource)
-indexes, and `status` looks healthy:
-
-- stop the setup skill
-- use the `autorag` skill for normal queries (`autorag search`, then
-  `autorag feedback` with the returned `sessionId`)
-- return to this skill when the user adds folders, fixes providers/models, or
-  needs a deliberate reindex
+- `status` is model-free and path-opaque.
+- `health` resolves the single model, checks credential presence, and normally
+  performs one live completion probe.
+- `health --skip-probes` is only for intentionally offline validation and does
+  not prove live provider access.
+- `refresh` syncs parsed mirrors, BM25, MinSync, Jikji, and authorized
+  datasources. `--method <csv>` may deliberately narrow it
+  (`parsed,bm25,minsync,datasources,jikji,all`).
+- Use `refresh --force` for a full resync only when incremental refresh is not
+  enough. Keep destructive reset/rebuild operations scoped to workspace
+  `.autorag` indexes, never source documents.
+
+## Keep indexes fresh
+
+For continuous freshness, create or verify an OS-appropriate scheduled
+`autorag watch --once` job, typically every 15–30 minutes. Prefer cron or
+launchd on macOS, cron or a user systemd timer on Linux, and Task Scheduler on
+Windows. Use the same config as search, avoid overlapping runs, and keep logs
+outside source trees.
+
+## Environment overrides
+
+- `AUTORAG_HOME`
+- `AUTORAG_CONFIG`
+- `AUTORAG_SEARCH_PATHS`
+- `AUTORAG_WORKSPACE`
+- `AUTORAG_MEMORY_PATH`
+- `AUTORAG_MODEL_PROVIDER`
+- `AUTORAG_MODEL_ID`
+
+## Completion condition
+
+Setup is complete only when the CLI is installed, roots are approved, a
+non-secret single-model config is written, `status` is acceptable, live `health`
+passes, `refresh` builds the requested indexes, one real structured search
+succeeds, and any requested ongoing schedule is installed or verified.