@autorag/librarian 2.5.1 → 2.5.3

package/skills/autorag-doctor/SKILL.md

@@ -0,0 +1,238 @@
+---
+name: autorag-doctor
+description: Diagnose and repair a broken or half-working AutoRAG install so every configured source is both indexed and searchable. Checks AutoRAG, MinSync, Jikji, the embedding gateway, and every CLI-backed datasource (lazykatok, discrawl, slacrawl, wacrawl, telecrawl, notcrawl, qmd, mailcrawl, rclone, Spotlight), then fixes orphan locks, orphan processes, embedding dimension or identity mismatches, stale indexes, and missing setup. Use when search returns nothing or too little, refresh hangs or fails, a datasource disappeared from results, indexes look stale, the gateway will not start, or the user asks to check, diagnose, verify, or repair AutoRAG.
+license: MIT
+---
+
+# AutoRAG doctor
+
+Goal: every configured datasource is **indexed** *and* **actually returns
+hits**. A green `status` is not enough — the run is done only when a real query
+returns a real hit per datasource, or the datasource is reported as genuinely
+empty or not configured.
+
+Always finish with the status table in [Report](#report).
+
+## Safety
+
+- Never delete, move, or edit source documents.
+- Never delete a datasource's **native** store (`~/Library/Application Support/katok`, `~/.discrawl`,
+  `~/.mailcrawl`, `.qmd`, Telegram/WhatsApp/Notion archives). AutoRAG only
+  reads them; rebuilding them is the owning CLI's job.
+- Only AutoRAG-owned state under `AUTORAG_HOME` and the workspace `.autorag`
+  directory may be reset.
+- Never print token or password values. Report credential **names** only.
+- Kill a process only after confirming it is an AutoRAG-owned orphan.
+
+## 1. Triage the core
+
+```bash
+autorag status --json
+autorag health --json
+autorag gateway status --format json
+```
+
+- `status` reports `state`, `stale`, `diagnostics`, and per-component state for
+  `minsync`, `jikji`, and `datasources`. `stale: true` or any `stale-index`
+  diagnostic means the corpus changed since the last successful refresh.
+- `health` resolves the single search model and does one live completion probe.
+  `--skip-probes` only proves config shape, never live access; do not claim a
+  healthy model from it.
+- The gateway is on-demand: `stopped` is normal when nothing is embedding.
+  `unavailable` **while** a refresh is running is a real failure.
+
+Config lives at `--config`, `AUTORAG_CONFIG`, `$AUTORAG_HOME/config.json`, or
+`~/.autorag/config.json`. Read `searchPaths`, `workspacePath`, `minSync`,
+`jikji`, `datasources`, and `datasourceAccess` before changing anything.
+
+If the CLI itself is missing or the config does not exist, stop and run the
+`autorag-setup` skill first — doctor repairs an existing install, it does not
+create one.
+
+## 2. Probe every datasource natively
+
+Each CLI owns its archive, so ask the CLI, not AutoRAG. A datasource is only
+`active` when its binary exists, its store is present, and its own check passes.
+
+| Datasource | Native check | Re-sync when empty or stale |
+|---|---|---|
+| MinSync (local docs) | `minsync status`, `minsync check`, `minsync verify` | `autorag refresh --method minsync` |
+| Jikji (discovery) | `jikji doctor` | `autorag refresh --method jikji` |
+| KakaoTalk | `lazykatok doctor` | `lazykatok sync && lazykatok index` |
+| Discord | `discrawl --json metadata` | `discrawl sync` |
+| Slack | `slacrawl --json doctor` | `slacrawl sync` |
+| WhatsApp | `wacrawl --json doctor` | `wacrawl import` |
+| Telegram | `telecrawl --json doctor`, `telecrawl --json status` | `telecrawl import` |
+| Notion | `notcrawl doctor`, `notcrawl status` | `notcrawl sync --source desktop` |
+| Obsidian / notes | `qmd status` | `qmd update && qmd embed` |
+| Mail | `mailcrawl doctor`, `mailcrawl status` | `mailcrawl sync && mailcrawl index` |
+| Cloud drive | `rclone listremotes` | `autorag refresh --method datasources` |
+| Spotlight (macOS) | `mdutil -s /` | indexed by the OS; no AutoRAG sync |
+| Lark / Feishu | `lark-cli auth status --format json` | no local sync; search is remote |
+
+Rules:
+
+- A missing binary is **not configured**, not a failure. Report it with the
+  install command and move on.
+- An empty store is a legitimate zero-hit result. `telecrawl` and `slacrawl`
+  return JSON `null` (not `[]`) for no hits — treat that as empty, not broken.
+- A configured datasource whose native check fails is a **failure** and must be
+  repaired or reported explicitly. Never report a skip as a pass.
+- Zero hits with a non-empty store means the CLI's search path is suspect, not
+  the data. Cross-check against the raw index before concluding (for example
+  `slacrawl sql "select count(*) from message_fts where message_fts match 'x';"`
+  against `slacrawl search x`). A populated index plus an empty CLI result is an
+  upstream bug: report it with the exact reproduction instead of re-syncing.
+
+## 3. Prove searchability
+
+Indexing without retrieval is a failed run. Probe retrieval per datasource with
+the model-free path, then once end to end:
+
+```bash
+autorag lite retrieve "a word that certainly appears" --top-k 3 --json --debug
+autorag lite retrieve "recent topic" --tags discord --top-k 3 --json --debug
+autorag lite retrieve "recent mail subject" --scope "/mailcrawl/**" --top-k 3 --json --debug
+autorag search "summarize the collection" --top-k 3 --json --debug
+```
+
+- **Always pass `--debug` when diagnosing, and read the diagnostics.** A run
+  that silently dropped a whole retrieval method still looks successful, just
+  with fewer results; only the diagnostics name it (`minsync-unavailable`,
+  `retrieval-method-failed`). `autorag search --json` hides `diagnostics`,
+  `sessionId`, and per-result evidence unless `--debug` is set. `lite retrieve
+  --json` always carries the `diagnostics` array, but its human-readable output
+  hides it without `--debug`.
+- A method missing from the returned `method` values means that method
+  contributed nothing. During a full MinSync re-sync this is expected: the
+  store is being rebuilt, `minsync status` reports `NotSynced`, and local-file
+  hits stay absent until it finishes. Confirm with `minsync status` before
+  treating it as a failure, and never kill a running sync to "fix" it.
+- `lite retrieve` needs no model, so it isolates retrieval from model failures.
+- Use `--tags` / `--scope` to force one datasource; they can only narrow
+  trusted access, never grant it. A datasource absent from `datasourceAccess`
+  returns nothing no matter how healthy its store is — fix the config, not the
+  store.
+- `autorag evidence SESSION --json` (session id comes from `--json --debug`)
+  shows the exact chunk behind a numbered result; use it to confirm a hit is
+  real and its source is readable.
+- Local-file hits must map to an absolute, existing path. Datasource hits keep
+  source-native identities such as `/kakao/personal/chunks/42`; those are not
+  filesystem paths and must never be passed to `cat`.
+
+## 4. Repair playbook
+
+### Orphan lock
+
+The embedding runtime keeps `embedding-runtime.lock`, `embedding-runtime.pid`,
+and `embedding-runtime.port` in `AUTORAG_HOME`. It reclaims them automatically
+when the recorded PID is dead; a `lock-conflict` means the PID is **alive**.
+
+```bash
+autorag gateway status --format json
+autorag gateway stop
+```
+
+Only when `gateway stop` cannot clear it, and the recorded PID is confirmed
+dead (`kill -0 PID` fails), remove the three files by hand and retry.
+
+Index locks are owned by their engines: MinSync/tantivy locks under
+`<workspace>/.autorag/`, and per-CLI locks such as
+`<workspace>/.autorag/datasources/discrawl/.discrawl-sync.lock`. Delete one
+only after confirming no owning process is alive; otherwise wait for the run
+that holds it.
+
+### Orphan process
+
+```bash
+pgrep -fl 'autorag|autorag-gateway|minsync' | grep -v pgrep
+```
+
+A refresh that was killed mid-run can leave the gateway or a `minsync` child
+alive. Stop the gateway with `autorag gateway stop` first; only kill a PID
+directly when it is confirmed orphaned (no parent CLI, no live refresh).
+Re-run `autorag status --json` afterwards to confirm `inFlight: false`.
+
+### Embedding model mismatch
+
+`embedding-identity-mismatch` or a dimension error means the vectors on disk
+were built with a different embedder than the configured one. Vectors of two
+different dimensions can never be compared, so the index must be rebuilt:
+
+```bash
+autorag models prefetch --profile qwen3-embedding-0.6b
+autorag index rebuild --method minsync
+```
+
+The current default is the local `native:Qwen/Qwen3-Embedding-0.6B` runtime at
+1024 dimensions. A workspace still pinned to the legacy 768-dimension
+Ollama/TEI path must be reindexed explicitly, or pinned to an explicit profile.
+Changing `embedder.dimension` in config **without** a rebuild leaves retrieval
+silently empty. Keep embeddings local; do not point the embedder at a remote
+endpoint to work around a local failure.
+
+### Stale index
+
+`stale-index` diagnostics or `stale: true` mean sources changed after the last
+refresh.
+
+```bash
+autorag refresh --method parsed,minsync --json
+autorag refresh --force --json
+```
+
+Use `--force` only when incremental refresh does not clear it. For continuous
+freshness install an hourly `autorag watch --once` job (cron, launchd, systemd
+timer, or Task Scheduler).
+
+### Missing or wrong setup
+
+- `unknown-datasource-skill`: the config key is not a builtin template and has
+  no `"type"`. It is skipped, not fatal — fix the name or add `"type"`.
+- `datasource-index-failed` / `sync-failed`: the backing CLI errored. Run that
+  CLI's own check from the table above and fix it there.
+- `minsync-unavailable` / `jikji-unavailable`: the binary is missing and
+  auto-install failed. Both install through cargo; verify the Rust toolchain,
+  then `autorag refresh --method minsync` to retry.
+- `auth-error` / `rate-limited`: model or datasource credentials. Report the
+  missing environment-variable **name** and let the user supply it.
+- A datasource configured but not listed in `datasourceAccess.allowedTags` /
+  `allowedScopes` is default-denied and invisible to search. Add it there.
+
+## Diagnostic codes
+
+| Code | Meaning | First move |
+|---|---|---|
+| `stale-index` | Sources changed since last refresh | `autorag refresh --method parsed,minsync` |
+| `index-not-ready` | Index missing or never built | `autorag refresh --json` |
+| `minsync-unavailable` | MinSync binary missing or install failed | Check cargo, retry refresh |
+| `jikji-unavailable` | Jikji binary missing or install failed | Check cargo, retry refresh |
+| `embedding-identity-mismatch` | Indexed vectors use a different embedder | `autorag index rebuild --method minsync` |
+| `embedder-unavailable` | Embedding gateway or runtime down | `autorag gateway status --format json` |
+| `lock-conflict` | Another runtime holds the lock | `autorag gateway stop` |
+| `datasource-index-failed` | Backing CLI failed to index | Run that CLI's own doctor |
+| `datasource-empty` | Store has no matching content | Re-sync with the CLI, or accept as empty |
+| `unknown-datasource-skill` | Config name is not a known template | Fix the name or add `"type"` |
+| `retrieval-method-failed` | One method errored during the query | Read `--debug` diagnostics |
+| `auth-error` | Credentials missing or rejected | Report the env var name |
+
+## Report
+
+Always end with this table, one row per datasource and per AutoRAG component:
+
+| Source | Configured | Indexed | Searchable | Issue found | Fix applied |
+|---|---|---|---|---|---|
+| minsync | yes | yes | yes (3 hits) | – | – |
+| discord | yes | yes | no | stale archive | `discrawl sync` |
+| telegram | no | – | – | CLI not installed | reported |
+
+`Searchable` must come from an actual query in step 3, never inferred from
+index state. Follow the table with the exact remaining action for every row
+that is not fully green.
+
+## Completion condition
+
+Done only when: core triage is clean or every remaining diagnostic is
+explained, every configured datasource passed its native check, every one of
+them returned a real hit or is proven empty, every repair was re-verified by
+re-running the failing check, and the report table was delivered.

package/skills/autorag-lite-search/SKILL.md

@@ -58,6 +58,7 @@ The JSON envelope is:
       "content": "..."
     }
   ],
+  "unsearched": [],
   "diagnostics": []
 }
 ```
@@ -79,11 +80,32 @@ The JSON envelope is:
   `method` that produced it. Source identities are source-native: real file
   paths for local files, datasource identities for datasource results. Never
   rewrite, guess, or flatten them.
-- `diagnostics` are path-opaque: `source` is a component or method label,
-  never a real filesystem path. Codes include `retrieval-method-failed` and
-  `minsync-unavailable`. A degraded component produces a diagnostic, not a
-  failed run; check `diagnostics` before trusting an empty result set.
-- `--debug` adds diagnostic detail to human output without printing real
+- `unsearched` lists the retrieval surfaces that did not run for this query.
+  `ok: true` with a non-empty `unsearched` means the answer is partial: for
+  example local MinSync files are skipped while an index sync holds the
+  workspace lock, leaving only datasource hits. Check `unsearched.length > 0`
+  before concluding that the returned sources are the whole corpus. Each entry
+  is:
+
+  ```json
+  {
+    "surface": "minsync",
+    "methods": ["hybrid", "minsync"],
+    "reason": "MinSyncQueryError: another sync is in progress (/Users/me/corpus/.autorag/minsync)"
+  }
+  ```
+
+  `surface` is `minsync` for local files or the datasource id. `reason` is the
+  underlying failure verbatim — the CLI's failure kind, exit status, stderr and
+  real paths — so the operator can act on it directly. Report it to the user as
+  given; do not summarize it away.
+- `diagnostics` carry the same transparency: `source` is the component or
+  method label, `reason` repeats the underlying error, and codes include
+  `retrieval-method-failed` and `minsync-unavailable`. A degraded component
+  produces a diagnostic, not a failed run; check `diagnostics` before trusting
+  an empty result set.
+- `--debug` adds per-result metadata and the diagnostics list to human output.
+  Skipped-surface warnings print without it, and their reasons carry real
   filesystem paths.
 - Exit codes: 0 on success (results may be empty, and a stale index is a
   warning rather than a failure), 2 for usage, config, not-ready, or
@@ -168,6 +190,6 @@ autorag feedback <sessionId> --useful 1,3 --not-useful 2 --json
 - Never expose provider credentials or authentication payloads.
 - Treat report `source` values as opaque: persist them verbatim, never read
   the filesystem through them.
-- Keep diagnostics path-opaque; do not reconstruct or disclose real paths
-  from them.
+- Surface diagnostics and `unsearched` reasons to the user as written; they
+  carry the real error text needed to fix the failure.
 - Prefer `--json` whenever another agent consumes the output.

package/skills/autorag-lite-setup/SKILL.md

@@ -63,15 +63,18 @@ include `AUTORAG_HOME`, `AUTORAG_CONFIG`, `AUTORAG_SEARCH_PATHS`,
 setup. Configure datasources directly in trusted config, wizard-style:
 
 1. Probe every datasource for setup feasibility before asking the user
-   anything: the backing CLI exists (`katok`, `discrawl`, `slacrawl`,
-   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`), its
-   local store or archive is present, and any credentials it needs are
-   available as environment variables or in the tool's own external
-   configuration.
+   anything: the backing CLI exists (`lazykatok`, `discrawl`, `slacrawl`,
+   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`) and its
+   local store or archive is present. CLI-backed datasources own their own
+   archive, index, and authentication, so environment credentials (such as bot
+   tokens) are never required or checked for them. Non-CLI connectors
+   (such as `github`) require their credential environment variable
+   (`GITHUB_TOKEN`).
 2. Auto-configure every datasource that probes feasible — write its trusted
    `datasources` / `datasourceAccess` entries without asking. For example,
-   when Slack (`slacrawl`) and Discord (`discrawl`) are installed, set both up
-   automatically.
+   when Slack (`slacrawl`) and Discord (`discrawl`) are installed with local
+   stores present, set both up automatically. Discord uses discrawl's local
+   desktop wiretap archive; no Discord bot token is configured or needed.
 3. Skip every datasource that probes infeasible (for example Notion or
    Telegram when their CLIs are not installed) and always report the skipped
    list to the user, with what is missing for each.
@@ -85,7 +88,7 @@ Datasource skills belong in trusted config and remain default-deny.
 
 Config keys may be builtin template names (`kakao`, `whatsapp`, `telegram`,
 `slack`, `discord`, `clawgallery`, `notion`, `github`, `cloud-drive`,
-`mail-export`, `mailcrawl`, `obsidian`, `rss`, `spotlight`) or connection
+`mail-export`, `mailcrawl`, `obsidian`, `rss`, `spotlight`, `lark`) or connection
 aliases with `"type": "<template>"`. Unknown names are skipped with an
 `unknown-datasource-skill` warning; they do not fail config resolution.
 `datasourceAccess.allowedTags` and `allowedScopes` narrow trusted access and
@@ -112,11 +115,14 @@ autorag lite refresh --force --json
 - MinSync and Jikji auto-install on first use by default. If they are missing
   or broken, run a full refresh or return to setup rather than silently
   degrading to lexical-only search.
-- MinSync's default embedder is local EmbeddingGemma (768 dimensions, served
-  locally via Ollama): no embedder flags and no API key are needed, and no
-  corpus text leaves the machine. During setup, verify the local model is
-  available (`ollama pull embeddinggemma` with `ollama serve` running);
-  override it only when intentionally using a remote embedder.
+- MinSync's default embedder is in-process native Qwen3 embeddings
+  (`native:Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions, MinSync 0.4.5+): no
+  embedder flags, no API key, and no external daemon (such as Ollama) are
+  needed, and no corpus text leaves the machine. For gateway-profiled
+  embeddings, prefetch with `autorag models prefetch --profile qwen3-embedding-0.6b`.
+  The legacy Ollama/TEI adapter path (EmbeddingGemma, 768 dimensions) is for
+  manual QA only. Override the embedder only when intentionally using a
+  different provider.
 - Exact duplicate exclusion during refresh is enabled by default via the
   external `dupey` CLI. Install dupey during setup when it is missing
   (`command -v dupey || cargo install dupey`) and tell the user the feature is
@@ -174,8 +180,9 @@ autorag lite duplicates --json
 ## Unavailable components and failure handling
 
 Missing optional components degrade gracefully: refresh and retrieval continue
-with path-opaque diagnostics such as `minsync-unavailable` or
-`retrieval-method-failed` instead of failing the whole run. Exit codes are 0
+with diagnostics such as `minsync-unavailable` or `retrieval-method-failed`
+instead of failing the whole run. Each one quotes the underlying error verbatim,
+and `lite retrieve --json` also names the skipped surface under `unsearched`. Exit codes are 0
 on success, 2 for config or usage errors, and 1 for runtime errors. When a
 component stays unavailable after a full refresh, return to setup rather than
 accepting silently degraded search.

package/skills/autorag-setup/SKILL.md

@@ -143,11 +143,17 @@ is impossible, refresh continues without this optimization and the user is
 told duplicate exclusion is off. Set `"excludeExactDuplicates": false` to
 index every copy.
 
-MinSync's default embedder is local EmbeddingGemma (768 dimensions, served
-locally via Ollama). The default needs no embedder flags and no API key, and
-no corpus text is sent to a remote embedding service. During setup, verify the
-local model is available (`ollama pull embeddinggemma` with `ollama serve`
-running) instead of silently falling back.
+MinSync's default embedder is in-process native Qwen3 embeddings
+(`native:Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions, MinSync 0.4.5+). The
+default needs no embedder flags, no API key, and no external daemon (such as
+Ollama) — all embeddings run in-process locally and privately. For workspaces
+using the loopback llama-server gateway, prefetch the verified model with
+`autorag models prefetch --profile qwen3-embedding-0.6b` (or verify with
+`autorag models verify --profile qwen3-embedding-0.6b`).
+
+The legacy Ollama/TEI adapter path (EmbeddingGemma, 768 dimensions) is
+supported only for existing legacy workspaces and manual QA; do not use it as a
+fresh install default.
 
 Override the embedder only when intentionally using a different, for example
 remote, provider:
@@ -163,26 +169,36 @@ autorag init \
 
 Only store the environment-variable name, never its value. Dimension and batch
 size must be positive integers, and the dimension must match the embedder
-(EmbeddingGemma is 768; text-embedding-3-small is 1536).
+(default Qwen3 is 1024; legacy EmbeddingGemma is 768; text-embedding-3-small is 1536).
 
 ## Probe and configure datasource skills (setup wizard)
 
+Use `autorag setup` to probe the local runtime, model profile, and known
+datasources automatically:
+
+```bash
+autorag setup --format json
+```
+
 `autorag ui` is still in development — do not recommend it for datasource
 setup. Configure datasources directly in trusted config, wizard-style:
 
 1. Probe every datasource for setup feasibility before asking the user
-   anything: the backing CLI exists (`katok`, `discrawl`, `slacrawl`,
-   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`), its
-   local store or archive is present, and any credentials it needs are
-   available as environment variables or in the tool's own external
-   configuration.
+   anything: the backing CLI exists (`lazykatok`, `discrawl`, `slacrawl`,
+   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`, `lark-cli`) and its
+   local store or archive is present. CLI-backed datasources own their own
+   archive, index, and authentication, so environment credentials (such as bot
+   tokens) are never required or checked for them. Non-CLI connectors
+   (such as `github`) require their credential environment variable
+   (`GITHUB_TOKEN`).
 2. Auto-configure every datasource that probes feasible — write its trusted
    `datasources` / `datasourceAccess` entries without asking. For example,
-   when Slack (`slacrawl`) and Discord (`discrawl`) are installed, set both up
-   automatically.
+   when Slack (`slacrawl`) and Discord (`discrawl`) are installed with local
+   stores present, set both up automatically. Discord uses discrawl's local
+   desktop wiretap archive; no Discord bot token is configured or needed.
 3. Skip every datasource that probes infeasible (for example Notion or
-   Telegram when their CLIs are not installed) and always report the skipped
-   list to the user, with what is missing for each.
+   Telegram when their CLIs or native stores are not present) and always
+   report the skipped list to the user, with what is missing for each.
 4. Set up a skipped datasource only when the user explicitly asks for it:
    install or authenticate the backing CLI first, then configure it.
 5. E-mail datasources (`mail-export`, `mailcrawl`) matter to most
@@ -191,8 +207,8 @@ setup. Configure datasources directly in trusted config, wizard-style:
 
 Datasource skills belong in trusted config and remain default-deny. Builtin
 template names are `kakao`, `whatsapp`, `telegram`, `slack`, `discord`,
-`clawgallery`, `notion`, `github`, `cloud-drive`, `mail-export`,
-`mailcrawl`, `obsidian`, `rss`, and `spotlight`. Config keys may be connection
+`clawgallery`, `notion`, `github`, `github-gist`, `cloud-drive`, `mail-export`,
+`mailcrawl`, `obsidian`, `rss`, `spotlight`, and `lark`. Config keys may be connection
 aliases with `"type": "<template>"`. Unknown names are skipped with an
 `unknown-datasource-skill` warning; they do not fail config resolution.
 `scope` and tags can narrow trusted access but cannot grant it.
@@ -201,6 +217,7 @@ aliases with `"type": "<template>"`. Unknown names are skipped with an
 {
   "datasources": {
     "github": { "connector": { "repos": ["owner/repo"], "tokenEnv": "GITHUB_TOKEN" } },
+    "github-gist": { "connector": { "tokenEnv": "GITHUB_TOKEN" } },
     "google-drive": { "type": "cloud-drive", "connector": { "provider": "google-drive", "remote": "gdrive:" } },
     "archive-drive": { "type": "cloud-drive", "connector": { "remote": "archive:" } },
     "mailcrawl": { "instanceId": "personal", "connector": { "account": "personal", "mailbox": "INBOX", "binaryPath": "mailcrawl" } },
@@ -217,11 +234,13 @@ aliases with `"type": "<template>"`. Unknown names are skipped with an
 Tokens are environment-variable names, not raw secrets. CLI-backed connectors
 keep authentication in their external tool configuration.
 
-Mailcrawl must be installed separately (`@nomadamas/mailcrawl@0.1.6` or newer)
+Mailcrawl must be installed separately (`@nomadamas/mailcrawl@0.2.0` or newer)
 and configured through its own Himalaya account. AutoRAG runs its local `sync`
 and `index` lifecycle, then uses the mailcrawl CLI for BM25, semantic, or
 hybrid search. Do not use 0.1.3 or earlier: a no-op sync followed by `index`
-fails with `text array must be non-empty`.
+fails with `text array must be non-empty`. 0.2.0 defaults to the in-process
+native `Qwen/Qwen3-Embedding-0.6B` embedder and keeps vectors in LanceDB, so a
+cold cache makes the first `index` download ONNX weights and run for minutes.
 Use mailcrawl for Gmail, IMAP, and Maildir retrieval. The former Gmail REST
 datasource is removed.
 
@@ -230,12 +249,14 @@ datasource is removed.
 Configuration alone is not a successful setup:
 
 ```bash
+autorag setup --format json
 autorag status --json
 autorag health --json
 autorag refresh --json
 autorag search "summarize the collection" --top-k 3 --json --debug
 ```
 
+- `setup` probes runtime health, model profile, and datasource readiness.
 - `status` is model-free and path-opaque.
 - `health` resolves the single model, checks credential presence, and normally
   performs one live completion probe.

package/dist/agent/search-datasource-tool.d.ts

@@ -1,29 +0,0 @@
-import type { AgentTool } from "@earendil-works/pi-agent-core";
-import { Type } from "typebox";
-import type { RetrievalDiagnostic, RetrievalResult } from "../retrieval/types.js";
-import { type SearchDocumentRetrievalTraceResult } from "./search-documents.js";
-export declare const SEARCH_DATASOURCE_DOCUMENTS_TOOL_NAME = "search_datasource_documents";
-declare const searchDatasourceSchema: Type.TObject<{
-    query: Type.TString;
-    topK: Type.TOptional<Type.TInteger>;
-    scope: Type.TOptional<Type.TString>;
-}>;
-export interface SearchDatasourceDocumentsDetails {
-    readonly method: "datasource";
-    readonly resultCount: number;
-    readonly sources: readonly string[];
-    readonly diagnostics: readonly RetrievalDiagnostic[];
-    /** Top candidates in traceable shape (additive; used for the run's retrieval trace). */
-    readonly results?: readonly SearchDocumentRetrievalTraceResult[];
-}
-export interface DatasourceSearchProvider {
-    searchDatasourceDocuments(query: string, options?: {
-        readonly topK?: number;
-        readonly scope?: string;
-    }): Promise<{
-        readonly results: readonly RetrievalResult[];
-        readonly diagnostics: readonly RetrievalDiagnostic[];
-    }>;
-}
-export declare function createSearchDatasourceDocumentsTool(provider: DatasourceSearchProvider): AgentTool<typeof searchDatasourceSchema, SearchDatasourceDocumentsDetails>;
-export {};

package/dist/datasource/skills/katok/client.d.ts

@@ -1,21 +0,0 @@
-import type { KatokChunkResult, KatokContextResult, KatokDoctorResult, KatokIndexResult, KatokOptions, KatokParentResult, KatokSearchMode, KatokSearchOptions, KatokSearchResult, KatokSyncResult } from "./types.js";
-/**
- * Thin external `katok` CLI wrapper. Every method spawns the `katok` binary as
- * a child process, parses JSON from stdout, and returns a discriminated ok/fail
- * union. No method throws for expected failures (missing binary, CLI nonzero
- * exit, timeout, oversized output, or invalid JSON). The client never opens the
- * KakaoTalk database directly.
- */
-export declare class KatokClient {
-    private readonly options;
-    constructor(options?: KatokOptions);
-    doctor(signal?: AbortSignal): Promise<KatokDoctorResult>;
-    sync(signal?: AbortSignal): Promise<KatokSyncResult>;
-    index(signal?: AbortSignal): Promise<KatokIndexResult>;
-    search(mode: KatokSearchMode, query: string, options?: KatokSearchOptions): Promise<KatokSearchResult>;
-    chunkGet(chunkId: string, signal?: AbortSignal): Promise<KatokChunkResult>;
-    context(chunkId: string, signal?: AbortSignal): Promise<KatokContextResult>;
-    parent(chunkId: string, signal?: AbortSignal): Promise<KatokParentResult>;
-    /** Single retrieval pipeline: build env, spawn, parse-free raw result. */
-    private run;
-}

package/dist/datasource/skills/katok/index.d.ts

@@ -1,12 +0,0 @@
-/**
- * KakaoTalk datasource skill barrel.
- *
- * Re-exports the retrieval methods and skill plus the underlying client/types
- * from sibling modules. Public source paths are path-opaque virtual paths of
- * the form `/kakao/<instance-id>/chunks/<chunk-id>`.
- */
-export { KatokClient } from "./client.js";
-export { KatokBm25Method, type KatokMethodOptions, type KatokSearchClient, KatokSemanticMethod } from "./methods.js";
-export { KatokSkill, type KatokSkillClient, type KatokSkillOptions } from "./skill.js";
-export type { KatokChunkResult, KatokDoctorResult, KatokFailureReason, KatokHit, KatokIndexResult, KatokOptions, KatokSearchMode, KatokSearchOptions, KatokSearchResult, KatokSyncResult, } from "./types.js";
-export { DEFAULT_KATOK_OPTIONS } from "./types.js";

package/dist/datasource/skills/katok/paths.d.ts

@@ -1,40 +0,0 @@
-/**
- * Path helpers for the KakaoTalk (`katok`) datasource skill.
- *
- * All paths are workspace-relative and live under
- * `<workspaceRoot>/.autorag/datasources/katok`. The client never exposes raw
- * filesystem paths to callers; chunk identifiers are surfaced as opaque
- * slash-hierarchical sources of the form
- * `/kakao/<instance-id>/chunks/<chunk-id>`.
- */
-export declare const AUTORAG_DIRNAME = ".autorag";
-export declare const DATASOURCES_DIRNAME = "datasources";
-export declare const KATOK_DATASOURCE_DIRNAME = "katok";
-/** Logical datasource kind for source identifiers. */
-export declare const KATOK_SOURCE_KIND: "kakao";
-/**
- * Root directory for katok-managed state under a workspace. This is where the
- * external `katok` CLI stores its index, chunks, and sync state.
- */
-export declare function katokDatasourceRoot(workspaceRoot: string): string;
-/** Directory holding the katok chunk index. */
-export declare function katokIndexPath(workspaceRoot: string): string;
-/** Directory holding materialized katok chunks. */
-export declare function katokChunksPath(workspaceRoot: string): string;
-/** Directory holding katok sync state. */
-export declare function katokSyncPath(workspaceRoot: string): string;
-/**
- * Builds the path-opaque, slash-hierarchical source identifier for a single
- * katok chunk. Callers (the KatokSkill methods) use this to populate
- * `RetrievalResult.source` so no real filesystem path ever leaks.
- */
-export declare function katokSourcePath(instanceId: string, chunkId: string): string;
-/**
- * Parses a katok source identifier back into its `{ instanceId, chunkId }`
- * components. Returns `undefined` for malformed or non-kakao sources so that
- * deny decisions remain explicit rather than undefined-as-deny.
- */
-export declare function parseKatokSourcePath(source: string): {
-    readonly instanceId: string;
-    readonly chunkId: string;
-} | undefined;