@autorag/librarian 2.5.1 → 2.5.2

package/skills/autorag-doctor/SKILL.md

@@ -0,0 +1,237 @@
+---
+name: autorag-doctor
+description: Diagnose and repair a broken or half-working AutoRAG install so every configured source is both indexed and searchable. Checks AutoRAG, MinSync, Jikji, the embedding gateway, and every CLI-backed datasource (katok, discrawl, slacrawl, wacrawl, telecrawl, notcrawl, qmd, mailcrawl, rclone, Spotlight), then fixes orphan locks, orphan processes, embedding dimension or identity mismatches, stale indexes, and missing setup. Use when search returns nothing or too little, refresh hangs or fails, a datasource disappeared from results, indexes look stale, the gateway will not start, or the user asks to check, diagnose, verify, or repair AutoRAG.
+license: MIT
+---
+
+# AutoRAG doctor
+
+Goal: every configured datasource is **indexed** *and* **actually returns
+hits**. A green `status` is not enough — the run is done only when a real query
+returns a real hit per datasource, or the datasource is reported as genuinely
+empty or not configured.
+
+Always finish with the status table in [Report](#report).
+
+## Safety
+
+- Never delete, move, or edit source documents.
+- Never delete a datasource's **native** store (`~/.katok`, `~/.discrawl`,
+  `~/.mailcrawl`, `.qmd`, Telegram/WhatsApp/Notion archives). AutoRAG only
+  reads them; rebuilding them is the owning CLI's job.
+- Only AutoRAG-owned state under `AUTORAG_HOME` and the workspace `.autorag`
+  directory may be reset.
+- Never print token or password values. Report credential **names** only.
+- Kill a process only after confirming it is an AutoRAG-owned orphan.
+
+## 1. Triage the core
+
+```bash
+autorag status --json
+autorag health --json
+autorag gateway status --format json
+```
+
+- `status` reports `state`, `stale`, `diagnostics`, and per-component state for
+  `minsync`, `jikji`, and `datasources`. `stale: true` or any `stale-index`
+  diagnostic means the corpus changed since the last successful refresh.
+- `health` resolves the single search model and does one live completion probe.
+  `--skip-probes` only proves config shape, never live access; do not claim a
+  healthy model from it.
+- The gateway is on-demand: `stopped` is normal when nothing is embedding.
+  `unavailable` **while** a refresh is running is a real failure.
+
+Config lives at `--config`, `AUTORAG_CONFIG`, `$AUTORAG_HOME/config.json`, or
+`~/.autorag/config.json`. Read `searchPaths`, `workspacePath`, `minSync`,
+`jikji`, `datasources`, and `datasourceAccess` before changing anything.
+
+If the CLI itself is missing or the config does not exist, stop and run the
+`autorag-setup` skill first — doctor repairs an existing install, it does not
+create one.
+
+## 2. Probe every datasource natively
+
+Each CLI owns its archive, so ask the CLI, not AutoRAG. A datasource is only
+`active` when its binary exists, its store is present, and its own check passes.
+
+| Datasource | Native check | Re-sync when empty or stale |
+|---|---|---|
+| MinSync (local docs) | `minsync status`, `minsync check`, `minsync verify` | `autorag refresh --method minsync` |
+| Jikji (discovery) | `jikji doctor` | `autorag refresh --method jikji` |
+| KakaoTalk | `katok doctor` | `katok sync && katok index` |
+| Discord | `discrawl --json metadata` | `discrawl sync` |
+| Slack | `slacrawl --json doctor` | `slacrawl sync` |
+| WhatsApp | `wacrawl --json doctor` | `wacrawl import` |
+| Telegram | `telecrawl --json doctor`, `telecrawl --json status` | `telecrawl import` |
+| Notion | `notcrawl doctor`, `notcrawl status` | `notcrawl sync --source desktop` |
+| Obsidian / notes | `qmd status` | `qmd update && qmd embed` |
+| Mail | `mailcrawl doctor`, `mailcrawl status` | `mailcrawl sync && mailcrawl index` |
+| Cloud drive | `rclone listremotes` | `autorag refresh --method datasources` |
+| Spotlight (macOS) | `mdutil -s /` | indexed by the OS; no AutoRAG sync |
+
+Rules:
+
+- A missing binary is **not configured**, not a failure. Report it with the
+  install command and move on.
+- An empty store is a legitimate zero-hit result. `telecrawl` and `slacrawl`
+  return JSON `null` (not `[]`) for no hits — treat that as empty, not broken.
+- A configured datasource whose native check fails is a **failure** and must be
+  repaired or reported explicitly. Never report a skip as a pass.
+- Zero hits with a non-empty store means the CLI's search path is suspect, not
+  the data. Cross-check against the raw index before concluding (for example
+  `slacrawl sql "select count(*) from message_fts where message_fts match 'x';"`
+  against `slacrawl search x`). A populated index plus an empty CLI result is an
+  upstream bug: report it with the exact reproduction instead of re-syncing.
+
+## 3. Prove searchability
+
+Indexing without retrieval is a failed run. Probe retrieval per datasource with
+the model-free path, then once end to end:
+
+```bash
+autorag lite retrieve "a word that certainly appears" --top-k 3 --json --debug
+autorag lite retrieve "recent topic" --tags discord --top-k 3 --json --debug
+autorag lite retrieve "recent mail subject" --scope "/mailcrawl/**" --top-k 3 --json --debug
+autorag search "summarize the collection" --top-k 3 --json --debug
+```
+
+- **Always pass `--debug` when diagnosing, and read the diagnostics.** A run
+  that silently dropped a whole retrieval method still looks successful, just
+  with fewer results; only the diagnostics name it (`minsync-unavailable`,
+  `retrieval-method-failed`). `autorag search --json` hides `diagnostics`,
+  `sessionId`, and per-result evidence unless `--debug` is set. `lite retrieve
+  --json` always carries the `diagnostics` array, but its human-readable output
+  hides it without `--debug`.
+- A method missing from the returned `method` values means that method
+  contributed nothing. During a full MinSync re-sync this is expected: the
+  store is being rebuilt, `minsync status` reports `NotSynced`, and local-file
+  hits stay absent until it finishes. Confirm with `minsync status` before
+  treating it as a failure, and never kill a running sync to "fix" it.
+- `lite retrieve` needs no model, so it isolates retrieval from model failures.
+- Use `--tags` / `--scope` to force one datasource; they can only narrow
+  trusted access, never grant it. A datasource absent from `datasourceAccess`
+  returns nothing no matter how healthy its store is — fix the config, not the
+  store.
+- `autorag evidence SESSION --json` (session id comes from `--json --debug`)
+  shows the exact chunk behind a numbered result; use it to confirm a hit is
+  real and its source is readable.
+- Local-file hits must map to an absolute, existing path. Datasource hits keep
+  source-native identities such as `/kakao/personal/chunks/42`; those are not
+  filesystem paths and must never be passed to `cat`.
+
+## 4. Repair playbook
+
+### Orphan lock
+
+The embedding runtime keeps `embedding-runtime.lock`, `embedding-runtime.pid`,
+and `embedding-runtime.port` in `AUTORAG_HOME`. It reclaims them automatically
+when the recorded PID is dead; a `lock-conflict` means the PID is **alive**.
+
+```bash
+autorag gateway status --format json
+autorag gateway stop
+```
+
+Only when `gateway stop` cannot clear it, and the recorded PID is confirmed
+dead (`kill -0 PID` fails), remove the three files by hand and retry.
+
+Index locks are owned by their engines: MinSync/tantivy locks under
+`<workspace>/.autorag/`, and per-CLI locks such as
+`<workspace>/.autorag/datasources/discrawl/.discrawl-sync.lock`. Delete one
+only after confirming no owning process is alive; otherwise wait for the run
+that holds it.
+
+### Orphan process
+
+```bash
+pgrep -fl 'autorag|autorag-gateway|minsync' | grep -v pgrep
+```
+
+A refresh that was killed mid-run can leave the gateway or a `minsync` child
+alive. Stop the gateway with `autorag gateway stop` first; only kill a PID
+directly when it is confirmed orphaned (no parent CLI, no live refresh).
+Re-run `autorag status --json` afterwards to confirm `inFlight: false`.
+
+### Embedding model mismatch
+
+`embedding-identity-mismatch` or a dimension error means the vectors on disk
+were built with a different embedder than the configured one. Vectors of two
+different dimensions can never be compared, so the index must be rebuilt:
+
+```bash
+autorag models prefetch --profile qwen3-embedding-0.6b
+autorag index rebuild --method minsync
+```
+
+The current default is the local `native:Qwen/Qwen3-Embedding-0.6B` runtime at
+1024 dimensions. A workspace still pinned to the legacy 768-dimension
+Ollama/TEI path must be reindexed explicitly, or pinned to an explicit profile.
+Changing `embedder.dimension` in config **without** a rebuild leaves retrieval
+silently empty. Keep embeddings local; do not point the embedder at a remote
+endpoint to work around a local failure.
+
+### Stale index
+
+`stale-index` diagnostics or `stale: true` mean sources changed after the last
+refresh.
+
+```bash
+autorag refresh --method parsed,minsync --json
+autorag refresh --force --json
+```
+
+Use `--force` only when incremental refresh does not clear it. For continuous
+freshness install an hourly `autorag watch --once` job (cron, launchd, systemd
+timer, or Task Scheduler).
+
+### Missing or wrong setup
+
+- `unknown-datasource-skill`: the config key is not a builtin template and has
+  no `"type"`. It is skipped, not fatal — fix the name or add `"type"`.
+- `datasource-index-failed` / `sync-failed`: the backing CLI errored. Run that
+  CLI's own check from the table above and fix it there.
+- `minsync-unavailable` / `jikji-unavailable`: the binary is missing and
+  auto-install failed. Both install through cargo; verify the Rust toolchain,
+  then `autorag refresh --method minsync` to retry.
+- `auth-error` / `rate-limited`: model or datasource credentials. Report the
+  missing environment-variable **name** and let the user supply it.
+- A datasource configured but not listed in `datasourceAccess.allowedTags` /
+  `allowedScopes` is default-denied and invisible to search. Add it there.
+
+## Diagnostic codes
+
+| Code | Meaning | First move |
+|---|---|---|
+| `stale-index` | Sources changed since last refresh | `autorag refresh --method parsed,minsync` |
+| `index-not-ready` | Index missing or never built | `autorag refresh --json` |
+| `minsync-unavailable` | MinSync binary missing or install failed | Check cargo, retry refresh |
+| `jikji-unavailable` | Jikji binary missing or install failed | Check cargo, retry refresh |
+| `embedding-identity-mismatch` | Indexed vectors use a different embedder | `autorag index rebuild --method minsync` |
+| `embedder-unavailable` | Embedding gateway or runtime down | `autorag gateway status --format json` |
+| `lock-conflict` | Another runtime holds the lock | `autorag gateway stop` |
+| `datasource-index-failed` | Backing CLI failed to index | Run that CLI's own doctor |
+| `datasource-empty` | Store has no matching content | Re-sync with the CLI, or accept as empty |
+| `unknown-datasource-skill` | Config name is not a known template | Fix the name or add `"type"` |
+| `retrieval-method-failed` | One method errored during the query | Read `--debug` diagnostics |
+| `auth-error` | Credentials missing or rejected | Report the env var name |
+
+## Report
+
+Always end with this table, one row per datasource and per AutoRAG component:
+
+| Source | Configured | Indexed | Searchable | Issue found | Fix applied |
+|---|---|---|---|---|---|
+| minsync | yes | yes | yes (3 hits) | – | – |
+| discord | yes | yes | no | stale archive | `discrawl sync` |
+| telegram | no | – | – | CLI not installed | reported |
+
+`Searchable` must come from an actual query in step 3, never inferred from
+index state. Follow the table with the exact remaining action for every row
+that is not fully green.
+
+## Completion condition
+
+Done only when: core triage is clean or every remaining diagnostic is
+explained, every configured datasource passed its native check, every one of
+them returned a real hit or is proven empty, every repair was re-verified by
+re-running the failing check, and the report table was delivered.

package/skills/autorag-lite-search/SKILL.md

@@ -58,6 +58,7 @@ The JSON envelope is:
       "content": "..."
     }
   ],
+  "unsearched": [],
   "diagnostics": []
 }
 ```
@@ -79,11 +80,32 @@ The JSON envelope is:
   `method` that produced it. Source identities are source-native: real file
   paths for local files, datasource identities for datasource results. Never
   rewrite, guess, or flatten them.
-- `diagnostics` are path-opaque: `source` is a component or method label,
-  never a real filesystem path. Codes include `retrieval-method-failed` and
-  `minsync-unavailable`. A degraded component produces a diagnostic, not a
-  failed run; check `diagnostics` before trusting an empty result set.
-- `--debug` adds diagnostic detail to human output without printing real
+- `unsearched` lists the retrieval surfaces that did not run for this query.
+  `ok: true` with a non-empty `unsearched` means the answer is partial: for
+  example local MinSync files are skipped while an index sync holds the
+  workspace lock, leaving only datasource hits. Check `unsearched.length > 0`
+  before concluding that the returned sources are the whole corpus. Each entry
+  is:
+
+  ```json
+  {
+    "surface": "minsync",
+    "methods": ["hybrid", "minsync"],
+    "reason": "MinSyncQueryError: another sync is in progress (/Users/me/corpus/.autorag/minsync)"
+  }
+  ```
+
+  `surface` is `minsync` for local files or the datasource id. `reason` is the
+  underlying failure verbatim — the CLI's failure kind, exit status, stderr and
+  real paths — so the operator can act on it directly. Report it to the user as
+  given; do not summarize it away.
+- `diagnostics` carry the same transparency: `source` is the component or
+  method label, `reason` repeats the underlying error, and codes include
+  `retrieval-method-failed` and `minsync-unavailable`. A degraded component
+  produces a diagnostic, not a failed run; check `diagnostics` before trusting
+  an empty result set.
+- `--debug` adds per-result metadata and the diagnostics list to human output.
+  Skipped-surface warnings print without it, and their reasons carry real
   filesystem paths.
 - Exit codes: 0 on success (results may be empty, and a stale index is a
   warning rather than a failure), 2 for usage, config, not-ready, or
@@ -168,6 +190,6 @@ autorag feedback <sessionId> --useful 1,3 --not-useful 2 --json
 - Never expose provider credentials or authentication payloads.
 - Treat report `source` values as opaque: persist them verbatim, never read
   the filesystem through them.
-- Keep diagnostics path-opaque; do not reconstruct or disclose real paths
-  from them.
+- Surface diagnostics and `unsearched` reasons to the user as written; they
+  carry the real error text needed to fix the failure.
 - Prefer `--json` whenever another agent consumes the output.

package/skills/autorag-lite-setup/SKILL.md

@@ -64,14 +64,17 @@ setup. Configure datasources directly in trusted config, wizard-style:
 
 1. Probe every datasource for setup feasibility before asking the user
    anything: the backing CLI exists (`katok`, `discrawl`, `slacrawl`,
-   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`), its
-   local store or archive is present, and any credentials it needs are
-   available as environment variables or in the tool's own external
-   configuration.
+   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`) and its
+   local store or archive is present. CLI-backed datasources own their own
+   archive, index, and authentication, so environment credentials (such as bot
+   tokens) are never required or checked for them. Non-CLI connectors
+   (such as `github`) require their credential environment variable
+   (`GITHUB_TOKEN`).
 2. Auto-configure every datasource that probes feasible — write its trusted
    `datasources` / `datasourceAccess` entries without asking. For example,
-   when Slack (`slacrawl`) and Discord (`discrawl`) are installed, set both up
-   automatically.
+   when Slack (`slacrawl`) and Discord (`discrawl`) are installed with local
+   stores present, set both up automatically. Discord uses discrawl's local
+   desktop wiretap archive; no Discord bot token is configured or needed.
 3. Skip every datasource that probes infeasible (for example Notion or
    Telegram when their CLIs are not installed) and always report the skipped
    list to the user, with what is missing for each.
@@ -112,11 +115,14 @@ autorag lite refresh --force --json
 - MinSync and Jikji auto-install on first use by default. If they are missing
   or broken, run a full refresh or return to setup rather than silently
   degrading to lexical-only search.
-- MinSync's default embedder is local EmbeddingGemma (768 dimensions, served
-  locally via Ollama): no embedder flags and no API key are needed, and no
-  corpus text leaves the machine. During setup, verify the local model is
-  available (`ollama pull embeddinggemma` with `ollama serve` running);
-  override it only when intentionally using a remote embedder.
+- MinSync's default embedder is in-process native Qwen3 embeddings
+  (`native:Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions, MinSync 0.4.5+): no
+  embedder flags, no API key, and no external daemon (such as Ollama) are
+  needed, and no corpus text leaves the machine. For gateway-profiled
+  embeddings, prefetch with `autorag models prefetch --profile qwen3-embedding-0.6b`.
+  The legacy Ollama/TEI adapter path (EmbeddingGemma, 768 dimensions) is for
+  manual QA only. Override the embedder only when intentionally using a
+  different provider.
 - Exact duplicate exclusion during refresh is enabled by default via the
   external `dupey` CLI. Install dupey during setup when it is missing
   (`command -v dupey || cargo install dupey`) and tell the user the feature is
@@ -174,8 +180,9 @@ autorag lite duplicates --json
 ## Unavailable components and failure handling
 
 Missing optional components degrade gracefully: refresh and retrieval continue
-with path-opaque diagnostics such as `minsync-unavailable` or
-`retrieval-method-failed` instead of failing the whole run. Exit codes are 0
+with diagnostics such as `minsync-unavailable` or `retrieval-method-failed`
+instead of failing the whole run. Each one quotes the underlying error verbatim,
+and `lite retrieve --json` also names the skipped surface under `unsearched`. Exit codes are 0
 on success, 2 for config or usage errors, and 1 for runtime errors. When a
 component stays unavailable after a full refresh, return to setup rather than
 accepting silently degraded search.

package/skills/autorag-setup/SKILL.md

@@ -143,11 +143,17 @@ is impossible, refresh continues without this optimization and the user is
 told duplicate exclusion is off. Set `"excludeExactDuplicates": false` to
 index every copy.
 
-MinSync's default embedder is local EmbeddingGemma (768 dimensions, served
-locally via Ollama). The default needs no embedder flags and no API key, and
-no corpus text is sent to a remote embedding service. During setup, verify the
-local model is available (`ollama pull embeddinggemma` with `ollama serve`
-running) instead of silently falling back.
+MinSync's default embedder is in-process native Qwen3 embeddings
+(`native:Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions, MinSync 0.4.5+). The
+default needs no embedder flags, no API key, and no external daemon (such as
+Ollama) — all embeddings run in-process locally and privately. For workspaces
+using the loopback llama-server gateway, prefetch the verified model with
+`autorag models prefetch --profile qwen3-embedding-0.6b` (or verify with
+`autorag models verify --profile qwen3-embedding-0.6b`).
+
+The legacy Ollama/TEI adapter path (EmbeddingGemma, 768 dimensions) is
+supported only for existing legacy workspaces and manual QA; do not use it as a
+fresh install default.
 
 Override the embedder only when intentionally using a different, for example
 remote, provider:
@@ -163,26 +169,36 @@ autorag init \
 
 Only store the environment-variable name, never its value. Dimension and batch
 size must be positive integers, and the dimension must match the embedder
-(EmbeddingGemma is 768; text-embedding-3-small is 1536).
+(default Qwen3 is 1024; legacy EmbeddingGemma is 768; text-embedding-3-small is 1536).
 
 ## Probe and configure datasource skills (setup wizard)
 
+Use `autorag setup` to probe the local runtime, model profile, and known
+datasources automatically:
+
+```bash
+autorag setup --format json
+```
+
 `autorag ui` is still in development — do not recommend it for datasource
 setup. Configure datasources directly in trusted config, wizard-style:
 
 1. Probe every datasource for setup feasibility before asking the user
    anything: the backing CLI exists (`katok`, `discrawl`, `slacrawl`,
-   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`), its
-   local store or archive is present, and any credentials it needs are
-   available as environment variables or in the tool's own external
-   configuration.
+   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`) and its
+   local store or archive is present. CLI-backed datasources own their own
+   archive, index, and authentication, so environment credentials (such as bot
+   tokens) are never required or checked for them. Non-CLI connectors
+   (such as `github`) require their credential environment variable
+   (`GITHUB_TOKEN`).
 2. Auto-configure every datasource that probes feasible — write its trusted
    `datasources` / `datasourceAccess` entries without asking. For example,
-   when Slack (`slacrawl`) and Discord (`discrawl`) are installed, set both up
-   automatically.
+   when Slack (`slacrawl`) and Discord (`discrawl`) are installed with local
+   stores present, set both up automatically. Discord uses discrawl's local
+   desktop wiretap archive; no Discord bot token is configured or needed.
 3. Skip every datasource that probes infeasible (for example Notion or
-   Telegram when their CLIs are not installed) and always report the skipped
-   list to the user, with what is missing for each.
+   Telegram when their CLIs or native stores are not present) and always
+   report the skipped list to the user, with what is missing for each.
 4. Set up a skipped datasource only when the user explicitly asks for it:
    install or authenticate the backing CLI first, then configure it.
 5. E-mail datasources (`mail-export`, `mailcrawl`) matter to most
@@ -230,12 +246,14 @@ datasource is removed.
 Configuration alone is not a successful setup:
 
 ```bash
+autorag setup --format json
 autorag status --json
 autorag health --json
 autorag refresh --json
 autorag search "summarize the collection" --top-k 3 --json --debug
 ```
 
+- `setup` probes runtime health, model profile, and datasource readiness.
 - `status` is model-free and path-opaque.
 - `health` resolves the single model, checks credential presence, and normally
   performs one live completion probe.