@autorag/librarian 2.2.0 → 2.4.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@autorag/librarian",
-  "version": "2.2.0",
+  "version": "2.4.0",
   "description": "Self-evolving librarian agent on Pi framework with pluggable retrieval methods",
   "license": "MIT",
   "repository": {
@@ -28,39 +28,41 @@
     "access": "public"
   },
   "scripts": {
-    "build": "bun build src/index.ts src/cli/index.ts src/subagents/explorer-tools-extension.ts --outdir dist --target node --packages external && tsc -p tsconfig.build.json && node scripts/fixup-dts.mjs",
+    "build": "bun build src/index.ts src/cli/index.ts --outdir dist --target node --packages external && tsc -p tsconfig.build.json && node scripts/fixup-dts.mjs",
     "prepublishOnly": "bun run typecheck && bun run build",
     "prepare": "bun run build",
-    "test": "vitest run",
+    "test": "node scripts/run-test-files.mjs",
+    "test:windows": "node scripts/run-test-files.mjs",
     "check": "biome check --write .",
+    "lint": "biome check .",
     "typecheck": "tsc --noEmit"
   },
   "engines": {
     "node": ">=24.0.0"
   },
   "dependencies": {
-    "@earendil-works/pi-agent-core": "^0.84.2",
-    "@earendil-works/pi-ai": "^0.84.2",
-    "@earendil-works/pi-coding-agent": "0.84.2",
-    "@opendataloader/pdf": "^2.5.2",
-    "@pngwasi/node-tantivy-binding": "^0.3.4",
+    "@earendil-works/pi-agent-core": "^0.84.4",
+    "@earendil-works/pi-ai": "^0.84.4",
+    "@earendil-works/pi-coding-agent": "0.84.4",
+    "@earendil-works/pi-tui": "0.84.4",
+    "@opendataloader/pdf": "^2.5.7",
     "@rhwp/core": "0.8.4",
     "chardet": "^2.2.0",
-    "fast-xml-parser": "^5.11.0",
+    "fast-xml-parser": "^5.11.1",
     "iconv-lite": "^0.7.3",
     "jszip": "^3.10.1",
-    "mailparser": "^3.9.15",
-    "pi-subagents": "0.53.0",
-    "slides-grab": "^1.5.0",
+    "mailparser": "^3.9.20",
+    "slides-grab": "^1.5.1",
     "smol-toml": "1.8.0",
     "tesseract.js": "^7.0.0",
-    "typebox": "^1.3.16",
+    "typebox": "^1.3.25",
+    "xlsx": "^0.18.5",
     "yaml": "^2.9.0"
   },
   "devDependencies": {
-    "@biomejs/biome": "2.5.9",
+    "@biomejs/biome": "2.5.12",
     "@types/mailparser": "^3.4.6",
-    "@types/node": "^26.2.0",
+    "@types/node": "^26.4.1",
     "typescript": "^7.0.2",
     "vitest": "^4.1.11"
   }

package/skills/autorag/SKILL.md

@@ -1,149 +1,120 @@
 ---
 name: autorag
-description: Use an already configured AutoRAG librarian agent to search, summarize, compare, and answer questions from local document collections. Use autorag-setup instead for provider/model discovery, first-time configuration, or selecting folders to index.
+description: Search, summarize, compare, and answer questions from an already configured AutoRAG librarian over local documents and authorized datasources. Use when the user asks AutoRAG to search PDFs, wikis, notes, or a knowledge base. Use autorag-setup for install, model, roots, indexing, or datasource changes.
+license: MIT
 ---
 
 # AutoRAG Librarian Skill
 
 Use this skill when AutoRAG is already configured and the user asks to search,
 summarize, compare, or answer questions from local PDFs, wikis, notes, research
-papers, or knowledge bases. For first-time configuration, missing model/provider
-settings, subscription or API-provider detection, or changing indexed folders,
-use the `autorag-setup` skill instead.
+papers, knowledge bases, or authorized datasources.
 
-AutoRAG is invoked through the `autorag` CLI. It is non-destructive: it reads
-source files and writes indexes under the configured workspace's `.autorag/`
-directory and, when enabled, Jikji's per-source `.jikji/` caches. Never move,
-rename, or delete source files.
+AutoRAG is the specialized librarian agent. One configured model plans the
+search, calls BM25, MinSync, Jikji, datasource, and filesystem tools, reads
+sources, judges evidence, and curates the final answer. There is no subagent or
+separate model role.
+
+AutoRAG reads source documents and writes indexes only under the configured
+workspace `.autorag/` directory and Jikji's per-source `.jikji/` caches. Never
+move, rename, edit, or delete source files.
 
 ## Preflight
 
-Confirm that `~/.autorag/config.json` exists (or an explicit `--config` /
-`AUTORAG_CONFIG` path). Inspect that non-secret config for usable
-`searchPaths`/`agents` without printing credentials. Then run:
+Confirm a config exists at `--config`, `AUTORAG_CONFIG`,
+`$AUTORAG_HOME/config.json`, or `~/.autorag/config.json`. Inspect only
+non-secret `searchPaths` and `model` metadata, then run:
 
 ```bash
-autorag status
-autorag health
+autorag duplicates --json
+autorag status --json
+autorag health --json
 ```
 
-`status` is model-free and path-opaque: it reports corpus freshness and index
-health, not absolute filesystem paths or role-model auth. `health` checks
-model/provider auth and explorer subagent setup: it resolves both role models,
-verifies credential presence, and (unless `--skip-probes`) probes a completion
-call per role. Use `health` to diagnose model, provider, auth, timeout, or
-subagent dispatch failures before searching. If configuration, authentication,
-role models, or indexes are missing or unhealthy, stop this workflow and use
-`autorag-setup`; do not guess private providers or model IDs.
+### Duplicate-file review
+
+Use `autorag duplicates` when the user asks to find duplicate files, choose
+likely latest copies, or reduce corpus/index space. The command and the
+`scan_duplicate_documents` Agent tool are read-only and never delete or move
+source files. Exact means dupey's canonical extracted-text hash matches;
+near/contains families require human review. Exact duplicate exclusion during
+refresh is enabled by default and can be disabled with
+`"excludeExactDuplicates": false` in `config.json`.
+
+`status` is model-free and path-opaque. `health` resolves the single model,
+checks credential presence, and normally probes one live completion. If the
+model, authentication, configuration, or indexes are unhealthy, use
+`autorag-setup` rather than guessing private provider details.
 
-## Searching
+BM25, MinSync, and Jikji should normally be healthy. MinSync and Jikji
+auto-install on first use by default. If they are missing or stale, run a full
+`autorag refresh` or return to setup rather than silently degrading to
+lexical-only search.
+
+## Search
+
+Prefer `--json --debug` when another agent will consume the result or record
+feedback. `--json` alone omits `sessionId`. `--debug` adds session/diagnostics
+fields and does not print filesystem paths.
 
 ```bash
-autorag search "what were the key findings in the Q3 report" --top-k 5
+autorag search "what were the key findings in the Q3 report" --top-k 5 --json --debug
 ```
 
-AutoRAG returns curated, numbered knowledge units grounded in sources plus a
-`sessionId` needed for feedback. Use:
+`--json --debug` includes `answer`, numbered `results` (`number`, `title`,
+`summary`, optional `source`), and `sessionId`. Use that `sessionId` for
+feedback. `--json` without `--debug` is only `answer` plus `results`.
 
-- `--scope` to narrow to a configured virtual sub-path
-- `--tags tag1,tag2` only when trusted datasource access is already configured
-  server-side; tags narrow already-allowed results and never grant new access
-- `--json` for structured output (`sessionId`, numbered `results`, `answer`)
-- `--debug` only when diagnostics are needed
+- `--scope` narrows datasource retrieval to a requested sub-path; it cannot
+  grant access.
+- `--tags` further narrows already-authorized datasource results and never
+  grants new access.
+- `--json` is required for programmatic consumption.
+- `--debug` is required for `sessionId`, evidence, and diagnostics.
 
-Do not bypass AutoRAG with ad hoc raw search when the user explicitly requested
-the librarian agent. Search requires a resolvable orchestrator/explorer model
-pair from config, flags, env, or the authenticated local runtime. When `autorag
-search` fails for a model, provider, auth, timeout, or subagent reason, the
-error output includes a hint pointing to `autorag health` for diagnosis.
+Do not bypass the librarian with ad hoc raw search when the user requested
+AutoRAG. The search loop can use Jikji, BM25, MinSync, datasource retrieval, and
+direct source reading as appropriate. If search fails because of model,
+provider, auth, or timeout problems, diagnose with `autorag health --json`.
 
-Record feedback so retrieval memory learns which results were useful:
+Record feedback so retrieval memory can learn. Numbers refer to the returned
+knowledge units. Supply at least one feedback list:
 
 ```bash
-autorag feedback <sessionId> --useful 1,3 --not-useful 2
+autorag feedback <sessionId> --useful 1,3 --not-useful 2 --json
 ```
 
-Supply at least one of `--useful` or `--not-useful`. Numbers refer to the
-numbered knowledge units from that session's search output.
-
 ## Maintenance
 
 ```bash
-autorag status
-autorag health
-autorag refresh
-autorag refresh --method bm25,minsync
-autorag watch --once
+autorag status --json
+autorag health --json
+autorag refresh --json
+autorag refresh --method bm25,minsync,jikji --json
+autorag watch --once --json
 autorag watch
-autorag refresh --force
-autorag index rebuild --yes
-autorag index reset --method bm25 --yes
-autorag memory inspect
+autorag refresh --force --json
+autorag index rebuild --yes --json
+autorag index reset --method bm25 --yes --json
+autorag memory inspect --json
 ```
 
-Use `refresh` after source documents change (parses sources and resyncs BM25 /
-MinSync / datasources / optional Jikji prepare). Prefer bounded refresh over
-reset. `watch --once` is the preferred single tick for scheduled jobs; long-running
-`watch` keeps an fs event loop open for interactive sessions.
-`--method <csv>` (e.g. `--method bm25,minsync,parsed`) restricts which methods
-refresh/index run; when omitted all methods run. BM25 and MinSync are enabled
-by default — no explicit configuration is needed for standard lexical + semantic
-retrieval. MinSync uses a pre-installed binary (`autoInstall: false`); configure
-`minSync.embedder` via `autorag init --embedder-*` flags for remote embedding.
-
-### Keep indexes fresh on a schedule (agent responsibility)
-
-There is **no always-on network service** in the CLI. Agents and operators must
-install an OS-appropriate periodic job that runs a model-free index tick:
-
-```bash
-# Preferred scheduled command (single non-daemon refresh)
-NODE_OPTIONS=--max-old-space-size=16384 autorag watch --once
-# Equivalent one-shot refresh
-NODE_OPTIONS=--max-old-space-size=16384 autorag refresh
-```
+Prefer a full refresh so parsed mirrors, BM25, MinSync, Jikji, and configured
+datasources stay aligned. `--method` accepts
+`parsed,bm25,minsync,datasources,jikji,all`. Use it only for deliberate
+narrowing. Scheduled maintenance should use non-daemon `autorag watch --once`,
+typically every 15–30 minutes, with the same config used by search and no
+overlapping runs.
 
-Schedule guidance by OS (typical interval: every 15–30 minutes; never more often
-than the corpus can finish refreshing):
-
-| OS | Preferred scheduler | Pattern |
-|---|---|---|
-| macOS | `cron` or `launchd` (`~/Library/LaunchAgents`) | `*/30 * * * * ... autorag watch --once` or a KeepAlive=false StartInterval plist |
-| Linux | `cron` / `systemd --user` timer | crontab `*/30 * * * *` or a oneshot service + timer |
-| Windows | Task Scheduler | repeating task every 30 minutes running `autorag watch --once` under the user profile |
-
-Rules for scheduled watch:
-
-1. Prefer `autorag watch --once` (or `autorag refresh`) over a permanent
-   long-running `autorag watch` daemon in user agents — daemons die on reboot
-   and are harder to supervise from skill workflows.
-2. Use the same config the search path uses (`~/.autorag/config.json` or an
-   explicit `--config` / `AUTORAG_CONFIG`).
-3. Raise Node heap for large home trees: `NODE_OPTIONS=--max-old-space-size=16384`.
-4. Redirect logs somewhere under the home/user temp tree, never into source
-   document folders.
-5. After install/setup, **create or verify** the scheduled job before claiming
-   ongoing indexing is covered. Re-check with `crontab -l`, `systemctl --user list-timers`,
-   or Task Scheduler inspection when the user asks about freshness.
-6. Do not schedule concurrent overlapping ticks; if a prior refresh is still
-   running, skip or wait (lock/log rather than stampeding MiniSync/BM25 writers).
-
-Destructive index commands:
-
-- `autorag index reset --yes` removes parsed, BM25, and MinSync directories under
-  workspace `.autorag` only. Add `--method bm25|minsync|parsed` to scope which
-  indexes are removed (e.g. `--method bm25` removes only the BM25 index).
-- `autorag index rebuild --yes` resets those indexes then forced-refreshes.
-  `--method` scopes both the reset and the rebuild refresh.
-
-Never run reset/rebuild against source documents. `memory inspect` is
-read-only and path-opaque.
+Reset and rebuild commands remove only selected workspace `.autorag` indexes.
+Never target source documents. `memory inspect` is read-only and path-opaque.
 
 ## Rules
 
 - Use only configured and approved search paths.
 - Never expose provider credentials or authentication payloads.
-- Never invent or reveal private provider names or model IDs.
-- Never treat a consumer subscription as API access without runtime evidence.
+- Never invent provider identities or model ids.
+- A Pi-usable subscription is valid; a subscription Pi cannot invoke is not.
 - Preserve real source mapping and numbered feedback identifiers.
-- Prefer `--json` when another agent must consume the result programmatically.
+- Prefer `--json --debug` when another agent consumes search output or will
+  call `autorag feedback`.