@autorag/librarian 2.4.1 → 2.5.0

package/licenses/gemma-terms-of-use.txt

@@ -0,0 +1,158 @@
+Gemma Terms of Use
+
+Source: https://ai.google.dev/gemma/terms
+Retrieved for this compliance bundle on 2026-09-14. The text below is copied from the current source page.
+
+The terms below apply to Gemma models listed in the Appendix at bottom of this page. For Gemma 4 terms, see the Gemma 4 license.
+
+Last modified: April 1, 2026
+
+By using, reproducing, modifying, distributing, performing or displaying any portion or element of Gemma, Model Derivatives including via any Hosted Service, (each as defined below) (collectively, the "Gemma Services") or otherwise accepting the terms of this Agreement, you agree to be bound by this Agreement.
+
+Section 1: DEFINITIONS
+
+1.1 Definitions
+
+(a) "Agreement" or "Gemma Terms of Use" means these terms and conditions that govern the use, reproduction, Distribution or modification of the Gemma Services and any terms and conditions incorporated by reference.
+
+(b) "Distribution" or "Distribute" means any transmission, publication, or other sharing of Gemma or Model Derivatives to a third party, including by providing or making Gemma or its functionality available as a hosted service via API, web access, or any other electronic or remote means ("Hosted Service").
+
+(c) "Gemma" means the set of machine learning language models, trained model weights and parameters identified in the Appendix, regardless of the source that you obtained it from.
+
+(d) "Google" means Google LLC.
+
+(e) "Model Derivatives" means all (i) modifications to Gemma, (ii) works based on Gemma, or (iii) any other machine learning model which is created by transfer of patterns of the weights, parameters, operations, or Output of Gemma, to that model in order to cause that model to perform similarly to Gemma, including distillation methods that use intermediate data representations or methods based on the generation of synthetic data Outputs by Gemma for training that model. For clarity, Outputs are not deemed Model Derivatives.
+
+(f) "Output" means the information content output of Gemma or a Model Derivative that results from operating or otherwise using Gemma or the Model Derivative, including via a Hosted Service.
+
+1.2
+
+As used in this Agreement, "including" means "including without limitation".
+
+Section 2: ELIGIBILITY AND USAGE
+
+2.1 Eligibility
+
+You represent and warrant that you have the legal capacity to enter into this Agreement (including being of sufficient age of consent). If you are accessing or using any of the Gemma Services for or on behalf of a legal entity, (a) you are entering into this Agreement on behalf of yourself and that legal entity, (b) you represent and warrant that you have the authority to act on behalf of and bind that entity to this Agreement and (c) references to "you" or "your" in the remainder of this Agreement refers to both you (as an individual) and that entity.
+
+2.2 Use
+
+You may use, reproduce, modify, Distribute, perform or display any of the Gemma Services only in accordance with the terms of this Agreement, and must not violate (or encourage or permit anyone else to violate) any term of this Agreement.
+
+Section 3: DISTRIBUTION AND RESTRICTIONS
+
+3.1 Distribution and Redistribution
+
+You may reproduce or Distribute copies of Gemma or Model Derivatives if you meet all of the following conditions:
+
+You must include the use restrictions referenced in Section 3.2 as an enforceable provision in any agreement (e.g., license agreement, terms of use, etc.) governing the use and/or distribution of Gemma or Model Derivatives and you must provide notice to subsequent users you Distribute to that Gemma or Model Derivatives are subject to the use restrictions in Section 3.2.
+
+You must provide all third party recipients of Gemma or Model Derivatives a copy of this Agreement.
+
+You must cause any modified files to carry prominent notices stating that you modified the files.
+
+All Distributions (other than through a Hosted Service) must be accompanied by a "Notice" text file that contains the following notice: "Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms".
+
+You may add your own intellectual property statement to your modifications and, except as set forth in this Section, may provide additional or different terms and conditions for use, reproduction, or Distribution of your modifications, or for any such Model Derivatives as a whole, provided your use, reproduction, modification, Distribution, performance, and display of Gemma otherwise complies with the terms and conditions of this Agreement. Any additional or different terms and conditions you impose must not conflict with the terms of this Agreement.
+
+3.2 Use Restrictions
+
+You must not use any of the Gemma Services:
+
+for the restricted uses set forth in the Gemma Prohibited Use Policy at ai.google.dev/gemma/prohibited_use_policy ("Prohibited Use Policy"), which is hereby incorporated by reference into this Agreement; or
+
+in violation of applicable laws and regulations.
+
+To the maximum extent permitted by law, Google reserves the right to restrict (remotely or otherwise) usage of any of the Gemma Services that Google reasonably believes are in violation of this Agreement.
+
+3.3 Generated Output
+
+Google claims no rights in Outputs you generate using Gemma. You and your users are solely responsible for Outputs and their subsequent uses.
+
+Section 4: ADDITIONAL PROVISIONS
+
+4.1 Updates
+
+Google may update Gemma from time to time.
+
+4.2 Trademarks
+
+Nothing in this Agreement grants you any rights to use Google's trademarks, trade names, logos or to otherwise suggest endorsement or misrepresent the relationship between you and Google. Google reserves any rights not expressly granted herein.
+
+4.3 DISCLAIMER OF WARRANTY
+
+UNLESS REQUIRED BY APPLICABLE LAW, THE GEMMA SERVICES, AND OUTPUTS, ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING, REPRODUCING, MODIFYING, PERFORMING, DISPLAYING OR DISTRIBUTING ANY OF THE GEMMA SERVICES OR OUTPUTS AND ASSUME ANY AND ALL RISKS ASSOCIATED WITH YOUR USE OR DISTRIBUTION OF ANY OF THE GEMMA SERVICES OR OUTPUTS AND YOUR EXERCISE OF RIGHTS AND PERMISSIONS UNDER THIS AGREEMENT.
+
+4.4 LIMITATION OF LIABILITY
+
+TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY, CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW, SHALL GOOGLE OR ITS AFFILIATES BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY, CONSEQUENTIAL, OR PUNITIVE DAMAGES, OR LOST PROFITS OF ANY KIND ARISING FROM THIS AGREEMENT OR RELATED TO, ANY OF THE GEMMA SERVICES OR OUTPUTS EVEN IF GOOGLE OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+4.5 Term, Termination, and Survival
+
+The term of this Agreement will commence upon your acceptance of this Agreement (including acceptance by your use, modification, or Distribution, reproduction, performance or display of any portion or element of the Gemma Services) and will continue in full force and effect until terminated in accordance with the terms of this Agreement. Google may terminate this Agreement if you are in breach of any term of this Agreement. Upon termination of this Agreement, you must delete and cease use and Distribution of all copies of Gemma and Model Derivatives in your possession or control. Sections 1, 2.1, 3.3, 4.2 to 4.9 shall survive the termination of this Agreement.
+
+4.6 Governing Law and Jurisdiction
+
+This Agreement will be governed by the laws of the State of California without regard to choice of law principles. The UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement. The state and federal courts of Santa Clara County, California shall have exclusive jurisdiction of any dispute arising out of this Agreement.
+
+4.7 Severability
+
+If any provision of this Agreement is held to be invalid, illegal or unenforceable, the remaining provisions shall be unaffected thereby and remain valid as if such provision had not been set forth herein.
+
+4.8 Entire Agreement
+
+This Agreement states all the terms agreed between the parties and supersedes all other agreements between the parties as of the date of acceptance relating to its subject matter.
+
+4.9 No Waiver
+
+Google will not be treated as having waived any rights by not exercising (or delaying the exercise of) any rights under this Agreement.
+
+Appendix
+
+Gemma 1
+
+Gemma 1.1
+
+Gemma 2
+
+Gemma 3
+
+Gemma 3n
+
+FunctionGemma
+
+EmbeddingGemma
+
+PaliGemma
+
+PaliGemma 2
+
+ShieldGemma
+
+ShieldGemma 2
+
+CodeGemma
+
+CodeGemma 1.1
+
+Gemma 2 JPN
+
+DataGemma RIG
+
+DataGemma RAG
+
+RecurrentGemma
+
+Gemma Scope
+
+Gemma-APS
+
+T5Gemma
+
+VaultGemma
+
+FunctionGemma
+
+T5Gemma 2
+
+TranslateGemma

package/licenses/llama.cpp-MIT.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023-2026 The ggml authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/licenses/qwen3-embedding-notice.txt

@@ -0,0 +1,13 @@
+Qwen3-Embedding-0.6B
+
+The Qwen3-Embedding-0.6B model asset is provided under the Apache License, Version 2.0. The pinned GGUF asset is published by Qwen at:
+https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF
+
+Model card and pinned asset revision:
+https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF/tree/370f27d7550e0def9b39c1f16d3fbaa13aa67728
+
+Apache License, Version 2.0:
+https://www.apache.org/licenses/LICENSE-2.0
+The text is also included in licenses/apache-2.0.txt.
+
+This notice identifies the model license and provenance. The model weights are not included in the npm package; release jobs fetch the immutable URL in licenses/embedding-assets.json and verify its SHA-256 before staging.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@autorag/librarian",
-  "version": "2.4.1",
+  "version": "2.5.0",
   "description": "Self-evolving librarian agent on Pi framework with pluggable retrieval methods",
   "license": "MIT",
   "repository": {
@@ -14,6 +14,10 @@
     ".": {
       "types": "./dist/index.d.ts",
       "default": "./dist/index.js"
+    },
+    "./core": {
+      "types": "./dist/core.d.ts",
+      "default": "./dist/core.js"
     }
   },
   "bin": {
@@ -22,31 +26,35 @@
   "files": [
     "dist",
     "skills",
-    "README.md"
+    "README.md",
+    "NOTICE",
+    "licenses"
   ],
   "publishConfig": {
     "access": "public"
   },
   "scripts": {
-    "build": "bun build src/index.ts src/cli/index.ts --outdir dist --target node --packages external && tsc -p tsconfig.build.json && node scripts/fixup-dts.mjs",
+    "build": "bun build src/index.ts src/core.ts src/cli/index.ts --outdir dist --target node --packages external && tsc -p tsconfig.build.json && node scripts/fixup-dts.mjs",
     "prepublishOnly": "bun run typecheck && bun run build",
     "prepare": "bun run build",
     "test": "node scripts/run-test-files.mjs",
     "test:windows": "node scripts/run-test-files.mjs",
     "check": "biome check --write .",
     "lint": "biome check .",
-    "typecheck": "tsc --noEmit"
+    "typecheck": "tsc --noEmit",
+    "check:embedding-manifest": "node scripts/check-embedding-manifest.mjs",
+    "supply-chain": "bun scripts/supply-chain/evaluate.ts gate --sbom sbom.local.cdx.json"
   },
   "engines": {
     "node": ">=24.0.0"
   },
   "dependencies": {
-    "@earendil-works/pi-agent-core": "^0.84.4",
-    "@earendil-works/pi-ai": "^0.84.4",
-    "@earendil-works/pi-coding-agent": "0.84.4",
-    "@earendil-works/pi-tui": "0.84.4",
+    "@earendil-works/pi-agent-core": "^0.85.1",
+    "@earendil-works/pi-ai": "^0.85.1",
+    "@earendil-works/pi-coding-agent": "0.85.1",
+    "@earendil-works/pi-tui": "0.85.1",
     "@opendataloader/pdf": "^2.5.7",
-    "@rhwp/core": "0.8.4",
+    "@rhwp/core": "0.8.6",
     "chardet": "^2.2.0",
     "fast-xml-parser": "^5.11.1",
     "iconv-lite": "^0.7.3",
@@ -63,7 +71,8 @@
     "@biomejs/biome": "2.5.12",
     "@types/mailparser": "^3.4.6",
     "@types/node": "^26.4.1",
+    "@types/ws": "^8.18.1",
     "typescript": "^7.0.2",
-    "vitest": "^4.1.11"
+    "vitest": "^5.0.0"
   }
 }

package/skills/autorag/SKILL.md

@@ -103,21 +103,27 @@ autorag feedback <sessionId> --useful 1,3 --not-useful 2 --json
 autorag status --json
 autorag health --json
 autorag refresh --json
-autorag refresh --method bm25,minsync,jikji --json
+autorag refresh --method minsync,jikji --json
 autorag watch --once --json
 autorag watch
 autorag refresh --force --json
 autorag index rebuild --yes --json
-autorag index reset --method bm25 --yes --json
+autorag index reset --method parsed --yes --json
 autorag memory inspect --json
+autorag tui
+autorag serve --force
+autorag p2p policy list --json
 ```
 
 Prefer a full refresh so parsed mirrors, MinSync, Jikji, and configured
 datasources stay aligned. `--method` accepts
-`parsed,minsync,datasources,jikji,all`. Use it only for deliberate
-narrowing. Scheduled maintenance should use non-daemon `autorag watch --once`,
-typically every 15–30 minutes, with the same config used by search and no
-overlapping runs.
+`parsed,minsync,datasources,jikji,all`. BM25 is a MinSync retrieval mode, not
+a `--method` name. Use `--method` only for deliberate narrowing. Scheduled
+maintenance should use non-daemon `autorag watch --once`, typically every
+1 hour, with the same config used by search and no overlapping runs.
+`autorag tui` is the shipped beta terminal UI. `autorag serve` / `autorag p2p`
+are opt-in SimpleX peer sharing (disabled until `p2p.enabled` is true, unless
+`--force`).
 
 Reset and rebuild commands remove only selected workspace `.autorag` indexes.
 Never target source documents. `memory inspect` is read-only and path-opaque.
@@ -131,3 +137,7 @@ Never target source documents. `memory inspect` is read-only and path-opaque.
 - Preserve real source mapping and numbered feedback identifiers.
 - Prefer `--json --debug` when another agent consumes search output or will
   call `autorag feedback`.
+- Do not invent CLI commands. `autorag --help` is the command list of record,
+  including the shipped `gateway`, `models`, `serve`, `p2p`, `tui`, and `lite`
+  commands. See `docs/embedding-runtime.md` for the local embedding runtime
+  and gateway setup.

package/skills/autorag-lite-search/SKILL.md

@@ -0,0 +1,155 @@
+---
+name: autorag-lite-search
+description: Retrieve documents from an indexed AutoRAG Lite corpus without a model, then persist externally curated reports, inspect evidence, and record numbered feedback. Use when the user wants model-free search, scoped or tag-narrowed retrieval, or the report/evidence/feedback workflow over local documents and authorized datasources.
+license: MIT
+---
+
+# AutoRAG Lite search
+
+Use this skill when AutoRAG Lite is already configured and indexed, and the
+user wants model-free retrieval, or wants to persist an externally curated
+report, inspect its evidence, or record feedback. Use `autorag-lite-setup`
+when config, indexing, or refresh is missing or broken. Use the `autorag`
+skill when a model-backed curated answer is requested.
+
+AutoRAG Lite never mutates sources. Retrieval reads local source documents and
+authorized datasource content; writes go only to workspace `.autorag/` indexes,
+the memory file, and Jikji `.jikji/` caches.
+
+## Preflight
+
+Retrieval requires a completed refresh. If `autorag lite retrieve` returns
+exit code 2 with an `index-not-ready` diagnostic, run `autorag lite refresh`
+(or `autorag refresh`) first, then retry. Confirm freshness with:
+
+```bash
+autorag lite status --json
+```
+
+`status` is model-free and path-opaque.
+
+## Retrieve
+
+```bash
+autorag lite retrieve "key findings in the Q3 report" --top-k 5 --json
+```
+
+The JSON envelope is:
+
+```json
+{
+  "ok": true,
+  "query": "key findings in the Q3 report",
+  "results": [
+    {
+      "number": 1,
+      "source": "<opaque source identifier>",
+      "method": "minsync",
+      "score": 0.83,
+      "metadata": {},
+      "content": "..."
+    }
+  ],
+  "diagnostics": []
+}
+```
+
+- `--top-k N` must be a positive integer. Invalid values exit with code 2 and
+  an `{"ok": false, "error": "..."}` rejection envelope.
+- Before any refresh, the envelope is `{"ok": false, "query": ..., "diagnostics":
+  [{"code": "index-not-ready", "severity": "error", ...}]}` with exit code 2.
+- Each result carries a provenance pair: the `source` identity and the
+  `method` that produced it. Source identities are source-native: real file
+  paths for local files, datasource identities for datasource results. Never
+  rewrite, guess, or flatten them.
+- `diagnostics` are path-opaque: `source` is a component or method label,
+  never a real filesystem path. Codes include `retrieval-method-failed` and
+  `minsync-unavailable`. A degraded component produces a diagnostic, not a
+  failed run; check `diagnostics` before trusting an empty result set.
+- `--debug` adds diagnostic detail to human output without printing real
+  filesystem paths.
+- Exit codes: 0 on success (results may be empty), 2 for usage, config, or
+  not-ready errors, 1 for runtime errors.
+
+Retrieval methods run in parallel and merge: parsed mirrors and MinSync
+lexical/vector/hybrid methods, plus configured datasource methods. Jikji is a
+local discovery/indexing preparer, not a retrieval method in the lite registry.
+`--method` selection applies to refresh, not retrieve; retrieval always queries
+every active retrieval method.
+
+## Scope and tag narrowing (default-deny)
+
+Datasource access is default-deny from trusted config
+(`datasourceAccess.allowedTags`, `allowedScopes`). Retrieval flags can only
+narrow that access, never widen it:
+
+- `--scope SCOPE` narrows datasource retrieval to a requested sub-path; it
+  cannot grant access to anything the config does not already allow.
+- `--tags tag1,tag2` further narrows already-authorized datasource results by
+  intersecting with configured allowed tags; it never grants new access.
+
+## Persisting a curated report
+
+An external agent curates the raw retrieval results, then persists the
+structured report so evidence and feedback commands work against it:
+
+```bash
+autorag lite report "key findings in the Q3 report" --input report.json --json
+cat report.json | autorag lite report "key findings in the Q3 report" --json
+```
+
+The input is the `emit_autorag_results` JSON schema: `answer`, numbered
+`results` (with `number`, `title`, `summary`, `confidence` in `[0, 1]`,
+optional `source`), and a `mapping` whose entries map each result number to
+its `method` and `source`, plus optional `evidenceRefs` and `warnings`.
+Result and mapping numbers must be one-to-one positive integers. Evidence
+confidence values must be in `[0, 1]`. Invalid input exits with code 2 and
+`{"ok": false, "error": "..."}`.
+
+`source` values in a report are opaque identifiers carried from retrieval
+output. Report persistence performs no filesystem reads against them; it
+stores the identifiers as given. Preserve the exact source-native identities
+from the retrieve envelope.
+
+On success the JSON envelope is:
+
+```json
+{
+  "ok": true,
+  "sessionId": "<uuid>",
+  "query": "key findings in the Q3 report",
+  "answer": "...",
+  "resultCount": 3
+}
+```
+
+Use the returned `sessionId` for evidence inspection and feedback.
+
+## Evidence and feedback
+
+```bash
+autorag evidence <sessionId> --result 1 --json
+autorag feedback <sessionId> --useful 1,3 --not-useful 2 --json
+```
+
+- `evidence` shows the persisted source, retrieval method, stable evidence ID,
+  excerpt or content, and any `chunkIndex`, `lineNumber`, and
+  `retrievalResultId` behind a numbered result. Omit `--result` to inspect the
+  whole session.
+- `feedback` records numbered usefulness so retrieval memory can learn.
+  Numbers refer to the report's result numbers. Supply at least one feedback
+  list.
+- Preserve real source mapping and numbered identifiers end to end: retrieve
+  output feeds the report, the report's session feeds evidence and feedback.
+
+## Rules
+
+- Refresh before retrieval; refresh again when roots change.
+- Use only configured and approved search paths and datasources.
+- Never move, rename, edit, or delete source documents.
+- Never expose provider credentials or authentication payloads.
+- Treat report `source` values as opaque: persist them verbatim, never read
+  the filesystem through them.
+- Keep diagnostics path-opaque; do not reconstruct or disclose real paths
+  from them.
+- Prefer `--json` whenever another agent consumes the output.

package/skills/autorag-lite-setup/SKILL.md

@@ -0,0 +1,182 @@
+---
+name: autorag-lite-setup
+description: Initialize, index, refresh, and maintain the model-free AutoRAG Lite lifecycle (config, roots, datasources, refresh, watch, status, index reset/rebuild) without configuring any model. Use when autorag lite init/refresh/status is needed, indexes are missing or stale, or the user wants local document indexing without a search model.
+license: MIT
+---
+
+# AutoRAG Lite setup
+
+Use this skill when the user wants AutoRAG's indexing lifecycle without a
+model: initialize a config, build and refresh indexes, watch roots, and inspect
+index health. Everything here is model-free. Use `autorag-setup` instead when a
+search model must be configured or repaired. Use `autorag-lite-search` for
+retrieval, reports, evidence, and feedback.
+
+## Safety
+
+- Inspect only non-secret config metadata: `searchPaths`, `workspacePath`,
+  `memoryPath`, `minSync`, `jikji`, `datasources`, `datasourceAccess`.
+- Never print, copy, migrate, compare, or persist credential values. Store only
+  environment-variable names such as `tokenEnv` or `apiKeyEnv`.
+- Never move, rename, edit, or delete source documents. Lite commands write
+  indexes only under the configured workspace `.autorag/` directory and
+  Jikji's per-source `.jikji/` caches.
+- Do not index system trees, app bundles, caches, credential stores,
+  `node_modules`, `.git`, `dist`, `build`, `target`, `.cache`, `.autorag`, or
+  `.jikji`.
+
+## Install the CLI if needed
+
+The CLI is `@autorag/librarian` (`autorag`). Runtime is Node.js >= 24 or Bun.
+
+```bash
+command -v autorag >/dev/null || bun install -g @autorag/librarian
+autorag lite --help
+```
+
+If Bun is unavailable, `npm install -g @autorag/librarian` is acceptable.
+
+## Initialize a model-free config
+
+```bash
+autorag lite init \
+  --search-paths "/path/to/documents,/path/to/notes" \
+  --workspace "/path/to/workspace" \
+  --memory-path "/path/to/memory.json"
+```
+
+`lite init` writes the configured model-free lifecycle config. Use `--force`
+only when intentionally replacing an existing config. Explicit user paths win;
+otherwise propose one to three document-dense roots and get approval before
+indexing. Supported parsed formats are `md`, `markdown`, `txt`, `text`, `pdf`,
+`docx`, `pptx`, `xlsx`, `xls`, `hwp`, `hwpx`, and `eml`. Legacy `.doc` is not
+a supported parsed format.
+
+Config resolution follows the usual order: `--config`, `AUTORAG_CONFIG`,
+`$AUTORAG_HOME/config.json`, or `~/.autorag/config.json`. Environment overrides
+include `AUTORAG_HOME`, `AUTORAG_CONFIG`, `AUTORAG_SEARCH_PATHS`,
+`AUTORAG_WORKSPACE`, and `AUTORAG_MEMORY_PATH`.
+
+## Probe and configure datasources (setup wizard)
+
+`autorag lite ui` is still in development — do not recommend it for datasource
+setup. Configure datasources directly in trusted config, wizard-style:
+
+1. Probe every datasource for setup feasibility before asking the user
+   anything: the backing CLI exists (`katok`, `discrawl`, `slacrawl`,
+   `wacrawl`, `telecrawl`, `notcrawl`, `qmd`, `mailcrawl`, `rclone`), its
+   local store or archive is present, and any credentials it needs are
+   available as environment variables or in the tool's own external
+   configuration.
+2. Auto-configure every datasource that probes feasible — write its trusted
+   `datasources` / `datasourceAccess` entries without asking. For example,
+   when Slack (`slacrawl`) and Discord (`discrawl`) are installed, set both up
+   automatically.
+3. Skip every datasource that probes infeasible (for example Notion or
+   Telegram when their CLIs are not installed) and always report the skipped
+   list to the user, with what is missing for each.
+4. Set up a skipped datasource only when the user explicitly asks for it:
+   install or authenticate the backing CLI first, then configure it.
+5. E-mail datasources (`mail-export`, `mailcrawl`) matter to most
+   users — always probe them and report their status, even when they end up
+   skipped.
+
+Datasource skills belong in trusted config and remain default-deny.
+
+Config keys may be builtin template names (`kakao`, `whatsapp`, `telegram`,
+`slack`, `discord`, `clawgallery`, `notion`, `github`, `cloud-drive`,
+`mail-export`, `mailcrawl`, `obsidian`, `rss`, `spotlight`) or connection
+aliases with `"type": "<template>"`. Unknown names are skipped with an
+`unknown-datasource-skill` warning; they do not fail config resolution.
+`datasourceAccess.allowedTags` and `allowedScopes` narrow trusted access and
+can never grant it. Store only env-var names such as `tokenEnv` or
+`apiKeyEnv`, never credential values.
+
+## Build and refresh indexes
+
+```bash
+autorag lite refresh --json
+autorag lite refresh --method parsed,minsync --json
+autorag lite refresh --full --json
+autorag lite refresh --force --json
+```
+
+- A plain `refresh` is incremental: it syncs parsed mirrors, MinSync, Jikji,
+  and configured datasources against what changed.
+- `--full` and `--force` both request a full resync. Use them only when
+  incremental refresh is not enough, such as after config changes to roots or
+  index settings.
+- `--method <csv>` deliberately narrows the refresh. Valid values are
+  `parsed`, `minsync`, `datasources`, `jikji`, and `all`. Omit the flag to run
+  all methods. Unknown values are rejected.
+- MinSync and Jikji auto-install on first use by default. If they are missing
+  or broken, run a full refresh or return to setup rather than silently
+  degrading to lexical-only search.
+- MinSync's default embedder is local EmbeddingGemma (768 dimensions, served
+  locally via Ollama): no embedder flags and no API key are needed, and no
+  corpus text leaves the machine. During setup, verify the local model is
+  available (`ollama pull embeddinggemma` with `ollama serve` running);
+  override it only when intentionally using a remote embedder.
+- Exact duplicate exclusion during refresh is enabled by default via the
+  external `dupey` CLI. Install dupey during setup when it is missing
+  (`command -v dupey || cargo install dupey`) and tell the user the feature is
+  available; when installation is impossible, refresh continues without it
+  and the user is told duplicate exclusion is off. Set
+  `"excludeExactDuplicates": false` to index every copy.
+
+Retrieval requires a completed refresh. `autorag lite retrieve` before any
+refresh exits with code 2 and an `index-not-ready` diagnostic; a successful
+refresh is recorded even when the corpus is empty or only a non-parsed method
+was selected. Always refresh first, and refresh again when roots change.
+Jikji is a discovery/indexing preparer, not a lite retrieval method.
+
+## Watch and scheduled freshness
+
+```bash
+autorag lite watch --once --json
+autorag lite watch
+```
+
+Prefer non-daemon `autorag lite watch --once` from cron, launchd, a systemd
+user timer, or Task Scheduler, hourly by default (every 1 hour; shorten only
+when the user asks for fresher indexes). Use the same config as retrieval,
+avoid overlapping runs, and keep logs outside source trees. `--immediate`
+triggers a first refresh on start and `--debounce-ms N` tunes change
+coalescing. Once the schedule is installed, tell the user right away that
+hourly freshness is set up.
+
+## Status and index maintenance
+
+```bash
+autorag lite status --json
+autorag lite health --json
+autorag lite index rebuild --yes --json
+autorag lite index reset --method parsed --yes --json
+autorag lite duplicates --json
+```
+
+- `status` shows path-opaque corpus freshness and index health. `health` is an
+  alias of `status`; neither resolves a model.
+- `index reset` and `index rebuild` remove or rebuild only workspace
+  `.autorag` indexes selected by `--method`. They never target source
+  documents.
+- `duplicates` scans duplicate document families read-only and never deletes
+  or moves files.
+
+## Unavailable components and failure handling
+
+Missing optional components degrade gracefully: refresh and retrieval continue
+with path-opaque diagnostics such as `minsync-unavailable` or
+`retrieval-method-failed` instead of failing the whole run. Exit codes are 0
+on success, 2 for config or usage errors, and 1 for runtime errors. When a
+component stays unavailable after a full refresh, return to setup rather than
+accepting silently degraded search.
+
+## Completion condition
+
+Setup is complete only when the CLI is installed, roots are approved, a
+non-secret model-free config is written, every datasource has been probed and
+the auto-configured and skipped lists reported to the user, dupey is installed
+or its absence reported, `refresh` has built the requested indexes, `status`
+reports healthy indexes, and any requested watch schedule is installed or
+verified with the user told it is active.