wiki-graph 0.4.1 → 0.5.0

package/dist/data/help/topics/config.jinja

@@ -11,6 +11,12 @@ Sections:
   `wikg://local/config/llm`
     LLM provider, model, optional base URL, client name, and API key.
 
+  `wikg://local/config/embeddings`
+    Embedding provider for Dense search index vectors and Dense query vectors:
+      - `provider`: `openai` or `openai-compatible`.
+      - `model`: embedding model name.
+      - `dimensions`: optional positive integer when the provider/model supports dimension control.
+
   `wikg://local/config/concurrent`
     Local concurrency settings:
       - `job`: simultaneous generation jobs. Default: 3.
@@ -25,17 +31,21 @@ Sections:
 Operations:
   Read a section:
     {{ commandName }} wikg://local/config/llm
+    {{ commandName }} wikg://local/config/embeddings
     {{ commandName }} wikg://local/config/concurrent
     {{ commandName }} wikg://local/config/wikispine
 
   Replace a section with a JSON object:
     {{ commandName }} wikg://local/config/llm set '{"provider":"openai","model":"gpt-4.1"}'
+    {{ commandName }} wikg://local/config/embeddings set '{"provider":"openai","model":"text-embedding-3-small"}'
     {{ commandName }} wikg://local/config/concurrent set '{"job":3,"request":6}'
     {{ commandName }} wikg://local/config/wikispine set '{"provider":"fetch"}'
 
   Update one key:
     {{ commandName }} wikg://local/config/llm put provider openai
     {{ commandName }} wikg://local/config/llm put model gpt-4.1
+    {{ commandName }} wikg://local/config/embeddings put provider openai
+    {{ commandName }} wikg://local/config/embeddings put model text-embedding-3-small
     {{ commandName }} wikg://local/config/concurrent put job 3
     {{ commandName }} wikg://local/config/concurrent put request 6
     {{ commandName }} wikg://local/config/wikispine put provider cli
@@ -43,6 +53,7 @@ Operations:
 
   Store the API key:
     {{ commandName }} wikg://local/config/llm put apiKey --secret
+    {{ commandName }} wikg://local/config/embeddings put apiKey --secret
 
 Secret handling:
   - `apiKey` is read through an interactive hidden prompt.
@@ -56,6 +67,18 @@ Minimum LLM-backed operation configuration:
   - `llm.model`
   - Provider credentials when required by the selected provider.
 
+Minimum Dense embedding configuration:
+  - `embeddings.provider`
+  - `embeddings.model`
+  - `embeddings.baseURL` when using `openai-compatible`
+  - Provider credentials when required by the selected provider.
+
+Dense search notes:
+  - FTS artifact builds do not use this section.
+  - Source and summary embedding artifact builds require this section.
+  - Index cache sync does not call the embeddings provider; it uses vectors already stored in artifacts.
+  - Query uses this section when the current index cache includes Dense vectors. If Dense query embedding fails, Hybrid query can still fall back to FTS.
+
 One-run overrides:
   `--llm <json>` can provide an LLM object for one invocation.
   It accepts direct LLM objects and common OpenAI-compatible aliases such as `baseUrl`.
@@ -68,6 +91,10 @@ Examples:
   {{ commandName }} wikg://local/config/llm put model gpt-4.1
   {{ commandName }} wikg://local/config/llm put apiKey --secret
   {{ commandName }} wikg://local/config/llm test
+  {{ commandName }} wikg://local/config/embeddings put provider openai
+  {{ commandName }} wikg://local/config/embeddings put model text-embedding-3-small
+  {{ commandName }} wikg://local/config/embeddings put apiKey --secret
+  {{ commandName }} wikg://local/config/embeddings test
   {{ commandName }} wikg://local/config/wikispine put provider fetch
   {{ commandName }} wikg://local/config/wikispine test
 

package/dist/data/help/topics/library.jinja

@@ -54,6 +54,7 @@ Library-wide query:
   - `{{ commandName }} wikg://lib/entity --query <query>` searches across present archive members through the library index.
   - `{{ commandName }} wikg://lib/triple --query <query>` searches library-wide Knowledge Graph triples.
   - `{{ commandName }} wikg://lib/chunk --query <query>` searches library-wide Reading Graph chunks.
+  - Add `--skip-unindexed` only when you intentionally want to search indexed chapters and ignore uncovered chapters.
   - Library-wide object predicates such as `evidence`, `related`, and `pack` use the same aggregate library index.
   - Library-wide object URIs such as `wikg://lib/entity/<qid>` are aggregate views over matching objects from present archive members.
   - Evidence and related output should be interpreted with the returned archive/member provenance; cite the specific returned evidence or member, not just the library-wide URI.
@@ -61,18 +62,21 @@ Library-wide query:
 
 Library index lifecycle:
   - `{{ commandName }} <library-uri>/index` reads library index status.
-  - `{{ commandName }} <library-uri>/index enable` rebuilds the aggregate index from registered present archives.
-  - `{{ commandName }} <library-uri>/index disable` removes the local library index materialization.
+  - `{{ commandName }} <library-uri>/index sync` rebuilds the aggregate index cache from registered present archives' artifacts.
+  - `{{ commandName }} <library-uri>/index clean` removes the local library index cache.
   - Library indexes are local derived state under Wiki Graph staging, not files inside the library folder.
-  - `scan`, `add`, archive `path set`, archive `remove`, and archive writes can make the aggregate index dirty.
-  - Enable the library index again when library-wide query reports that the index is missing, dirty, or stale.
-  - Library indexes do not support archive-style `embed` or `external` modes.
+  - Sync does not build missing chapter artifacts and does not call an embeddings provider.
+  - Dense query still requires `wikg://local/config/embeddings` to embed the query text.
+  - When both FTS and Dense exist, library query combines them automatically; there is no separate query mode flag.
+  - Library membership changes can make the aggregate index stale.
+  - Managed archive writes try to sync an existing library index cache immediately; if sync cannot complete, library-wide query reports the cache as stale or not ready.
+  - Sync the library index again when library-wide query reports that the index is missing, stale, or not ready.
 
 Concurrency:
   - Library registry and membership writes hold the library write lock.
-  - `arc scan`, `arc add`, library `path set`, library archive `path set`, library archive `remove`, and library index `enable` or `disable` cannot run concurrently for the same library.
+  - `arc scan`, `arc add`, library `path set`, library archive `path set`, library archive `remove`, and library index `sync` or `clean` cannot run concurrently for the same library.
   - Library-wide object `--query`, object scope enumeration, evidence, related, and pack commands read the current aggregate index.
-  - If a write marks the aggregate index dirty, rebuild it with `{{ commandName }} <library-uri>/index enable` before relying on library-wide query results.
+  - If a write cannot resync the aggregate index, rebuild it with `{{ commandName }} <library-uri>/index sync` before relying on library-wide query results.
 
 Upgrade and cleanup:
   - Upgrade a whole library with `{{ commandName }} maintenance upgrade wikg://lib` or `{{ commandName }} maintenance upgrade wikg://lib/<lib-id>`.

package/dist/data/help/topics/readiness.jinja

@@ -10,47 +10,67 @@ Purpose:
 Start with inspect:
   {{ commandName }} <archive-uri> inspect
 
-Archive FTS readiness:
-  Archive `--query` uses that archive's FTS index. Without a current index, keyword retrieval is unavailable for archive scopes, evidence, and related results.
-
-  Default behavior:
-    `{{ commandName }} <archive-uri>/index enable` enables the searchable index as local cache outside the `.wikg` archive.
-    This keeps the archive smaller, but the index does not travel with copied archive files.
-    If another user receives the `.wikg`, they usually need to enable their own local index.
-    After an index is enabled, CLI archive writes keep it synchronized automatically.
-    Do not enable the index again after every generation job.
-
-  Portable behavior:
-    `{{ commandName }} <archive-uri>/index embed` switches to an archive-embedded searchable index and ensures it is current.
-    This makes the archive more self-contained, but increases file size.
-    After embedding, CLI archive writes keep the embedded index synchronized automatically.
-    If you are unsure, enable the local index first and embed only when distribution convenience matters.
+Archive index readiness:
+  Ordinary archive source query needs chapter index artifacts. For each content chapter, at least one of these must be current:
+  - FTS artifact: `<chapter-uri>/index/fts`
+  - Source embedding artifact: `<chapter-uri>/index/embedding/source`
+
+  Summary semantic retrieval is separate:
+  - Summary embedding artifact: `<chapter-uri>/index/embedding/summary`
+  - Summary embedding alone does not satisfy ordinary archive source query readiness.
+
+  Artifact behavior:
+    Chapter artifacts are stored inside the `.wikg` archive.
+    Build FTS artifacts with the chapter index URI or queue target `index-fts`.
+    Build source embedding artifacts with the chapter index URI or queue target `index-embedding-source`.
+    Build summary embedding artifacts with the chapter index URI or queue target `index-embedding-summary`.
+    Building embedding artifacts requires `wikg://local/config/embeddings` and may call the configured provider.
+
+  Cache behavior:
+    `{{ commandName }} <archive-uri>/index sync` builds or repairs local `index.db` cache from artifacts.
+    Sync does not build missing artifacts and does not call an embeddings provider.
+    Archive-level query can create a missing cache from current artifacts on first use, but explicit sync avoids the first-query delay.
+    If query reports a missing or outdated cache, run `{{ commandName }} <archive-uri>/index sync`; library query always requires a current library cache.
+    `{{ commandName }} <archive-uri>/index clean` deletes only the local cache.
+
+  Query behavior:
+    FTS artifact only: FTS query.
+    Source embedding artifact only: semantic/Dense-only query.
+    Both artifacts: Hybrid query with automatic fusion.
+    Broad query is grouped by result type first: chapter/title, object, chunk, then source/summary text.
+    FTS, Dense, and Hybrid ranking affect ordering inside those result groups and evidence-backed object ranking; they do not flatten the type groups into one global rerank.
+    Missing both FTS and source embedding for any content chapter in scope: ordinary query fails.
+    Add `--skip-unindexed` to a scope query, `related --query`, or `evidence --query` only when you want to search indexed chapters and ignore uncovered chapters.
+    Results produced with `--skip-unindexed` are intentionally partial; inspect coverage before treating absent chapters as negative evidence.
+    Summary embedding artifacts are for summary semantic retrieval and are not the minimum requirement for ordinary source query.
 
   Commands:
     {{ commandName }} <archive-uri>/index
-    {{ commandName }} <archive-uri>/index enable --help
-    {{ commandName }} <archive-uri>/index disable --help
-    {{ commandName }} <archive-uri>/index embed --help
-    {{ commandName }} <archive-uri>/index external --help
+    {{ commandName }} <chapter-uri>/index/fts --help
+    {{ commandName }} <chapter-uri>/index/embedding/source --help
+    {{ commandName }} <chapter-uri>/index/embedding/summary --help
+    {{ commandName }} <archive-uri>/index sync --help
+    {{ commandName }} <archive-uri>/index clean --help
 
 Library index readiness:
-  Library-wide scopes use an aggregate library index instead of one standalone archive index.
-  Without a current library index, library-wide `--query`, evidence, related, pack, and scope enumeration may be unavailable.
+  Library-wide scopes use an aggregate library index cache built from member archives' artifacts.
+  Unlike a free archive, library query does not lazy-sync this cache. It must already be current.
 
   Default behavior:
-    `{{ commandName }} wikg://lib/index enable` rebuilds the default library index from registered present archives.
-    `{{ commandName }} wikg://lib/<lib-id>/index enable` rebuilds a non-default library index.
+    `{{ commandName }} wikg://lib/index sync` rebuilds the default library index cache from registered present archives.
+    `{{ commandName }} wikg://lib/<lib-id>/index sync` rebuilds a non-default library index cache.
     The index is local derived state under Wiki Graph staging, not a file inside the library folder.
+    Sync does not build missing chapter artifacts and does not call an embeddings provider.
 
-  Dirty behavior:
-    `scan`, `add`, archive `path set`, archive `remove`, archive writes, and membership changes can mark the aggregate index dirty.
-    Re-enable the library index when library-wide query reports that it is missing, dirty, or stale.
-    Library indexes do not support archive-style `embed` or `external` modes.
+  Stale behavior:
+    Library membership changes can make the aggregate index stale.
+    Managed archive writes try to sync an existing library index cache immediately.
+    Sync the library index when library-wide query reports that it is missing, stale, or not ready.
 
   Commands:
     {{ commandName }} wikg://lib/index
-    {{ commandName }} wikg://lib/index enable --help
-    {{ commandName }} wikg://lib/index disable --help
+    {{ commandName }} wikg://lib/index sync --help
+    {{ commandName }} wikg://lib/index clean --help
     {{ commandName }} help library
 
 Generation readiness:
@@ -82,6 +102,22 @@ LLM readiness:
     {{ commandName }} wikg://local/config/llm put apiKey --secret
     {{ commandName }} wikg://local/config/llm test
 
+Embeddings readiness:
+  `wikg://local/config/embeddings` is required for source or summary embedding artifact builds and for querying an index cache that includes Dense vectors.
+  Hybrid query can still fall back to FTS if a Dense query embedding request fails.
+
+  Minimum configuration:
+    provider
+    model
+    baseURL when using openai-compatible
+    provider credentials when required
+
+  Common path:
+    {{ commandName }} wikg://local/config/embeddings put provider openai
+    {{ commandName }} wikg://local/config/embeddings put model text-embedding-3-small
+    {{ commandName }} wikg://local/config/embeddings put apiKey --secret
+    {{ commandName }} wikg://local/config/embeddings test
+
 WikiSpine readiness:
   Knowledge Graph generation also requires WikiSpine.
   WikiSpine provides entity candidate lookup for Knowledge Graph construction.
@@ -108,13 +144,14 @@ WikiSpine readiness:
     https://raw.githubusercontent.com/oomol-lab/wiki-graph/refs/heads/main/docs/wikispine-runtime.md
 
 How to choose:
-  - Need keyword retrieval on this machine: enable local index.
-  - Need keyword retrieval across a managed folder: enable the library index.
-  - Need to send a more self-contained `.wikg`: consider embedding the index.
-  - Need to intentionally drop query support or index materialization: disable index.
+  - Need lexical retrieval: build chapter FTS artifacts, then sync cache if you do not want lazy sync.
+  - Need semantic retrieval: configure embeddings, build source embedding artifacts, then sync cache.
+  - Need both lexical and semantic retrieval: build both FTS and source embedding artifacts, then sync the archive index cache if it already exists or query reports it outdated; query can create a missing archive cache and chooses Hybrid automatically once the cache is current.
+  - Need searchable retrieval across a managed folder: sync the library index cache.
+  - Need to intentionally drop local cache: clean index cache; artifacts remain in the archive.
   - Need generated Reading Graph or summaries: configure LLM and use local jobs.
   - Need Knowledge Graph: configure LLM and WikiSpine, then use local jobs.
-  - Not sure whether to embed index or generate missing data: keep the archive external/local first, inspect again, and decide later.
+  - Not sure what is missing: run inspect and check index artifact coverage before syncing cache.
 
 Where to go next:
   - Normal workflow:

package/dist/data/help/topics/recipe.jinja

@@ -50,15 +50,18 @@ After inspect:
     {{ commandName }} wikg:///Users/me/book.wikg --query "attention memory"
     {{ commandName }} wikg:///Users/me/book.wikg/chapter/part --query "attention"
 
-  - If inspect reports missing or unavailable index support, inspect the index target before enabling anything.
+  - If inspect reports missing index artifacts or cache, inspect the index targets before syncing cache.
     {{ commandName }} <archive-uri>/index --help
-    {{ commandName }} <archive-uri>/index enable --help
-    Read readiness before deciding whether to keep the index local or embed it:
+    {{ commandName }} <archive-uri>/chapter/<chapter-id>/index/fts --help
+    {{ commandName }} <archive-uri>/chapter/<chapter-id>/index/embedding/source --help
+    {{ commandName }} <archive-uri>/chapter/<chapter-id>/index/embedding/summary --help
+    {{ commandName }} <archive-uri>/index sync --help
+    Read readiness before deciding whether artifacts or cache are missing:
       {{ commandName }} help readiness
 
-  - If library-wide query reports a missing or dirty index, inspect the library index target and rebuild it when appropriate.
+  - If library-wide query reports a missing, stale, or not-ready index cache, inspect the library index target and sync it when appropriate.
     {{ commandName }} wikg://lib/index --help
-    {{ commandName }} wikg://lib/index enable --help
+    {{ commandName }} wikg://lib/index sync --help
     {{ commandName }} help library
 
   - If Reading Graph data is missing, chunk `--query`, related expansion, and pack may be unavailable or incomplete.
@@ -83,8 +86,8 @@ Finding material:
     {{ commandName }} wikg://book.wikg/chapter/part
 
   - Need to search across all archives in a managed library:
-    Use library-wide scopes after enabling the library index.
-      {{ commandName }} wikg://lib/index enable
+    Use library-wide scopes after syncing the library index cache.
+      {{ commandName }} wikg://lib/index sync
       {{ commandName }} wikg://lib/entity --query "attention" --evidence 2
       {{ commandName }} wikg://lib/triple --query "attention memory" --evidence 2
       {{ commandName }} wikg://lib/chunk --query "attention memory" --evidence 2
@@ -168,7 +171,7 @@ When to read deeper:
     {{ commandName }} help format
   - LLM, concurrency, and WikiSpine provider setup:
     {{ commandName }} help config
-  - FTS index, LLM, WikiSpine, and generated-data prerequisites:
+  - Search index, embeddings, LLM, WikiSpine, and generated-data prerequisites:
     {{ commandName }} help readiness
   - Library registry, membership, path binding, and aggregate index:
     {{ commandName }} help library

package/dist/data/help/topics/uri.jinja

@@ -48,7 +48,7 @@ Short and located URIs:
 
 Command routing:
   - `{{ commandName }} <scope-uri>` enumerates objects.
-  - `{{ commandName }} <scope-uri> --query <query>` applies keyword retrieval within that scope.
+  - `{{ commandName }} <scope-uri> --query <query>` applies search index retrieval within that scope.
   - `{{ commandName }} <object-uri>` reads one object.
   - `{{ commandName }} <uri> --help` explains that URI target and its supported predicates.
   - `{{ commandName }} <uri> <predicate> --help` explains a URI-bound predicate.
@@ -69,12 +69,16 @@ Common command shapes:
   {{ commandName }} wikg://lib/entity --query <query>
   {{ commandName }} wikg://lib/arc/<archive-id>/entity --query <query>
   {{ commandName }} wikg://local/job
-  {{ commandName }} wikg://local/job add --input <archive-uri|chapter-uri> --task <reading-graph|reading-summary|knowledge-graph> --accept-cost
+  {{ commandName }} wikg://local/job add --input <archive-uri|chapter-uri> --task <index-fts|index-embedding-source|index-embedding-summary|reading-graph|reading-summary|knowledge-graph> [--accept-cost]
   {{ commandName }} wikg://local/job/<job-id>
 
 Scope URIs:
-  Scope URIs enumerate objects by default. With `--query <query>`, they apply keyword retrieval within that scope.
+  Scope URIs enumerate objects by default. With `--query <query>`, they apply search index retrieval within that scope.
   A chapter URI used as a scope covers that chapter's full subtree by default.
+  Query is strict by default: every chapter in scope must have a current FTS artifact or source embedding artifact.
+  Add `--skip-unindexed` only when you intentionally want to search indexed chapters and ignore uncovered chapters.
+  Broad query is grouped by result type before ranking within each group. Use `/chunk`, `/entity`, or `/triple` lenses when you want one object family instead of the mixed broad result. Query an archive or chapter scope to find source or summary text, then read the returned `/source` or `/summary` object.
+  Search uses FTS-only, Dense-only, or Hybrid automatically from the current artifacts and index cache; use `{{ commandName }} help readiness` to inspect what must exist.
   Add `--depth <N>` to limit chapter subtree expansion:
     - no `--depth`: full subtree
     - `--depth 0`: only the current chapter, or root-level chapters for `<archive-uri>/chapter`
@@ -156,6 +160,7 @@ Literal source passages:
   Use `--query` on the narrowest useful scope when the task is to find source wording or passages mentioning a literal term:
     {{ commandName }} <archive-uri> --query "term"
     {{ commandName }} <archive-uri>/chapter/<path> --query "term"
+    {{ commandName }} <archive-uri> --query "term" --skip-unindexed
     {{ commandName }} <archive-uri>/chapter/<path>/source#20..30
 
 Entity-grounded evidence:
@@ -178,6 +183,8 @@ Recovery hints:
     {{ commandName }} <archive-uri>/chapter/tree
     {{ commandName }} help readiness
   - Missing generated objects: start the relevant generation job for the chapter.
+    {{ commandName }} wikg://local/job add --input <archive-uri|chapter-uri> --task index-fts
+    {{ commandName }} wikg://local/job add --input <archive-uri|chapter-uri> --task index-embedding-source --accept-cost
     {{ commandName }} wikg://local/job add --input <archive-uri|chapter-uri> --task reading-graph --accept-cost
     {{ commandName }} wikg://local/job add --input <archive-uri|chapter-uri> --task reading-summary --accept-cost
     {{ commandName }} help readiness