@zosmaai/pi-llm-wiki 0.8.1 → 0.9.0

package/CHANGELOG.md

@@ -6,6 +6,41 @@
 - **Personal wiki created at doubled path `~/.llm-wiki/.llm-wiki/…`**: `getPersonalWikiRoot()` returned the dot-dir itself (`~/.llm-wiki`) while `getVaultPaths()` then appended another `.llm-wiki/` segment, so the personal vault was written to `~/.llm-wiki/.llm-wiki/wiki/…`. Fixed by aligning `getPersonalWikiRoot()` with the same "root = parent of `.llm-wiki/`" contract used by project vaults. `WIKI_HOME` continues to override the parent.
 
 ### Added
+- **Model selection surface for background tasks** (Issue #69, part of epic #63): the wiki background lane (ingest synthesis, etc.) now has a user-facing surface to choose its model, defaulting to the **session model** with zero config.
+  - **`/wiki-model` slash command**: run with no argument for an interactive picker (lists `modelRegistry.getAvailable()`); `/wiki-model provider/id` to set directly (scriptable, no UI needed); `/wiki-model session` (or `clear`/`default`/`reset`) to revert to the session model. The choice is **persisted** to project settings (`.pi/settings.json` under `llm-wiki.taskModel`, preserving other keys) and applied immediately, with a status-bar label of the active model.
+  - **Per-call `model` override** on heavy tools (`wiki_ingest`): an optional `'provider/id'` param that overrides the configured `taskModel` for that one call. Precedence is **override > configured taskModel > session model**; each layer is applied only when the model is in the registry, and a missing/unknown layer warns (when UI is available) and falls through — so a bad ref degrades gracefully instead of failing.
+  - **`parseModelRef` / `persistTaskModel`** helpers (`extensions/llm-wiki/lib/task-config.ts`): parse a `provider/id` ref (split on the first slash so slash-bearing model ids survive) and read-modify-write the namespaced project settings (creating `.pi/` as needed, clearing the key on `undefined`). `Runtime.resolveModel(ctx, override?)` gained the override layer; `formatActiveModelLabel` renders the status line.
+  - **15 tests** (`test/model-selection.test.ts`): ref parsing (incl. slash-bearing ids and rejects), persistence round-trip (write/read/clear/preserve-other-keys/create-dir), `resolveModel` precedence (override > config > session, with graceful fallback + warning), and the active-model label. Written red-first.
+- **Formal two-stage recall (links-first, gated by vault size)** (Issue #68, part of epic #63): recall now scales as the vault grows by switching from inline content previews to a **ranked list of links** once the vault crosses a configurable size threshold, with the agent expanding chosen links on demand via `read` (memex-style two-stage retrieval).
+  - **Stage 1 (links-first)**: above the threshold, both `wiki_recall` and the `before_agent_start` auto-injection return ranked links carrying `id`, `title`, `type`, `score`, and a single short snippet — **no full previews** — so a large vault never floods the system prompt. **Stage 2 (expand)**: the agent calls `read`/`wiki_read` on the links it actually needs. The two-step contract is documented in `SKILL.md`.
+  - **No regression for small vaults**: at or below the threshold the existing preview-inline rendering is byte-for-byte unchanged (`formatRecallContext` default path), so small vaults keep today's behavior.
+  - **Page-count gate (deliberate choice over token-budget)**: the threshold is the registered page count from `meta/registry.json` — an O(1) read with **zero page-body I/O**, so the gate itself never reads page contents as the vault grows (a token-budget gate would have to touch every page, defeating the cheap-recall goal). New `recallLinksThreshold` setting (namespaced `llm-wiki`, default **50**, clamped to a non-negative integer). Set `0` to force links-first for any non-empty vault, or a large value to always keep previews inline. The auto-injection counts only the project vault (matching its search scope); the `wiki_recall` tool counts project + personal.
+  - **7 tests** (`test/recall.test.ts`): page-count derivation, threshold gating in both directions (incl. the `0`-forces-links-first edge), small-vault preview regression (byte-for-byte equal to `linksOnly:false`), large-vault links-only output shape (id/title/type/score/snippet, no full preview), single-line snippet truncation, threshold config parsing/clamping, and end-to-end below-vs-above gating.
+- **Hybrid lexical + semantic recall ranking, LLM-free hot path** (Issue #67, part of epic #63): recall now blends the existing weighted lexical score with **semantic cosine similarity** against the precomputed `meta/embeddings.json` sidecar (#66), so paraphrased queries surface pages that pure keyword matching misses.
+  - **No re-embedding per query**: page vectors are computed at write time (#66). The only per-query embedding work is a **single, cached** lookup of the (short) query string via the configured embedder (`searchWikiHybrid` in `extensions/llm-wiki/lib/recall.ts`); repeated recalls of the same query in a session reuse the cached vector. Ranking itself (`searchWiki`) stays **synchronous and offline** — pure vector math, no network.
+  - **Score fusion**: the lexical score keeps its original absolute scale and the semantic signal is added as a bounded, weighted, non-negative boost (`fuseScores` = `lexical + semanticWeight × SEMANTIC_SCALE × max(cosine, 0)`). This preserves `minScore` semantics for auto-injection noise control — a semantic-only match must be **strongly** relevant (cosine ≳ 0.84 at the default weight) to clear the auto-inject threshold. New `semanticWeight` setting (namespaced `llm-wiki`, default `0.5`, clamped to `[0,1]`).
+  - **Pure-lexical fallback preserved**: with no embeddings sidecar (or no embedder configured) the query embedding is **skipped entirely** (zero network) and recall is byte-for-byte the prior lexical behavior. Embedding/network failures degrade gracefully to lexical. The `wiki_recall` tool and the `before_agent_start` auto-injection both use the hybrid path.
+  - **8 tests** (`test/recall.test.ts`): score-fusion math, paraphrase recall (semantic-only page surfaced), ranking reorder vs pure lexical, no-sidecar regression (identical to lexical), empty/missing-sidecar safety, `minScore` still filtering weak cosine, and the hybrid wrapper skipping vs performing+caching the single query embedding. Embedding is mocked — no network in tests.
+- **Background semantic embeddings computed at write time** (Issue #66, part of epic #63): every wiki page gets a normalized embedding vector computed in the **background**, so future semantic retrieval (#67) can rank pages **without any embedding/LLM call in the query hot path**.
+  - `extensions/llm-wiki/lib/embeddings.ts`: stores vectors in a `meta/embeddings.json` sidecar keyed by page id, each with a **content hash + model** for staleness detection. `embedPages()` embeds a set of pages (skipping fresh ones unless `force`); `reindexEmbeddings()` backfills an entire vault and prunes entries for deleted pages. Vectors are L2-normalized for cosine; `cosineSimilarity()` is exported for #67.
+  - **Write-time triggers (all background, never block the agent)**: ingest commit embeds the source + entity/concept pages it wrote; `wiki_ensure_page` embeds the new page; manual wiki edits are re-embedded after the end-of-turn metadata rebuild. All single-flight per label via the #64 runtime.
+  - **Configurable, OpenAI-compatible provider** (mirrors memex): new `embeddingProvider` / `embeddingModel` / `embeddingBaseUrl` / `embeddingApiKey` / `embeddingApiKeyEnv` fields in the namespaced `llm-wiki` settings (`extensions/llm-wiki/lib/task-config.ts`). The embedding API key has its **own** auth path, independent of the chat-model resolution.
+  - **New `wiki_reindex_embeddings` tool**: backfill an existing vault or refresh stale pages (`force` to re-embed everything).
+  - **Fully optional, no-op by default**: with no `embeddingProvider` configured, embeddings are disabled silently and existing lexical search is completely unaffected. Opt-in is explicit — an ambient `OPENAI_API_KEY` alone does **not** enable it.
+  - **23 tests** (`test/embeddings.test.ts`, additions to `test/runtime.test.ts`): write-time embedding + normalized storage, content-hash and model staleness, `force`, backfill + prune, the no-provider no-op (and no auto-enable), vector math, and config parsing. Embedding is mocked — no network in tests.
+- **Background ingest synthesis** (Issue #65, part of epic #63): `wiki_ingest` now synthesizes captured sources in the **background by default** — the main agent is no longer blocked while source pages and entity/concept pages are written.
+  - New `background` parameter (default `true`). When a task model resolves, each source is dispatched to a background sub-agent via the #64 runtime and the tool returns immediately with a non-blocking notification; the main agent is told **not** to synthesize those sources itself. The sub-agent makes one structured `commit_synthesis` call.
+  - **Graceful fallback**: when no model/API key is available (or `background=false`), `wiki_ingest` returns the extracted batch exactly as before for the main agent to synthesize — fully backward compatible.
+  - `extensions/llm-wiki/lib/ingest-worker.ts`: `commitSynthesis()` deterministically rewrites the source page (`status: skeleton → ingested`), creates missing entity/concept pages (existing pages are linked, never overwritten), records an `ingest` event, and rebuilds metadata. `runIngestSynthesis()` drives the synthesis sub-agent. `buildIngestedSourcePage()` renders the filled page.
+  - **10 tests** (`test/ingest-worker.test.ts`, `test/ingest-tool.test.ts`): deterministic page/entity/concept writing, link-don't-overwrite, event logging, empty-slug safety, and tool-level background dispatch + all three fallback paths (no model, `background=false`, no runtime).
+- **Background-task lane infrastructure** (Issue #64, part of epic #63): foundational runtime so wiki tasks can run off the main agent thread without blocking the user.
+  - `Runtime` (`extensions/llm-wiki/lib/runtime.ts`): `launchTask()` fires detached, single-flight-per-label background work with isolated error handling; `resolveModel()` picks the configured `taskModel` → session-model fallback → API-key resolution, returning a graceful `{ ok: false }` so callers keep the synchronous path when no model/key is available; `awaitAll()` drains in-flight work.
+  - `registerBackgroundRuntime()`: wires the runtime into the extension lifecycle, draining in-flight tasks on `session_before_compact` and `session_shutdown` so background work is never lost.
+  - `loadTaskConfig()` (`extensions/llm-wiki/lib/task-config.ts`): reads an optional `taskModel` from pi's namespaced `llm-wiki` settings (global + project). Zero-config by default.
+  - `runSubAgent()` (`extensions/llm-wiki/lib/subagent.ts`): thin, generic `agentLoop` wrapper for focused background sub-agents.
+  - **15 unit tests** (`test/runtime.test.ts`) covering config precedence, model resolution + fallbacks, single-flight, concurrency, error isolation, and drain.
+  - No user-facing behavior change yet; concrete workers land in #65 (background ingest) and #66 (background embeddings).
+- **Test isolation fix** (`test/recall.test.ts`): the personal-vault recall suite now sandboxes `WIKI_HOME` to a temp dir per test, so it no longer reads (or writes) the developer's real `~/.llm-wiki`. Previously these tests passed only by scheduling luck and could pollute the real home vault.
 - **`migrateDoubledPersonalVault()`** helper (`extensions/llm-wiki/lib/utils.ts`): Idempotent, in-place flatten of any vault that was already written to the broken doubled layout. Moves entries from `<root>/.llm-wiki/.llm-wiki/*` up to `<root>/.llm-wiki/*`, preserves outer entries on collision, removes the inner dir only when fully drained. Returns `null` when the layout is already correct, so it is safe to call on every session start.
 - **Auto-migration on `session_start`**: The extension now runs `migrateDoubledPersonalVault()` on the personal wiki at every session start. Existing broken vaults are flattened the next time the user opens or reloads pi — no manual step required. A one-line status message is shown when a flatten actually happens; otherwise the check is silent.
 - **`scripts/migrate-llm-wiki.js --fix-doubled`** flag: Manual recovery for arbitrary roots (`--fix-doubled ~/`, `--fix-doubled /some/project`). Supports `--dry-run` and `--force`.

package/README.md

@@ -399,9 +399,39 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, test patterns, and
 
 ## Contributors
 
-<a href="https://github.com/zosmaai/pi-llm-wiki/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=zosmaai/pi-llm-wiki" alt="Contributors" />
-</a>
+Thanks to everyone who has contributed! This list is regenerated automatically by [`.github/workflows/contributors.yml`](.github/workflows/contributors.yml) — see [#60](https://github.com/zosmaai/pi-llm-wiki/issues/60) for the rationale.
+
+<!-- readme: contributors -start -->
+<table>
+	<tbody>
+		<tr>
+            <td align="center">
+                <a href="https://github.com/arjun-zosma">
+                    <img src="https://avatars.githubusercontent.com/u/25246034?v=4" width="64;" alt="arjun-zosma"/>
+                    <br />
+                    <sub><b>Arjun Nayak</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/jfraser">
+                    <img src="https://avatars.githubusercontent.com/u/165964?v=4" width="64;" alt="jfraser"/>
+                    <br />
+                    <sub><b>James Fraser</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/Shanvit7">
+                    <img src="https://avatars.githubusercontent.com/u/64424817?v=4" width="64;" alt="Shanvit7"/>
+                    <br />
+                    <sub><b>Shanvit S Shetty</b></sub>
+                </a>
+            </td>
+		</tr>
+	<tbody>
+</table>
+<!-- readme: contributors -end -->
+
+<sub>Full history: [contributors graph](https://github.com/zosmaai/pi-llm-wiki/graphs/contributors).</sub>
 
 ---
 

package/extensions/llm-wiki/index.ts

@@ -2,13 +2,25 @@ import { existsSync, readFileSync, writeFileSync } from "node:fs";
 import { basename, join } from "node:path";
 import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
 import { installGuardrails } from "./lib/guardrails.js";
+import {
+  MODEL_STATUS_KEY,
+  formatActiveModelLabel,
+  registerWikiModelCommand,
+} from "./lib/model-command.js";
 import {
   createReminderState,
   registerObservationReminder,
   registerWikiObserve,
 } from "./lib/observation.js";
-import { formatRecallContext, registerWikiRecall, searchWikiLayered } from "./lib/recall.js";
+import {
+  formatRecallContext,
+  registerWikiRecall,
+  searchWikiHybrid,
+  shouldUseLinksFirst,
+  vaultPageCount,
+} from "./lib/recall.js";
 import { registerWikiRetro } from "./lib/retro.js";
+import { registerBackgroundRuntime } from "./lib/runtime.js";
 import {
   registerWikiBootstrap,
   registerWikiCaptureSource,
@@ -17,6 +29,7 @@ import {
   registerWikiLint,
   registerWikiLogEvent,
   registerWikiRebuildMeta,
+  registerWikiReindexEmbeddings,
   registerWikiSearch,
   registerWikiStatus,
   registerWikiWatch,
@@ -49,23 +62,32 @@ import {
  */
 
 export default function (pi: ExtensionAPI) {
+  // Background-task lane (issues #64, #65): shared runtime for off-thread LLM
+  // work. Created first so tools (e.g. wiki_ingest) can dispatch to it.
+  const runtime = registerBackgroundRuntime(pi);
+
   registerWikiBootstrap(pi);
-  registerWikiCaptureSource(pi);
-  registerWikiIngest(pi);
-  registerWikiEnsurePage(pi);
+  registerWikiCaptureSource(pi, runtime);
+  registerWikiIngest(pi, runtime);
+  registerWikiEnsurePage(pi, runtime);
   registerWikiSearch(pi);
   registerWikiLint(pi);
   registerWikiStatus(pi);
   registerWikiRebuildMeta(pi);
+  registerWikiReindexEmbeddings(pi, runtime);
   registerWikiLogEvent(pi);
   registerWikiWatch(pi);
-  registerWikiRecall(pi);
-  registerWikiRetro(pi);
+  registerWikiRecall(pi, runtime);
+  registerWikiRetro(pi, runtime);
+  // Model selection surface (issue #69): /wiki-model command to view/set the
+  // background task model. The taskModel config field + resolveModel already
+  // exist; this exposes them to the user (default stays the session model).
+  registerWikiModelCommand(pi, runtime);
   const reminderState = createReminderState();
-  registerWikiObserve(pi, reminderState);
+  registerWikiObserve(pi, runtime, reminderState);
   registerObservationReminder(pi, reminderState);
 
-  installGuardrails(pi);
+  installGuardrails(pi, runtime);
 
   // Track if wiki was just auto-created and needs topic inference
   let needsTopicInference = false;
@@ -123,6 +145,12 @@ export default function (pi: ExtensionAPI) {
     }
 
     ctx.ui.setStatus("llm-wiki", "🧠 LLM Wiki (13 tools, observe + recall active)");
+
+    // Surface the active background task model (issue #69). Defaults to the
+    // session model when no taskModel is configured.
+    runtime.ensureConfig(process.cwd());
+    const modelLabel = formatActiveModelLabel(runtime.config, (ctx.model as { id?: string })?.id);
+    ctx.ui.setStatus(MODEL_STATUS_KEY, `🧠 wiki model: ${modelLabel}`);
   });
 
   // ─── Layered recall + topic inference hook ──────────
@@ -182,9 +210,20 @@ Then call wiki_bootstrap with the inferred topic and mode to finalize the setup.
       // or multiple body matches. This eliminates accidental body-only
       // substring matches (e.g. a Tally page matching on common words).
       // includePersonal=false: personal vault is excluded from auto-injection.
-      const results = searchWikiLayered(paths, prompt, 3, 5, false);
+      // Hybrid: blends semantic cosine when embeddings exist (single cached
+      // query embedding); degrades to pure lexical otherwise. minScore=5 still
+      // gates noise — a semantic-only match must be strongly relevant to pass.
+      runtime.ensureConfig(process.cwd());
+      const results = await searchWikiHybrid(paths, prompt, 3, 5, false, {
+        config: runtime.config,
+      });
       if (results.length > 0) {
-        const recallContext = formatRecallContext(results);
+        // Two-stage gate (issue #68): above the vault-size threshold, inject
+        // ranked LINKS only (the agent expands them on demand via `read`) so a
+        // large vault never floods the system prompt with inline previews.
+        // includePersonal=false here mirrors the auto-injection search scope.
+        const linksOnly = shouldUseLinksFirst(vaultPageCount(paths, false), runtime.config);
+        const recallContext = formatRecallContext(results, { linksOnly });
         if (recallContext) {
           injectedContext += `\n\n${recallContext}`;
         }

package/extensions/llm-wiki/lib/embeddings.ts

@@ -0,0 +1,420 @@
+import { createHash } from "node:crypto";
+import { existsSync, readFileSync } from "node:fs";
+import { request as httpRequest } from "node:http";
+import { request as httpsRequest } from "node:https";
+import { join } from "node:path";
+import type { Registry } from "./metadata.js";
+import type { LaunchCtx, Runtime } from "./runtime.js";
+import type { TaskConfig } from "./task-config.js";
+import { type VaultPaths, parseFrontmatter, readJson, writeJson } from "./utils.js";
+
+/**
+ * Background semantic embeddings, computed at write time (issue #66, epic #63).
+ *
+ * Every wiki page gets a normalized embedding vector stored in a sidecar
+ * (`meta/embeddings.json`), keyed by page id with a content hash for staleness
+ * detection. Embeddings are computed in the background via the #64 runtime so
+ * the main agent is never blocked, and so that semantic retrieval (#67) can
+ * rank pages WITHOUT any embedding/LLM call in the query hot path.
+ *
+ * Design principles:
+ *   - Fully optional: with no embedding provider configured, `resolveEmbedder`
+ *     returns undefined and every entry point no-ops silently. Existing lexical
+ *     search (lib/recall.ts) is untouched. This is the default.
+ *   - Embeddings have their OWN auth path (an embedding API key + an
+ *     OpenAI-compatible endpoint), independent of the chat-model resolution in
+ *     Runtime.resolveModel.
+ *   - The compute/store functions take an injected `Embedder`, so unit tests
+ *     mock embedding with NO network.
+ */
+
+// ── constants ─────────────────────────────────────────────
+export const DEFAULT_EMBEDDING_MODEL = "text-embedding-3-small";
+export const DEFAULT_EMBEDDING_BASE_URL = "https://api.openai.com";
+/** Cap on body chars fed into a single embedding (keep prompts bounded). */
+const MAX_BODY_CHARS = 8_000;
+const STORE_VERSION = "1.0";
+
+// ── types ─────────────────────────────────────────────────
+
+/** Embeds a batch of texts into raw (un-normalized) vectors. */
+export type EmbedFn = (texts: string[]) => Promise<number[][]>;
+
+/** A resolved embedding backend: a model label + the embed function. */
+export interface Embedder {
+  model: string;
+  embed: EmbedFn;
+}
+
+export interface EmbeddingEntry {
+  /** sha256 of the exact text that was embedded — drives staleness detection. */
+  hash: string;
+  /** Embedding model label that produced this vector. */
+  model: string;
+  /** Vector dimensionality. */
+  dim: number;
+  /** Normalized (unit-length) vector for cosine similarity. */
+  vector: number[];
+  /** ISO timestamp of when this entry was written. */
+  updated: string;
+}
+
+export interface EmbeddingStore {
+  version: string;
+  /** Keyed by folder-qualified page id (e.g. "concepts/rag"). */
+  entries: Record<string, EmbeddingEntry>;
+}
+
+export interface EmbedStats {
+  embedded: number;
+  skipped: number;
+  total: number;
+}
+
+export interface ReindexStats extends EmbedStats {
+  pruned: number;
+}
+
+// ── vector math (shared with retrieval #67) ───────────────
+
+/** Normalize a vector to unit length so dot product == cosine similarity. */
+export function normalizeVector(vec: number[]): number[] {
+  const sanitized = vec.map((v) => (Number.isFinite(v) ? v : 0));
+  const magnitude = Math.sqrt(sanitized.reduce((sum, v) => sum + v * v, 0));
+  if (magnitude < 1e-10) return new Array(sanitized.length).fill(0);
+  return sanitized.map((v) => v / magnitude);
+}
+
+/** Cosine similarity of two vectors. Robust to un-normalized input. */
+export function cosineSimilarity(a: number[], b: number[]): number {
+  if (a.length === 0 || a.length !== b.length) return 0;
+  let dot = 0;
+  let normA = 0;
+  let normB = 0;
+  for (let i = 0; i < a.length; i++) {
+    dot += a[i] * b[i];
+    normA += a[i] * a[i];
+    normB += b[i] * b[i];
+  }
+  const denom = Math.sqrt(normA) * Math.sqrt(normB);
+  if (denom === 0) return 0;
+  return dot / denom;
+}
+
+/** Stable content hash of the text that was (or will be) embedded. */
+export function contentHash(text: string): string {
+  return createHash("sha256").update(text).digest("hex");
+}
+
+// ── embedding text ────────────────────────────────────────
+
+/**
+ * Build the text to embed for a page: its title + salient frontmatter +
+ * (bounded) body. Mirrors memex's `buildEmbeddingText` — front-loading the
+ * high-signal metadata then appending the body content.
+ */
+export function buildEmbeddingText(
+  id: string,
+  frontmatter: Record<string, unknown>,
+  body: string,
+): string {
+  const parts: string[] = [];
+
+  const title = frontmatter.title;
+  parts.push(`title: ${typeof title === "string" && title.trim() ? title.trim() : id}`);
+  if (typeof frontmatter.type === "string" && frontmatter.type.trim()) {
+    parts.push(`type: ${frontmatter.type.trim()}`);
+  }
+
+  for (const key of [
+    "aliases",
+    "recall_triggers",
+    "summary",
+    "description",
+    "tags",
+    "category",
+    "domain",
+  ]) {
+    const val = frontmatter[key];
+    if (typeof val === "string" && val.trim()) {
+      parts.push(`${key}: ${val.trim()}`);
+    } else if (Array.isArray(val) && val.length > 0) {
+      parts.push(`${key}: ${val.map((v) => String(v)).join(", ")}`);
+    }
+  }
+
+  const head = parts.join("\n");
+  const trimmedBody = body.trim().slice(0, MAX_BODY_CHARS);
+  return trimmedBody ? `${head}\n\n${trimmedBody}` : head;
+}
+
+// ── sidecar store I/O ─────────────────────────────────────
+
+export function embeddingStorePath(paths: VaultPaths): string {
+  return join(paths.meta, "embeddings.json");
+}
+
+export function readEmbeddingStore(paths: VaultPaths): EmbeddingStore {
+  const store = readJson<EmbeddingStore>(embeddingStorePath(paths), {
+    version: STORE_VERSION,
+    entries: {},
+  });
+  if (!store.entries || typeof store.entries !== "object") {
+    return { version: STORE_VERSION, entries: {} };
+  }
+  return store;
+}
+
+export function writeEmbeddingStore(paths: VaultPaths, store: EmbeddingStore): void {
+  writeJson(embeddingStorePath(paths), store);
+}
+
+/** True if the page id has no fresh embedding for the given hash + model. */
+export function isStale(store: EmbeddingStore, id: string, hash: string, model: string): boolean {
+  const entry = store.entries[id];
+  if (!entry) return true;
+  return entry.hash !== hash || entry.model !== model;
+}
+
+// ── compute ───────────────────────────────────────────────
+
+interface PageText {
+  id: string;
+  text: string;
+  hash: string;
+}
+
+/** Read a page file (if present) and derive its embedding text + hash. */
+function readPageText(paths: VaultPaths, id: string): PageText | undefined {
+  const pagePath = join(paths.wiki, `${id}.md`);
+  if (!existsSync(pagePath)) return undefined;
+  const raw = readFileSync(pagePath, "utf-8");
+  const { frontmatter, body } = parseFrontmatter(raw);
+  const text = buildEmbeddingText(id, frontmatter, body);
+  return { id, text, hash: contentHash(text) };
+}
+
+/**
+ * Embed the given page ids, writing fresh vectors into the sidecar store.
+ * Stale-aware: pages whose hash + model already match are skipped (unless
+ * `force`). Pure async — pass a mock `Embedder` to test without a network.
+ */
+export async function embedPages(
+  paths: VaultPaths,
+  ids: string[],
+  embedder: Embedder,
+  opts: { force?: boolean } = {},
+): Promise<EmbedStats> {
+  const store = readEmbeddingStore(paths);
+  const targets: PageText[] = [];
+  let skipped = 0;
+
+  const seen = new Set<string>();
+  for (const id of ids) {
+    if (seen.has(id)) continue;
+    seen.add(id);
+    const page = readPageText(paths, id);
+    if (!page) continue;
+    if (!opts.force && !isStale(store, id, page.hash, embedder.model)) {
+      skipped++;
+      continue;
+    }
+    targets.push(page);
+  }
+
+  if (targets.length > 0) {
+    const vectors = await embedder.embed(targets.map((t) => t.text));
+    const now = new Date().toISOString();
+    for (let i = 0; i < targets.length; i++) {
+      const vec = normalizeVector(vectors[i] ?? []);
+      store.entries[targets[i].id] = {
+        hash: targets[i].hash,
+        model: embedder.model,
+        dim: vec.length,
+        vector: vec,
+        updated: now,
+      };
+    }
+    writeEmbeddingStore(paths, store);
+  }
+
+  return { embedded: targets.length, skipped, total: seen.size };
+}
+
+/**
+ * Embed every registered wiki page that has a backing file, skipping fresh
+ * ones (unless `force`), and prune sidecar entries for deleted pages. This is
+ * the backfill / re-embed-stale path used by the reindex command.
+ */
+export async function reindexEmbeddings(
+  paths: VaultPaths,
+  embedder: Embedder,
+  opts: { force?: boolean } = {},
+): Promise<ReindexStats> {
+  const registry = readJson<Registry>(join(paths.meta, "registry.json"), {
+    version: "1.0",
+    last_updated: "",
+    pages: {},
+  });
+
+  const ids = Object.keys(registry.pages).filter((id) => existsSync(join(paths.wiki, `${id}.md`)));
+
+  const stats = await embedPages(paths, ids, embedder, opts);
+
+  // Prune entries whose page file no longer exists.
+  const store = readEmbeddingStore(paths);
+  let pruned = 0;
+  for (const id of Object.keys(store.entries)) {
+    if (!existsSync(join(paths.wiki, `${id}.md`))) {
+      delete store.entries[id];
+      pruned++;
+    }
+  }
+  if (pruned > 0) writeEmbeddingStore(paths, store);
+
+  return { ...stats, pruned };
+}
+
+// ── provider resolution (OpenAI-compatible) ───────────────
+
+/** Compose the /v1/embeddings request path from an optional base path. */
+function embeddingsRequestPath(basePath: string): string {
+  if (!basePath || basePath === "/") return "/v1/embeddings";
+  if (basePath.endsWith("/v1")) return `${basePath}/embeddings`;
+  return `${basePath}/v1/embeddings`;
+}
+
+interface EmbeddingApiResponse {
+  data?: Array<{ index: number; embedding: number[] }>;
+  error?: { message?: string };
+}
+
+/**
+ * Create an `EmbedFn` backed by an OpenAI-compatible `/v1/embeddings`
+ * endpoint. Uses node's http/https directly (no SDK) so it works against
+ * OpenAI, Azure (with an api-key header), or any compatible gateway.
+ */
+export function createOpenAIEmbedFn(cfg: {
+  apiKey: string;
+  baseUrl: string;
+  model: string;
+  headers?: Record<string, string>;
+}): EmbedFn {
+  const parsed = new URL(cfg.baseUrl);
+  const basePath = parsed.pathname.replace(/\/$/, "");
+  const requestPath = embeddingsRequestPath(basePath);
+  const useHttp = parsed.protocol === "http:";
+  const port = parsed.port ? Number(parsed.port) : undefined;
+
+  return (texts) =>
+    new Promise<number[][]>((resolve, reject) => {
+      if (texts.length === 0) {
+        resolve([]);
+        return;
+      }
+      const body = JSON.stringify({ model: cfg.model, input: texts });
+      const reqFn = useHttp ? httpRequest : httpsRequest;
+      const req = reqFn(
+        {
+          hostname: parsed.hostname,
+          ...(port ? { port } : {}),
+          path: requestPath,
+          method: "POST",
+          headers: {
+            "Content-Type": "application/json",
+            Authorization: `Bearer ${cfg.apiKey}`,
+            ...(cfg.headers ?? {}),
+            "Content-Length": Buffer.byteLength(body),
+          },
+        },
+        (res) => {
+          let data = "";
+          res.on("data", (chunk) => {
+            data += chunk.toString();
+          });
+          res.on("end", () => {
+            try {
+              const parsedBody = JSON.parse(data) as EmbeddingApiResponse;
+              if (parsedBody.error) {
+                reject(new Error(`embedding API error: ${parsedBody.error.message ?? "unknown"}`));
+                return;
+              }
+              const rows = parsedBody.data ?? [];
+              const sorted = [...rows].sort((a, b) => a.index - b.index);
+              resolve(sorted.map((d) => d.embedding));
+            } catch (err) {
+              reject(new Error(`failed to parse embedding response: ${(err as Error).message}`));
+            }
+          });
+        },
+      );
+      req.on("error", reject);
+      req.write(body);
+      req.end();
+    });
+}
+
+/**
+ * Resolve an `Embedder` from config, or `undefined` when embeddings are not
+ * configured (the default — fully optional, silent no-op).
+ *
+ * Opt-in is explicit: `embeddingProvider` MUST be set (we do not auto-enable
+ * just because an ambient OPENAI_API_KEY happens to exist). Only the
+ * OpenAI-compatible provider is supported; anything else no-ops.
+ */
+export function resolveEmbedder(config: TaskConfig): Embedder | undefined {
+  const provider = config.embeddingProvider?.trim().toLowerCase();
+  if (!provider) return undefined;
+  if (provider !== "openai" && provider !== "openai-compatible") return undefined;
+
+  const keyEnv = config.embeddingApiKeyEnv?.trim() || "OPENAI_API_KEY";
+  const apiKey = config.embeddingApiKey?.trim() || process.env[keyEnv]?.trim();
+  if (!apiKey) return undefined;
+
+  const model = config.embeddingModel?.trim() || DEFAULT_EMBEDDING_MODEL;
+  const baseUrl =
+    config.embeddingBaseUrl?.trim() ||
+    process.env.OPENAI_BASE_URL?.trim() ||
+    DEFAULT_EMBEDDING_BASE_URL;
+
+  return { model, embed: createOpenAIEmbedFn({ apiKey, baseUrl, model }) };
+}
+
+// ── background launch helpers (used by tools/guardrails) ──
+
+/**
+ * Launch a background task that embeds a specific set of pages, if (and only
+ * if) an embedder is configured. No-op (returns false) otherwise. Single-flight
+ * per label, error-isolated, drained at compaction/shutdown — all via #64.
+ */
+export function launchEmbedPages(
+  runtime: Runtime,
+  ctx: LaunchCtx,
+  paths: VaultPaths,
+  ids: string[],
+  label: string,
+): boolean {
+  if (ids.length === 0) return false;
+  runtime.ensureConfig(paths.root);
+  const embedder = resolveEmbedder(runtime.config);
+  if (!embedder) return false;
+  runtime.launchTask(ctx, label, async () => {
+    await embedPages(paths, ids, embedder);
+  });
+  return true;
+}
+
+/**
+ * Launch a background reindex (embed all stale registered pages + prune
+ * deleted), if an embedder is configured. No-op otherwise. Single-flight per
+ * vault so repeated writes within a turn collapse into one pass.
+ */
+export function launchReindex(runtime: Runtime, ctx: LaunchCtx, paths: VaultPaths): boolean {
+  runtime.ensureConfig(paths.root);
+  const embedder = resolveEmbedder(runtime.config);
+  if (!embedder) return false;
+  runtime.launchTask(ctx, `embed:reindex:${paths.root}`, async () => {
+    await reindexEmbeddings(paths, embedder);
+  });
+  return true;
+}

package/extensions/llm-wiki/lib/guardrails.ts

@@ -1,6 +1,8 @@
 import { isToolCallEventType } from "@mariozechner/pi-coding-agent";
 import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
+import { scheduleReindex } from "./indexing.js";
 import { rebuildMetadataLight } from "./metadata.js";
+import type { Runtime } from "./runtime.js";
 import { isProtectedPath, resolveVaultPaths } from "./utils.js";
 
 /**
@@ -10,7 +12,7 @@ import { isProtectedPath, resolveVaultPaths } from "./utils.js";
 let pendingRebuild = false;
 
 /** Install guardrails on the extension API. */
-export function installGuardrails(pi: ExtensionAPI): void {
+export function installGuardrails(pi: ExtensionAPI, runtime?: Runtime): void {
   // Block direct edits to raw/ and meta/
   pi.on("tool_call", async (event) => {
     if (isToolCallEventType("write", event)) {
@@ -44,13 +46,22 @@ export function installGuardrails(pi: ExtensionAPI): void {
     }
   });
 
-  // Rebuild metadata at end of turn if wiki was modified
-  pi.on("turn_end", async (_event) => {
+  // Rebuild metadata at end of turn if wiki was modified, then refresh
+  // semantic embeddings in the background (#66) so manual page edits get
+  // re-embedded. Both are best-effort no-ops when nothing is configured.
+  pi.on("turn_end", async (_event, ctx) => {
     if (pendingRebuild) {
       pendingRebuild = false;
       try {
         const paths = resolveVaultPaths(process.cwd());
-        rebuildMetadataLight(paths);
+        // Manual page edits also rebuild off the critical path. Without a
+        // runtime (shouldn't happen in normal wiring) fall back to inline.
+        if (runtime) {
+          const launchCtx = ctx ? { hasUI: ctx.hasUI, ui: ctx.ui } : { hasUI: false as const };
+          scheduleReindex(runtime, launchCtx, paths);
+        } else {
+          rebuildMetadataLight(paths);
+        }
       } catch {
         // Silently fail — metadata rebuild is best-effort
       }