memorix 1.9.0 → 1.9.1

package/docs/CONFIGURATION.md

@@ -195,6 +195,28 @@ OpenAI-only default. The equivalent env-var form is
 `MEMORIX_EMBEDDING=api`, `MEMORIX_EMBEDDING_BASE_URL=https://openrouter.ai/api/v1`,
 `MEMORIX_EMBEDDING_MODEL=qwen/qwen3-embedding-8b`.
 
+### Large-store retrieval
+
+Memorix does not cap the number of durable observations to make search fast.
+For a large store, the runtime keeps the complete records in SQLite and uses
+derived indexes for bounded candidate retrieval. The optional LanceDB package
+is installed with the package's optional dependencies and loaded only at
+runtime; if its native binary is unavailable, SQLite FTS5 and the Orama
+fallback continue to work.
+
+```text
+MEMORIX_ORAMA_HYDRATION_THRESHOLD=10000   # working-set switch, not a quota
+MEMORIX_SEMANTIC_INDEX=auto               # auto | off | orama
+MEMORIX_SEMANTIC_INDEX_THRESHOLD=10000    # train ANN index after this many vectors
+```
+
+`MEMORIX_SEMANTIC_INDEX=auto` enables the local LanceDB shadow index when the
+optional native package is available. It stores vector IDs and filter metadata,
+while full memory text remains in SQLite. The semantic index uses HNSW with
+scalar quantization after its threshold is reached; before that, its table is
+still searchable without a training step. The derived index can be deleted and
+recreated by the background vector backfill without affecting durable memories.
+
 ### `[rerank]`
 
 Optional HTTP rerank for thorough search. Off by default.

package/docs/DEVELOPMENT.md

@@ -8,12 +8,12 @@ Memorix is a TypeScript project built around:
 - memcode bundled terminal-agent runtime
 - CLI workflows
 - SQLite canonical persistence with compatibility/fallback layers
-- Orama search
+- layered retrieval: SQLite FTS5, optional LanceDB semantic index, and Orama fallback
 - dashboard and HTTP service
 
 ## Current Development Baseline
 
-The current release line is **1.9.0** and package metadata is kept synchronized
+The current release line is **1.9.1** and package metadata is kept synchronized
 with the root package and MCP Registry manifest.
 
 Contributors should assume the following areas are part of the current release line:

package/docs/PERFORMANCE.md

@@ -12,7 +12,10 @@ Memorix is designed to be light for everyday memory use and explicit about heavi
 | `memorix` | Interactive terminal UI | Human operator workbench |
 | `memorix orchestrate` | Supervisor plus spawned CLI agent workers | Orchestrated subagent loops |
 
-The default memory path uses local SQLite as the canonical store and Orama for search/indexing. No cloud service is required.
+The default memory path uses local SQLite as the canonical store. Orama remains
+the in-process compatibility/semantic fallback; large corpora use SQLite FTS5
+and, when available, the local LanceDB semantic shadow index. No cloud service
+is required.
 
 ## Resource Safety Model
 
@@ -66,11 +69,11 @@ project-scoped tools (`memorix_project_context`, `memorix_context_pack`,
 CodeGraph Memory directly, so a new agent can get a useful brief without
 waiting for unrelated historical records to hydrate into Orama.
 
-Search, writes, and session operations still wait for the in-process retrieval
-runtime when they need it. Corpus-scale retention, consolidation, and Code
-Memory refresh are queued durably and run in an isolated child process. Vector
-backfill stays with the owning process because the Orama index is local to that
-process.
+Search and ordinary writes use the durable SQLite path directly when the
+persistent indexes are available. Corpus-scale retention, consolidation, and
+Code Memory refresh are queued durably and run in an isolated child process.
+Vector backfill updates the persistent semantic shadow index and keeps Orama as
+the compatibility fallback.
 
 ## What Is Lightweight
 
@@ -105,6 +108,9 @@ On the release development machine used for this check, the healthy HTTP service
 | `MEMORIX_RERANK_PROVIDER` | `off` | Set `http` to enable optional HTTP rerank |
 | `MEMORIX_RERANK_BASE_URL` | `[memory.llm].base_url` | Compatible `/rerank` API root (path `/rerank` is appended) |
 | `MEMORIX_EMBEDDING_CACHE_MAX_BYTES` | `268435456` | Disposable in-process API-vector cache budget; raise for large warm caches, or lower when the host is memory constrained |
+| `MEMORIX_ORAMA_HYDRATION_THRESHOLD` | `10000` | Above this durable corpus size, skip full Orama hydration and use persistent SQLite/semantic indexes; this is a working-set threshold, not a memory quota |
+| `MEMORIX_SEMANTIC_INDEX` | `auto` | `auto` uses the optional local LanceDB shadow index when installed; `off` or `orama` keeps the legacy semantic path |
+| `MEMORIX_SEMANTIC_INDEX_THRESHOLD` | `10000` | Minimum vectors before the optional HNSW/SQ index is trained; vectors remain searchable before training |
 | `memorix memory search --quality fast` | n/a | Force a fully local retrieval path for a latency-sensitive call |
 | `npm run benchmark:retrieval -- --records 1000 --runs 100` | n/a | Reproduce hot in-process lexical retrieval latency; not an end-to-end claim |
 | `npm run gate:large-store -- --records 40000` | n/a | Exercise SDK, HTTP MCP, hook persistence, cache integrity, and reopen behavior against a large isolated store |
@@ -130,6 +136,10 @@ On the release development machine used for this check, the healthy HTTP service
   rising `rss` with stable heap, or rising V8 `heapUsed`; those indicate
   different classes of problem and should not be “fixed” by lowering the
   memory corpus limit.
+- When a corpus crosses `MEMORIX_ORAMA_HYDRATION_THRESHOLD`, the complete
+  durable corpus remains in SQLite. Only the per-query candidate set is loaded
+  into the agent process. Management actions such as export, retention, and
+  detail views may intentionally load the requested full/project slice.
 
 ## Large-Store Release Gate
 
@@ -143,22 +153,69 @@ report fails when an interactive path exceeds its release budget. It also
 verifies that the personal hook candidate remains durable on disk without
 becoming visible to an unbound SDK reader.
 
-Startup hydration uses bounded Orama batches. This keeps HTTP health responsive
-while avoiding a full index rebalance for every persisted observation. The MCP
-search number is intentionally a cold control-plane measurement: it includes
-any lexical hydration the fresh service still owes before its first search.
+Small stores may use bounded Orama batches. Large stores skip that hydration and
+use the persistent FTS5 path immediately; the optional semantic shadow index is
+opened independently. The MCP search number is intentionally a cold
+control-plane measurement and includes only the work still owed by the selected
+derived indexes.
 
 Results are machine-specific and should be compared on the same OS and Node
 version. The release gate is an integrity and regression check, not a public
 latency guarantee.
 
-## Current Optimization Opportunities
-
-These are not release blockers, but they are reasonable future improvements:
-
-- Extend the retrieval benchmark with separately labelled cold CLI, MCP, and remote-provider cases when a release needs those comparisons.
-- Add dashboard-side performance telemetry for API latency and payload sizes.
-- Document recommended retention schedules for large projects.
-- Evaluate a persistent cross-process retrieval index only as a major-version
-  architecture change; Orama is intentionally process-local today.
-- Explore slimmer Docker layers if image size becomes a common pain point.
+## 1.9.1 Large-Scale Retrieval Contract
+
+The 1.9.1 line improves retrieval latency without shrinking the user's durable
+memory corpus. SQLite remains the source of truth. Search indexes are derived,
+rebuildable working structures, following the same boundary used by mature
+memory systems that keep human-readable or durable records separate from their
+vector/lexical indexes.
+
+The release contract is:
+
+1. A normal lexical or exact-identifier query must be able to use a persistent
+   SQLite FTS5 index and fetch only bounded candidates. It must not require
+   hydrating every observation into Orama first.
+2. FTS5 must be feature-detected. Older or degraded SQLite runtimes fall back
+   to the existing Orama path without losing data or changing write semantics.
+3. Inserts, updates, deletes, imports, and sync applies must keep the derived
+   lexical index current. A rebuild command/test path must repair it after an
+   interrupted migration or manual database recovery.
+4. Candidate generation, graph/code references, and optional reranking must be
+   separately measurable. Reranking is applied only to a small candidate set;
+   it must never scan or send the whole corpus to a provider.
+5. No durable observation count, retention rule, or user-facing memory quota is
+   introduced by this work. Limits apply only to per-query candidates, caches,
+   and concurrent maintenance work.
+6. Large-store acceptance must report cold start, warm search, MCP search, RSS,
+   heap, index mode, result integrity, and recall of planted records at 1k,
+   10k, 40k, and 100k records. A slower machine is evidence to report, not a
+   reason to quietly lower the corpus size.
+
+The implementation target is SQLite FTS5 plus an optional local LanceDB shadow
+index. LanceDB stores only vector IDs and filter metadata, and is loaded at
+runtime so the main package still builds on platforms where its native binary
+is unavailable. HNSW with scalar quantization is trained only after the vector
+table crosses the semantic-index threshold; before that, the table remains
+queryable without paying a large training spike. Orama remains the compatibility
+fallback. A server-only vector database is not a default dependency for the
+local-first product.
+
+The retrieval shape is:
+
+```text
+SQLite observations (durable truth)
+  -> FTS5 exact/lexical candidates
+  -> optional vector candidates
+  -> optional CodeGraph/knowledge candidates
+  -> rank fusion
+  -> optional rerank on the short list
+  -> compact result IDs, then detail on demand
+```
+
+Measured Windows/Node 22 evidence for this implementation: 1k, 10k, 40k, and
+100k records all passed the large-store gate. At 100k, peak RSS was about
+419MB, SDK reopen was about 131ms, and the cold HTTP MCP search was about
+296ms. These are same-machine acceptance numbers, not universal latency
+promises. The release is complete only when the measurements and fallback
+behavior remain recorded alongside the implementation.

package/docs/README.md

@@ -125,7 +125,7 @@ Historical/deep-reference documents may describe older designs. If they conflict
 
 ## Current Product Line
 
-The current product line is **1.9.0**. Multi-device store sync is opt-in and
+The current product line is **1.9.1**. Multi-device store sync is opt-in and
 keeps local SQLite canonical; existing users do not upload anything until they
 configure a relay provider explicitly.
 The authoritative acceptance contract for the current implementation is

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "memorix",
   "mcpName": "io.github.AVIDS2/memorix",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Local-first shared memory layer for AI coding agents across MCP clients, Git history, reasoning context, and project sessions.",
   "type": "module",
   "sideEffects": false,
@@ -155,6 +155,7 @@
     "zod": "^3.24.0"
   },
   "optionalDependencies": {
+    "@lancedb/lancedb": "^0.38.0",
     "better-sqlite3": "^12.11.1"
   },
   "overrides": {

package/plugins/antigravity/memorix/plugin.json

@@ -1,4 +1,4 @@
 {
   "name": "memorix",
-  "version": "1.9.0"
+  "version": "1.9.1"
 }

package/plugins/claude/memorix/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://anthropic.com/claude-code/plugin.schema.json",
   "name": "memorix",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Shared workspace memory for Claude Code and other AI coding agents.",
   "author": {
     "name": "AVIDS2",

package/plugins/codebuddy/memorix/.codebuddy-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "memorix",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Local-first project memory for CodeBuddy Code and other coding agents.",
   "author": {
     "name": "AVIDS2",

package/plugins/codex/memorix/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "memorix",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Shared workspace memory for Codex and other AI coding agents.",
   "author": {
     "name": "AVIDS2",

package/plugins/copilot/memorix/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "memorix",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Shared local workspace memory for GitHub Copilot CLI and other AI coding agents.",
   "author": {
     "name": "AVIDS2",

package/plugins/gemini/memorix/gemini-extension.json

@@ -1,6 +1,6 @@
 {
   "name": "memorix",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "mcpServers": {
     "memorix": {
       "command": "memorix",

package/plugins/hermes/memorix/plugin.yaml

@@ -1,5 +1,5 @@
 name: memorix
-version: "1.9.0"
+version: "1.9.1"
 description: Shared workspace memory bridge for Hermes Agent.
 author: AVIDS2
 license: Apache-2.0

package/plugins/omp/memorix/package.json

@@ -1,6 +1,6 @@
 {
   "name": "memorix-omp-package",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Memorix shared workspace memory package for Oh-my-Pi.",
   "license": "Apache-2.0",
   "keywords": [

package/plugins/openclaw/memorix/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "memorix",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Shared workspace memory for OpenClaw and other AI coding agents.",
   "author": {
     "name": "AVIDS2",

package/plugins/pi/memorix/package.json

@@ -1,6 +1,6 @@
 {
   "name": "memorix-pi-package",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "Memorix shared workspace memory package for Pi coding agent.",
   "license": "Apache-2.0",
   "keywords": [

package/server.json

@@ -7,7 +7,7 @@
     "url": "https://github.com/AVIDS2/memorix",
     "source": "github"
   },
-  "version": "1.9.0",
+  "version": "1.9.1",
   "websiteUrl": "https://mem.rglens.com",
   "icons": [
     {
@@ -22,7 +22,7 @@
     {
       "registryType": "npm",
       "identifier": "memorix",
-      "version": "1.9.0",
+      "version": "1.9.1",
       "runtimeHint": "npx",
       "packageArguments": [
         {

package/src/compact/engine.ts

@@ -71,7 +71,7 @@ export async function compactSearch(options: SearchOptions, surface: RetrievalSu
   formatted: string;
   totalTokens: number;
 }> {
-  await ensureFreshIndex();
+  await ensureFreshIndex({ loadCorpus: false });
   const searchOptions = { ...options, query: normalizeMemoryBrowseQuery(options.query) };
   let entries = (await searchObservations(searchOptions)).map((entry) =>
     entry.projectId || !options.projectId ? entry : { ...entry, projectId: options.projectId },
@@ -102,9 +102,19 @@ export async function compactSearch(options: SearchOptions, surface: RetrievalSu
   let formatted = formatIndexTable(entries, searchOptions.query, !options.projectId, surface);
 
   if (entries.length === 0 && options.projectId) {
-    const allObservations = filterReadableObservations(getAllObservations(), options.reader);
     const projectAliases = new Set(await resolveAliases(options.projectId).catch(() => [options.projectId]));
-    const projectHasStoredMemory = allObservations.some((obs) => obs.projectId && projectAliases.has(obs.projectId));
+    let projectHasStoredMemory = false;
+    try {
+      const store = getObservationStore();
+      projectHasStoredMemory = (await Promise.all(
+        [...projectAliases].filter((projectId): projectId is string => Boolean(projectId)).map((projectId) => store.countByProject(projectId, { status: 'active' })),
+      )).some((count) => count > 0);
+    } catch {
+      const allObservations = filterReadableObservations(getAllObservations(), options.reader);
+      projectHasStoredMemory = allObservations.some((observation) =>
+        observation.projectId !== undefined && projectAliases.has(observation.projectId),
+      );
+    }
     if (!projectHasStoredMemory) {
       formatted =
         options.reader
@@ -115,7 +125,7 @@ export async function compactSearch(options: SearchOptions, surface: RetrievalSu
             `Memories will start appearing after observations, session summaries, hook captures, or git-memory are written.`;
     } else {
       formatted =
-        `No memories found matching "${options.query}".\n\n` +
+        `No memories found for the current query.\n\n` +
         `This project does have stored Memorix memories, but none matched the current query/filter. ` +
         `Try a more specific topic, use /memory status for runtime diagnostics, or ask for recent memory if you want a broad overview.`;
     }

package/src/memory/freshness.ts

@@ -90,9 +90,9 @@ export async function reindexMiniSkills(): Promise<number> {
  * Ensure both observations and mini-skills are fresh in the Orama index.
  * Returns true if any data source was refreshed.
  */
-export async function ensureFreshIndex(): Promise<boolean> {
+export async function ensureFreshIndex(options: { loadCorpus?: boolean } = {}): Promise<boolean> {
   let anyStale = false;
-  const obsStale = await ensureFreshObservations();
+  const obsStale = await ensureFreshObservations(options);
   if (obsStale) anyStale = true;
   const skillsStale = await ensureFreshMiniSkills();
   if (skillsStale) anyStale = true;
@@ -108,8 +108,11 @@ export async function ensureFreshIndex(): Promise<boolean> {
  *
  * Phase 3a: replaces withFreshObservations() at all retrieval call sites.
  */
-export async function withFreshIndex<T>(fn: () => T | Promise<T>): Promise<T> {
-  await ensureFreshIndex();
+export async function withFreshIndex<T>(
+  fn: () => T | Promise<T>,
+  options: { loadCorpus?: boolean } = {},
+): Promise<T> {
+  await ensureFreshIndex(options);
   return fn();
 }
 

package/src/memory/observations.ts

@@ -29,6 +29,7 @@ import {
   getDb,
   getDeferredCachedVectorHydration,
   getVectorDimensions,
+  queueSemanticVector,
   hydrateIndexForStartup,
   hasObservationVector,
   isEmbeddingEnabled,
@@ -64,13 +65,25 @@ export interface ObservationRuntimeOptions {
    * tens of thousands of rows on every PostToolUse.
    */
   skipCorpusLoad?: boolean;
+  /** Force a maintenance worker to load the corpus for vector backfill. */
+  forceCorpusLoad?: boolean;
 }
 
 let embeddingWriteMode: ObservationEmbeddingWriteMode = 'background';
 let embeddingWorkerProjectRoot: string | undefined;
 let corpusLoaded = false;
+let largeCorpusLazyMode = false;
 let loadedObservationStore: ObservationStore | null = null;
 
+const DEFAULT_ORAMA_HYDRATION_THRESHOLD = 10_000;
+
+function resolveOramaHydrationThreshold(): number {
+  const configured = Number.parseInt(process.env.MEMORIX_ORAMA_HYDRATION_THRESHOLD ?? '', 10);
+  return Number.isSafeInteger(configured) && configured > 0
+    ? configured
+    : DEFAULT_ORAMA_HYDRATION_THRESHOLD;
+}
+
 /** @internal Reset in-process observation state between tests. */
 export function resetObservationRuntime(): void {
   observations = [];
@@ -78,6 +91,7 @@ export function resetObservationRuntime(): void {
   projectDir = null;
   searchIndexPrepared = false;
   corpusLoaded = false;
+  largeCorpusLazyMode = false;
   loadedObservationStore = null;
   embeddingWriteMode = 'background';
   embeddingWorkerProjectRoot = undefined;
@@ -148,6 +162,22 @@ function vectorBackfillError(error: unknown): string {
   return sanitizeCredentials(normalizeEmbeddingFailure(error).message).slice(0, 1_000);
 }
 
+function queueSemanticVectorSafely(
+  dataDir: string,
+  record: Parameters<typeof queueSemanticVector>[1],
+): void {
+  try {
+    // The derived index is best-effort. Catch both synchronous compatibility
+    // failures from older test adapters and asynchronous native-index errors.
+    void Promise.resolve(queueSemanticVector(dataDir, record)).catch(() => {
+      // Durable SQLite/FTS remains authoritative; backfill can rebuild vectors.
+    });
+  } catch {
+    // Test adapters and older runtimes may not expose the optional derived
+    // index hook. The durable observation and Orama fallback remain valid.
+  }
+}
+
 function queueVectorBackfill(
   projectId: string,
   options: { detachedWorker?: boolean; projectRoot?: string } = {},
@@ -300,6 +330,22 @@ export async function initObservations(
     loadedObservationStore = store;
     return;
   }
+
+  const durableCount = await store.countAll?.();
+  largeCorpusLazyMode = Boolean(
+    !options.forceCorpusLoad &&
+    durableCount !== undefined &&
+    durableCount >= resolveOramaHydrationThreshold() &&
+    store.hasLexicalIndex?.(),
+  );
+  if (largeCorpusLazyMode) {
+    // SQLite/FTS5 and the optional persistent semantic index serve the search
+    // path. Management/detail paths explicitly call ensureCorpusLoaded().
+    observations = [];
+    corpusLoaded = false;
+    loadedObservationStore = store;
+    return;
+  }
   observations = await store.loadAll();
   corpusLoaded = true;
   loadedObservationStore = store;
@@ -349,6 +395,15 @@ function scheduleObservationEmbedding(input: {
       try {
         await removeObservation(makeOramaObservationId(projectId, observationId));
         await insertObservation(Object.assign({}, doc, { embedding }));
+        if (projectDir) {
+          queueSemanticVectorSafely(projectDir, {
+            observationId,
+            projectId,
+            status: doc.status,
+            visibility: doc.visibility,
+            vector: embedding,
+          });
+        }
         vectorMissingIds.delete(observationId);
       } catch {
         console.error(`[memorix] Embedding index update failed for obs-${observationId} (kept in backfill queue)`);
@@ -387,16 +442,27 @@ function scheduleObservationEmbedding(input: {
  *
  * For DegradedBackend this is a no-op (always returns false).
  */
-export async function ensureFreshObservations(): Promise<boolean> {
-  if (!projectDir || !corpusLoaded) return false;
+export async function ensureFreshObservations(options: { loadCorpus?: boolean } = {}): Promise<boolean> {
+  if (!projectDir) return false;
+  const loadCorpus = options.loadCorpus !== false;
   try {
     const store = getObservationStore();
     const wasStale = await store.ensureFresh();
-    if (wasStale) {
+    if (!corpusLoaded && loadCorpus) {
       observations = await store.loadAll();
       nextId = await store.loadIdCounter();
-      await reindexObservations();
-      searchIndexPrepared = true;
+      corpusLoaded = true;
+      largeCorpusLazyMode = false;
+      searchIndexPrepared = false;
+      return true;
+    }
+    if (wasStale) {
+      nextId = await store.loadIdCounter();
+      if (corpusLoaded) {
+        observations = await store.loadAll();
+        await reindexObservations();
+        searchIndexPrepared = true;
+      }
       return true;
     }
   } catch {
@@ -405,6 +471,11 @@ export async function ensureFreshObservations(): Promise<boolean> {
   return false;
 }
 
+/** Load the full durable corpus for management/detail operations only. */
+export async function ensureCorpusLoaded(): Promise<void> {
+  await ensureFreshObservations({ loadCorpus: true });
+}
+
 /**
  * @internal Observation-only freshness gate.
  *
@@ -491,7 +562,7 @@ export async function storeObservation(input: {
   // Sync the local cache before using it as the topicKey fast path. This costs a
   // generation read in the normal case and only reloads when another process wrote.
   // Write-only / hook processes skip the corpus cache; disk findByTopicKey is authoritative.
-  await ensureFreshObservations();
+  await ensureFreshObservations({ loadCorpus: false });
 
   // Topic key upsert: fast-path check in-memory (optimistic, may be stale).
   // A second authoritative check happens inside the file lock to prevent TOCTOU races
@@ -612,6 +683,21 @@ export async function storeObservation(input: {
       // Phase 4a: confirm writeGeneration matches actual post-bump value
       observation.writeGeneration = store.getGeneration();
 
+      // A long-lived writer can cross the hydration threshold after startup.
+      // Release the hot Orama corpus at that point and let FTS5/semantic shadow
+      // indexes serve subsequent searches. Durable SQLite rows are untouched.
+      if (
+        corpusLoaded &&
+        observations.length >= resolveOramaHydrationThreshold() &&
+        store.hasLexicalIndex?.()
+      ) {
+        observations = [];
+        corpusLoaded = false;
+        largeCorpusLazyMode = true;
+        searchIndexPrepared = false;
+        await resetDb();
+      }
+
       if (upsertedInsideLock || reloadCacheAfterCommit) {
         if (corpusLoaded) {
           observations = await store.loadAll();
@@ -1101,6 +1187,11 @@ export function getObservationCount(): number {
   return observations.length;
 }
 
+/** The current durable data directory used by the observation runtime. */
+export function getObservationDataDir(): string | null {
+  return projectDir;
+}
+
 /**
  * Suggest a stable topic key from type + title.
  * Uses family heuristics (architecture/*, bug/*, decision/*, etc.)
@@ -1246,6 +1337,18 @@ export async function reindexObservations(): Promise<number> {
 export async function prepareSearchIndex(options: { skipCachedVectors?: boolean } = {}): Promise<number> {
   if (searchIndexPrepared) return 0;
 
+  if (largeCorpusLazyMode && getObservationStore().hasLexicalIndex?.()) {
+    // Search is served by SQLite FTS5 plus the optional persistent semantic
+    // shadow index. Do not pay the old full-corpus Orama hydration cost.
+    searchIndexPrepared = true;
+    if (!isEmbeddingExplicitlyDisabled()) {
+      const store = getObservationStore();
+      const projects = await store.listProjectIds?.() ?? [];
+      for (const projectId of projects) queueVectorBackfill(projectId);
+    }
+    return 0;
+  }
+
   const count = await hydrateIndexForStartup(observations as unknown as any[], {
     skipCachedVectors: options.skipCachedVectors,
   });
@@ -1487,6 +1590,15 @@ export async function backfillVectorEmbeddings(options: {
             embedding,
           };
           await insertObservation(doc);
+          if (projectDir) {
+            queueSemanticVectorSafely(projectDir, {
+              observationId: id,
+              projectId: obs.projectId,
+              status: doc.status,
+              visibility: doc.visibility,
+              vector: embedding,
+            });
+          }
           vectorMissingIds.delete(id);
           succeeded++;
         } catch (error) {

package/src/runtime/vector-backfill-runner.ts

@@ -252,7 +252,7 @@ export async function executeVectorBackfill(request: VectorBackfillRequest) {
   initProjectRoot(request.projectRoot);
   loadDotenv(request.projectRoot);
   await initObservationStore(request.dataDir);
-  await initObservations(request.dataDir);
+  await initObservations(request.dataDir, { forceCorpusLoad: true });
   await prepareSearchIndex();
   await getDeferredCachedVectorHydration()?.catch(() => {});