@nusoft/nuos-build-catalogue 0.1.0

package/README.md

@@ -0,0 +1,91 @@
+# nuos-build-catalogue
+
+Indexes the NuOS build catalogue (`docs/build/`, `docs/contracts/`, `docs/philosophy/`, `docs/guides/`) into NuVector for semantic search. Implements [WU 110](../nuos/docs/build/work-units/110-index-catalogue-into-nuvector.md).
+
+This is the first concrete step in NuOS taking over its own build, per [D040](../nuos/docs/build/decisions/D040-nuos-led-build-is-foundation-not-parallel-track.md). Before WU 110, finding things in the catalogue meant `grep`. After WU 110, it means semantic queries with metadata filters.
+
+## Setup
+
+```bash
+npm install
+```
+
+The embedder is selected via `NUOS_CATALOGUE_EMBEDDER`:
+
+| Value | Provider | Default model | Dimensions | Notes |
+|---|---|---|---|---|
+| `ollama` (default) | Local Ollama | `qwen3-embedding:8b` | 4096 | **Sovereignty by default.** No network egress. Override the model with `NUOS_CATALOGUE_OLLAMA_MODEL=qwen3-embedding:4b` (2560 dims) or `qwen3-embedding:0.6b` (1024 dims) for smaller boxes. Needs `ollama serve` running and the model pulled (`ollama pull qwen3-embedding:8b`). |
+| `vertex` | Google Vertex | `text-embedding-005` | 768 | Cloud Google. Needs `GOOGLE_CLOUD_PROJECT` plus a Vertex access token (set `GOOGLE_VERTEX_ACCESS_TOKEN`, or have `gcloud` on PATH and run `gcloud auth application-default login`). |
+| `openai` | OpenAI | `text-embedding-3-small` | 1536 | Cloud OpenAI. Needs `OPENAI_API_KEY`. |
+| `stub` | Hash-based, no API | — | 384 | Tests + dev only. Results are noisy. |
+
+Switching embedder (or model variant) requires a full reindex (`rm -rf .nuos-catalogue && npm run index`) because dimensions differ.
+
+## Quick start
+
+```bash
+# Pre-flight (one time):
+ollama serve                          # in another shell
+ollama pull qwen3-embedding:8b        # ~4.7 GB download
+
+# Index the catalogue (first time — takes ~20 min on 8b)
+npm run index
+
+# Search
+npm run search -- "module boundary enforcement"
+npm run search -- "epistemic discipline" --kind=decision --limit=5
+npm run search -- "EHCP lifecycle" --json
+
+# Re-run after editing a few files (only changed files re-embed)
+npm run index
+```
+
+## Storage
+
+Index lives at `.nuos-catalogue/index.nv` (file-backed NuVector store, REDB underneath) plus a sibling `hashes.json` mapping each file path to its content SHA-256 + the chunk IDs it produced. Both are committed to git so the index state is reproducible across machines (Topology A per [D041](../nuos/docs/build/decisions/D041-nuos-meta-package-integration-layer.md)).
+
+If `.nuos-catalogue/index.nv` ever gets corrupt or out-of-sync, delete the directory and re-run `npm run index`.
+
+## Verification gate
+
+Before any other code lands, the verification gate proves that `@nusoft/nuvector@0.1.0` actually persists file-backed storage across process restarts. Re-run it any time you bump the NuVector dep or suspect storage is broken:
+
+```bash
+npm run verify-storage
+```
+
+Pass = file storage works in the published binary; the indexer can use it. Fail = something has regressed and the package needs a Postgres fallback or a NuVector fix.
+
+## Architecture in one paragraph
+
+`crawl` walks the catalogue picking up `.md` files (skipping `_index.md`, `done/`, `archive/`, `superseded/`). Each file goes to `chunkMarkdown` which splits on H1/H2/H3 boundaries (preserving code fences) into ~600-token chunks with deterministic IDs. `extractMetadata` produces structured metadata per file (kind, idInKind, status, date, cross-refs). The `Embedder` then turns chunk texts into Float32Array vectors. The orchestrator (`runIndex`) only re-embeds files whose content hash has changed since the last run, then `upsert`s them into NuVector as `nuwiki_article_summary` records. `runSearch` embeds the query and calls `searchKnowledge` to retrieve the top-K hits, which the formatter renders as a human-readable list or JSON.
+
+## Out of scope (Phase 0)
+
+- Auto-running on commit — that's WU 128.
+- A GUI — CLI only.
+- Writing to NuVector via NuFlow workflows — that's WU 111.
+- Compiling NuWiki articles from indexed content — WU 113–115.
+- Multi-user / concurrent indexing.
+- Adopting `@nusoft/nuos` — uses NuVector directly until WU 130 ships.
+
+## Known API quirks (NuVector v0.1.0)
+
+Discovered during the WU 110 implementation; documented here so future contributors don't burn time rediscovering them:
+
+- `embedding` must be a `Float32Array`, not a plain `number[]`. Plain arrays fail with `Get TypedArray info failed on NvMemoryRecord.embedding`.
+- Search results expose the upsert-time `id` as `ref` on each item (asymmetry with the input shape).
+- `tenant` belongs on `MemoryRecord` (upsert) but not on `SearchKnowledgeRequest` — the store-level tenant from `NuVector.open` scopes search automatically.
+- `searchKnowledge` operates on Layer 1 records (`nuwiki_article_summary`); to make a chunk retrievable directly, index it as `nuwiki_article_summary`. `nuwiki_section` is for sections within an enclosing article and requires `searchSectionsInArticles` with the article IDs.
+
+## Tests
+
+```bash
+npx tsx --test tests/
+```
+
+13 tests across `chunk`, `metadata`, `crawl` cover the indexing primitives. End-to-end is exercised by running `npm run index` then `npm run search` against the real catalogue.
+
+## License
+
+Private; not published to npm.

package/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "@nusoft/nuos-build-catalogue",
+  "version": "0.1.0",
+  "description": "Indexes the NuOS build catalogue into NuVector for semantic search. WU 110.",
+  "type": "module",
+  "bin": {
+    "nuos-catalogue": "./dist/cli.js"
+  },
+  "files": [
+    "dist",
+    "scripts",
+    "README.md"
+  ],
+  "publishConfig": {
+    "access": "restricted"
+  },
+  "scripts": {
+    "build": "tsc",
+    "verify-storage": "tsx scripts/verify-persistence.ts",
+    "test": "tsx --test tests/chunk.test.ts tests/metadata.test.ts tests/crawl.test.ts",
+    "typecheck": "tsc --noEmit",
+    "index": "tsx src/cli.ts index",
+    "search": "tsx src/cli.ts search"
+  },
+  "dependencies": {
+    "@nusoft/nuvector": "^0.1.5"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "tsx": "^4.19.0",
+    "typescript": "^5.5.0"
+  }
+}

package/scripts/verify-persistence.ts

@@ -0,0 +1,143 @@
+/**
+ * WU 110 verification gate — Pattern J discipline.
+ *
+ * The @nusoft/nuvector README documents three storage backends:
+ *   - "memory:"            — in-memory, ephemeral
+ *   - "./project.nv"       — local file
+ *   - { kind: "postgres" } — Postgres
+ *
+ * Before WU 110 commits to file-backed storage, we must prove empirically
+ * that opening a file-backed store, upserting a record, closing it, and
+ * reopening in a fresh process actually returns the record. Hedge words
+ * ("the README says it works") are not acceptance — only this script
+ * passing is.
+ *
+ * This is run twice: once in --write mode, then again in --read mode.
+ * Run via: pnpm verify-storage  (does both phases in one process tree)
+ */
+
+import { NuVector } from '@nusoft/nuvector';
+import { spawn } from 'node:child_process';
+import { existsSync, rmSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+import path from 'node:path';
+
+const __filename = fileURLToPath(import.meta.url);
+const STORAGE_PATH = path.resolve(path.dirname(__filename), '../.verify-test.nv');
+const DIMENSIONS = 8; // tiny for this smoke test
+const TENANT = 'verify_test';
+const RECORD_ID = 'verify_record_001';
+
+// Deterministic non-zero embedding (NuVector requires Float32Array)
+const fixedEmbedding = new Float32Array(
+  Array.from({ length: DIMENSIONS }, (_, i) => (i + 1) / 10),
+);
+
+async function writePhase(): Promise<void> {
+  console.log('[write phase] cleaning previous state at', STORAGE_PATH);
+  if (existsSync(STORAGE_PATH)) {
+    rmSync(STORAGE_PATH, { recursive: true, force: true });
+  }
+
+  console.log('[write phase] opening NuVector with file storage');
+  const memory = await NuVector.open({
+    storage: STORAGE_PATH,
+    dimensions: DIMENSIONS,
+    tenant: TENANT,
+  });
+
+  console.log('[write phase] upserting record', RECORD_ID);
+  await memory.upsert({
+    id: RECORD_ID,
+    kind: 'nuwiki_article_summary',
+    embedding: fixedEmbedding,
+    text: 'verification record — must survive process restart',
+    tenant: TENANT,
+    metadata: { test: 'verify-persistence', written_at: Date.now() },
+  });
+
+  console.log('[write phase] record written, closing store');
+  // NuVector may or may not expose a close(); rely on process exit to flush.
+  process.exit(0);
+}
+
+async function readPhase(): Promise<void> {
+  console.log('[read phase] reopening NuVector at', STORAGE_PATH);
+  if (!existsSync(STORAGE_PATH)) {
+    console.error('[read phase] FAIL — storage path does not exist after write phase');
+    process.exit(2);
+  }
+
+  const memory = await NuVector.open({
+    storage: STORAGE_PATH,
+    dimensions: DIMENSIONS,
+    tenant: TENANT,
+  });
+
+  console.log('[read phase] searching for the verification record');
+  const result = await memory.searchKnowledge({
+    query: 'verification record',
+    embedding: fixedEmbedding,
+    budget: { maxTokens: 1000, maxArticles: 5 },
+  });
+
+  const items = (result?.items ?? []) as Array<{
+    ref?: string;
+    id?: string;
+    metadata?: Record<string, unknown>;
+  }>;
+  const found = items.some((item) => item.ref === RECORD_ID || item.id === RECORD_ID);
+
+  if (found) {
+    console.log('[read phase] PASS — record retrieved across process restart');
+    console.log('[read phase] verdict: file-backed persistence WORKS in this NuVector build');
+    process.exit(0);
+  } else {
+    console.error('[read phase] FAIL — record not found after restart');
+    console.error('[read phase] retrieved items:', JSON.stringify(items, null, 2));
+    console.error('[read phase] verdict: file-backed persistence does NOT work in this NuVector build');
+    console.error('[read phase] WU 110 must fall back to Postgres');
+    process.exit(3);
+  }
+}
+
+async function main(): Promise<void> {
+  const mode = process.argv[2];
+
+  if (mode === '--write') {
+    await writePhase();
+    return;
+  }
+  if (mode === '--read') {
+    await readPhase();
+    return;
+  }
+
+  // Orchestrator: spawn write then read in fresh processes
+  console.log('=== WU 110 storage-backend verification gate ===');
+  await new Promise<void>((resolve, reject) => {
+    const child = spawn('npx', ['tsx', __filename, '--write'], { stdio: 'inherit' });
+    child.on('exit', (code) => {
+      if (code === 0) resolve();
+      else reject(new Error(`write phase exited with code ${code}`));
+    });
+  });
+
+  await new Promise<void>((resolve, reject) => {
+    const child = spawn('npx', ['tsx', __filename, '--read'], { stdio: 'inherit' });
+    child.on('exit', (code) => {
+      if (code === 0) {
+        console.log('\n=== VERIFICATION GATE: PASS ===');
+        resolve();
+      } else {
+        console.error('\n=== VERIFICATION GATE: FAIL ===');
+        process.exit(code ?? 1);
+      }
+    });
+  });
+}
+
+main().catch((err) => {
+  console.error('verification script error:', err);
+  process.exit(1);
+});