moflo 4.8.38 → 4.8.40

package/.claude/guidance/shipped/memory-strategy.md

@@ -1,204 +1,276 @@
-# Memory & Semantic Search Strategy
-
-**Purpose:** How memory, embeddings, and semantic search work in moflo. Reference when debugging memory issues, understanding the search pipeline, or configuring memory for a consumer project.
-
----
-
-## Architecture Overview
-
-```
-Source Files (.claude/guidance/, docs/)
-         |
-         v
-index-guidance.mjs (chunking, RAG linking)
-         |
-         v
-.swarm/memory.db (SQLite - entries + metadata)
-         |
-         v
-build-embeddings.mjs (384-dim neural or hash vectors)
-         |
-         v
-Search layer (cosine similarity - MCP, CLI, or script)
-```
-
----
-
-## Key Files
-
-| File | Purpose |
-|------|---------|
-| `.swarm/memory.db` | SQLite database with all entries, embeddings, metadata |
-| `.swarm/code-map-hash.txt` | SHA-256 hash for incremental code map skip |
-| `.claude-flow/neural/patterns.json` | ReasoningBank learned patterns |
-| `bin/build-embeddings.mjs` | Generates 384-dim embeddings |
-| `bin/index-guidance.mjs` | Indexes guidance files with RAG linking |
-| `bin/generate-code-map.mjs` | Generates structural code map (projects, dirs, types, interfaces) |
-
----
-
-## Embedding Strategy
-
-**Primary model:** `Xenova/all-MiniLM-L6-v2` (384-dim, neural — used by `build-embeddings.mjs`)
-**Fallback model:** `domain-aware-hash-v1` (384-dim, hash — used when Transformers.js unavailable)
-
-**Critical rule:** Query embeddings MUST match stored embeddings. Both the search scripts and MCP tools auto-detect the stored model and generate matching query vectors. Cross-model cosine similarity is meaningless.
-
-**Neural embeddings (primary):**
-- Uses `@xenova/transformers` with ONNX WASM runtime
-- True semantic understanding — "soft delete" matches "mark as deleted" without keyword overlap
-- ~3s for 1000 entries, loaded lazily and cached
-
-**Domain-aware hash (fallback):**
-- 12 domain clusters with project-specific terms
-- SimHash-style word encoding + bigram/trigram features
-- Good keyword-level matching, misses semantic paraphrases
-- No external dependencies — always available, <1s for 1000 entries
-
-See `guidance-memory-strategy.md` for full embedding pipeline details.
-
----
-
-## Search Commands
-
-All methods auto-detect the stored embedding model and generate matching query vectors:
-
-**MCP (Preferred):** `mcp__moflo__memory_search` — `query: "your query", namespace: "guidance"`
-
-**CLI (Fallback):**
-```bash
-npx flo memory search --query "your query" --namespace guidance
-```
-
-**Search options:**
-
-| Flag | Default | Purpose |
-|------|---------|---------|
-| `--namespace` | all | Filter to specific namespace |
-| `--limit` | 5 | Number of results |
-| `--threshold` | 0.3 | Minimum similarity score |
-| `--json` | false | Output as JSON |
-
-### Code Map Search (for codebase navigation)
-
-When you need to find where a type, service, entity, or component lives — search `code-map` BEFORE using Glob/Grep:
-
-**MCP:** `mcp__moflo__memory_search` — `query: "payment service", namespace: "code-map"`
-
-**What code-map contains:**
-
-| Chunk prefix | What it answers |
-|--------------|-----------------|
-| `project:` | "What's in the api project?" |
-| `dir:` | "What types are in the entities directory?" |
-| `iface-map:` | "What implements IPaymentService?" |
-| `type-index:` | "Where is Service defined?" |
-
-**Regenerate:**
-```bash
-npx flo-codemap          # Incremental (skips if unchanged)
-npx flo-codemap --force  # Full rebuild
-```
-
----
-
-## Database Schema
-
-```sql
-memory_entries (
-  id TEXT PRIMARY KEY,
-  key TEXT NOT NULL,           -- e.g., "chunk-guidance-core-0"
-  namespace TEXT,              -- "guidance", "patterns", "default"
-  content TEXT,                -- Full text content
-  embedding TEXT,              -- JSON array of 384 floats
-  embedding_model TEXT,        -- "Xenova/all-MiniLM-L6-v2" or "domain-aware-hash-v1"
-  embedding_dimensions INTEGER,-- 384
-  metadata TEXT,               -- JSON: parentDoc, chunkTitle, prevChunk, nextChunk, siblings
-  tags TEXT,                   -- JSON array
-  status TEXT,                 -- "active"
-  created_at INTEGER,
-  updated_at INTEGER
-)
-```
-
----
-
-## RAG Linking (metadata fields)
-
-Each chunk includes navigation metadata:
-
-| Field | Purpose |
-|-------|---------|
-| `parentDoc` | Key of full document (e.g., `doc-guidance-core`) |
-| `prevChunk` | Previous chunk key for sequential reading |
-| `nextChunk` | Next chunk key |
-| `siblings` | All chunk keys from same document |
-| `hierarchicalParent` | H2 parent for H3 chunks |
-| `hierarchicalChildren` | H3 children for H2 chunks |
-| `contextBefore` | Overlapping text from previous chunk (20%) |
-| `contextAfter` | Overlapping text from next chunk (20%) |
-
----
-
-## Namespaces
-
-| Namespace | Content | Notes |
-|-----------|---------|-------|
-| `guidance` | Indexed guidance and docs | Largest — includes bundled moflo guidance |
-| `code-map` | Structural codebase index (projects, directories, types, interfaces) | Search BEFORE Glob/Grep for navigation |
-| `patterns` | Learned patterns from sessions | Grows over time |
-| `default` | Misc stored data | Small |
-
----
-
-## Session Start Indexing
-
-On every session start, moflo automatically runs three background indexers:
-
-| Indexer | Command | Namespace | What it does |
-|---------|---------|-----------|--------------|
-| Guidance | `npx flo-index` | `guidance` | Chunks markdown, builds RAG links, generates embeddings |
-| Code Map | `npx flo-codemap` | `code-map` | Scans source for types, interfaces, directories |
-| Learning | `npx flo-learn` | `patterns` | Pattern research on codebase |
-
-These are configured in the `SessionStart` hook in `.claude/settings.json` (set up by `npx flo init`).
-
-Indexing is incremental by default — files whose content hash hasn't changed are skipped. Use `--force` to reindex everything.
-
----
-
-## Troubleshooting
-
-| Symptom | Cause | Fix |
-|---------|-------|-----|
-| Search returns irrelevant results | Query/stored embedding model mismatch | Auto-detected now; verify with `--verbose` flag |
-| Low similarity scores | Query doesn't match domain terms | Include domain keywords in query |
-| "Vector: No" in list | Entry lacks embedding | Run `npx flo-index --force` |
-| Entries not found after adding file | Indexer hasn't run yet | Run `npx flo-index` or restart session |
-| Bundled moflo guidance not indexed | Not installed as dependency | Only indexes when `node_modules/moflo/.claude/guidance/` exists |
-
----
-
-## Verification Commands
-
-```bash
-# Test semantic search
-npx flo memory search --query "database entity pattern" --namespace guidance
-
-# Force reindex all guidance
-npx flo-index --force
-
-# Force rebuild embeddings
-npx flo-index --force
-
-# Check entry count (requires better-sqlite3 or sql.js)
-npx flo memory list --namespace guidance --limit 0
-```
-
----
-
-## See Also
-
-- `.claude/guidance/guidance-memory-strategy.md` - RAG system tuning guide
-- `.claude/guidance/agent-bootstrap.md` - Subagent bootstrap guide
-- `.claude/guidance/moflo.md` - Full CLI/MCP reference
+# Memory & Semantic Search Strategy
+
+**Purpose:** How memory, embeddings, and semantic search work in moflo. Reference when writing guidance documents, debugging search quality, configuring memory for a consumer project, or extending the system.
+
+---
+
+## Problem Statement
+
+Claude Code agents need project-specific knowledge — coding rules, architecture patterns, entity templates, testing conventions — delivered at the right moment. Without a retrieval system, agents either miss critical rules or require massive CLAUDE.md files that waste context window tokens.
+
+**Goals:**
+- Agents find relevant guidance automatically via semantic search
+- Subagents spawned by the coordinator inherit memory access
+- Search quality is high enough that agents don't need to read whole files
+- The system survives `npm install` (indexing runs on session start)
+
+---
+
+## Architecture Overview
+
+```
+Source Files (.claude/guidance/*.md, docs/*.md)
+         |
+         v
+index-guidance.mjs --- Chunk on ## headers, build RAG links
+         |                (prev/next, siblings, parent/child, context overlap)
+         v
+.swarm/memory.db ----- SQLite (entries + metadata + embedding vectors)
+         |
+         v
+build-embeddings.mjs - Generate 384-dim vectors per entry
+         |                (Xenova/all-MiniLM-L6-v2 neural, or domain-aware hash fallback)
+         v
+HNSW index ----------- Approximate nearest-neighbor search
+         v
+Search layer ---------- Three access paths:
+                          1. MCP tools (mcp__moflo__memory_search) -- preferred
+                          2. CLI (npx flo memory search) -- fallback
+                          3. Script (semantic-search.mjs) -- detailed output
+```
+
+**Key files:**
+
+| File | Purpose |
+|------|---------|
+| `.swarm/memory.db` | SQLite database with all entries, embeddings, metadata |
+| `.swarm/code-map-hash.txt` | SHA-256 hash for incremental code map skip |
+| `.claude-flow/neural/patterns.json` | ReasoningBank learned patterns |
+| `bin/build-embeddings.mjs` | Generates 384-dim embeddings |
+| `bin/index-guidance.mjs` | Indexes guidance files with RAG linking |
+| `bin/generate-code-map.mjs` | Generates structural code map (projects, dirs, types, interfaces) |
+| `bin/index-patterns.mjs` | Extracts per-file code patterns |
+| `bin/index-tests.mjs` | Indexes test structure and patterns |
+| `bin/index-all.mjs` | Runs the full indexing chain sequentially |
+
+---
+
+## Namespaces
+
+| Namespace | Content | Indexed By |
+|-----------|---------|------------|
+| `guidance` | Indexed guidance and docs | `index-guidance.mjs` |
+| `code-map` | Structural codebase index (projects, directories, types, interfaces) | `generate-code-map.mjs` |
+| `patterns` | Per-file code patterns (services, routes, error handling, exports) | `index-patterns.mjs` |
+| `tests` | Test structure and patterns | `index-tests.mjs` |
+
+---
+
+## Guidance Document Optimization Rules
+
+These rules determine how well your guidance documents retrieve via semantic search:
+
+### 1. Every file needs a Purpose line
+
+Add `**Purpose:**` as the first meaningful line after the title. Claude checks this first for relevance scoring. Without it, the chunk has no summary signal.
+
+### 2. H2 headings are the primary retrieval signal
+
+The indexer splits on `##`. Each heading becomes the chunk title, prepended to searchable content. Domain-specific keywords in headings dramatically improve recall.
+
+**Bad:** `## Overview`, `## Rules`, `## Pattern`
+**Good:** `## Soft Delete Rules`, `## JWT Authentication Pattern`, `## Database Entity Migration`
+
+### 3. Ideal chunk size: 1000-4000 characters
+
+Below 50 chars the chunk is dropped. Above 6000 the indexer force-splits on paragraphs, which breaks mid-thought. The sweet spot produces focused embeddings.
+
+### 4. Self-contained chunks
+
+Each H2 section must answer a question without needing the rest of the document. Include: the rule, a code example, and a cross-reference.
+
+### 5. Tables over prose
+
+Claude parses structured data more accurately than paragraphs. DO/DON'T tables, field reference tables, and command tables all retrieve better.
+
+### 6. Cross-references create a navigation graph
+
+The RAG indexer stores `prevChunk`/`nextChunk`/`siblings` metadata. Cross-references between documents let Claude follow chains: `core.md -> coding-rules.md -> database.md`.
+
+### 7. No decorative formatting
+
+ASCII boxes, excessive emoji, rhetorical questions, and motivational text all waste tokens without improving retrieval or comprehension.
+
+---
+
+## Embedding Strategy
+
+### Embedding Models
+
+| Model | Quality | Speed | When Used |
+|-------|---------|-------|-----------|
+| `Xenova/all-MiniLM-L6-v2` | High (true semantic) | ~3s for 1000 entries | Primary — `build-embeddings.mjs` uses this |
+| `domain-aware-hash-v1` | Good (domain clustering) | <1s for 1000 entries | Fallback when Transformers.js unavailable |
+
+**Neural embeddings (Xenova/all-MiniLM-L6-v2):**
+- Uses `@xenova/transformers` with ONNX WASM runtime
+- 384-dimensional vectors, L2-normalized
+- True semantic understanding — "soft delete" matches "mark as deleted" without keyword overlap
+- Loaded lazily on first use, cached for subsequent queries
+
+**Domain-aware hash embeddings (fallback):**
+- Custom SimHash-style algorithm with 12 domain clusters
+- Multi-position hashing with bigram/trigram features
+- Good at keyword-level matching but misses semantic paraphrases
+- No external dependencies — always available
+
+### The Embedding Alignment Problem
+
+**Critical rule:** Query embeddings MUST match stored embeddings. Computing cosine similarity between vectors from different models produces meaningless scores.
+
+Both the search scripts and the MCP memory tools auto-detect the stored embedding model and generate matching query vectors. Search also filters out entries with mismatched `embedding_model`.
+
+### Domain Cluster Tuning
+
+The hash fallback's domain clusters can be extended with project-specific terms:
+
+| Cluster | Example Terms |
+|---------|--------------|
+| `database` | your ORM, database engine, schema terms |
+| `frontend` | UI framework, component library terms |
+| `backend` | DI container, API framework terms |
+| `testing` | test framework, assertion library terms |
+| `security` | auth system, permission model terms |
+
+---
+
+## RAG Indexing Pipeline
+
+### How `index-guidance.mjs` Works
+
+1. **Scan** configured directories for `.md` files
+2. **Hash check** — Skip files whose content hash hasn't changed (unless `--force`)
+3. **Store full document** as `doc-{prefix}-{name}` (for complete retrieval)
+4. **Chunk on `##` headers** — Each H2 section becomes a separate entry
+5. **H3 subsections** become child chunks with parent H2 as context prefix
+6. **Force-split** sections over 4000 chars on paragraph boundaries
+7. **Build RAG metadata** for every chunk:
+
+| Metadata Field | Purpose |
+|---------------|---------|
+| `parentDoc` | Link back to full document |
+| `prevChunk` / `nextChunk` | Sequential navigation |
+| `siblings` | All chunk keys from same document |
+| `hierarchicalParent` / `hierarchicalChildren` | H2->H3 relationships |
+| `contextBefore` / `contextAfter` | 20% overlapping text from adjacent chunks |
+
+8. **Prepend context** — Each chunk's searchable content includes overlap from neighbors
+9. **Stale cleanup** — After indexing, remove entries for files that no longer exist on disk
+10. **Background embedding** — Spawn `build-embeddings.mjs` to generate vectors
+
+### Configuring Indexed Directories
+
+In `moflo.yaml`:
+
+```yaml
+guidance:
+  directories:
+    - .claude/guidance
+    - docs/guides
+```
+
+Default directories (when no config): `.claude/guidance`, `docs/guides`
+
+Moflo also automatically indexes its own bundled guidance from `node_modules/moflo/.claude/guidance/` when installed as a dependency.
+
+---
+
+## Search Commands
+
+All methods auto-detect the stored embedding model and generate matching query vectors:
+
+**MCP (Preferred):** `mcp__moflo__memory_search` — `query: "your query", namespace: "guidance"`
+
+**CLI (Fallback):**
+```bash
+npx flo memory search --query "your query" --namespace guidance
+```
+
+**Search options:**
+
+| Flag | Default | Purpose |
+|------|---------|---------|
+| `--namespace` | all | Filter to specific namespace |
+| `--limit` | 5 | Number of results |
+| `--threshold` | 0.3 | Minimum similarity score |
+| `--json` | false | Output as JSON |
+
+### Code Map Search (for codebase navigation)
+
+When you need to find where a type, service, entity, or component lives — search `code-map` BEFORE using Glob/Grep:
+
+**MCP:** `mcp__moflo__memory_search` — `query: "payment service", namespace: "code-map"`
+
+**What code-map contains:**
+
+| Chunk prefix | What it answers |
+|--------------|-----------------|
+| `project:` | "What's in the api project?" |
+| `dir:` | "What types are in the entities directory?" |
+| `iface-map:` | "What implements IPaymentService?" |
+| `type-index:` | "Where is Service defined?" |
+
+---
+
+## Session Start Indexing
+
+On every session start, `hooks.mjs` spawns `index-all.mjs` which runs the full chain:
+
+| Indexer | Namespace | What it does |
+|---------|-----------|--------------|
+| `index-guidance.mjs` | `guidance` | Chunks markdown, builds RAG links |
+| `generate-code-map.mjs` | `code-map` | Scans source for types, interfaces, directories |
+| `index-tests.mjs` | `tests` | Indexes test structure |
+| `index-patterns.mjs` | `patterns` | Extracts per-file code patterns |
+| `build-embeddings.mjs` | all | Generates vectors for any unembedded entries |
+
+Indexing is incremental by default — files whose content hash hasn't changed are skipped. Use `--force` to reindex everything.
+
+---
+
+## Replication Guide
+
+To set up this system in a new project:
+
+```bash
+npm install moflo
+npx flo init
+```
+
+Create `.claude/guidance/` with markdown files following the optimization rules above, then:
+
+```bash
+npx flo-index --force                                              # Index documents
+npx flo memory search --query "your domain query" --namespace guidance  # Verify
+```
+
+---
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Search returns irrelevant results | Query/stored embedding model mismatch | Auto-detected now; verify with `--verbose` flag |
+| Low similarity scores | Query doesn't match domain terms | Include domain keywords in query |
+| "Vector: No" in list | Entry lacks embedding | Run `node bin/build-embeddings.mjs` |
+| Entries not found after adding file | Indexer hasn't run yet | Run `node bin/index-all.mjs` or restart session |
+| Bundled moflo guidance not indexed | Not installed as dependency | Only indexes when `node_modules/moflo/.claude/guidance/` exists |
+| Empty namespace | Indexer never ran or DB was purged | See `memorydb-maintenance.md` for reindex/purge procedures |
+
+---
+
+## See Also
+
+- `memorydb-maintenance.md` — Database location, schema, purge/reindex procedures
+- `agent-bootstrap.md` — Subagent bootstrap guide
+- `moflo.md` — Full CLI/MCP reference

package/.claude/guidance/shipped/memorydb-maintenance.md

@@ -0,0 +1,111 @@
+# Memory Database Maintenance
+
+## Database Location
+
+- **Path:** `.swarm/memory.db`
+- **Engine:** sql.js (WASM-based SQLite — no native binaries needed)
+- **Single table:** `memory_entries` — stores content, embeddings, and metadata in one table
+
+## Schema
+
+```sql
+CREATE TABLE memory_entries (
+  id TEXT PRIMARY KEY,
+  key TEXT NOT NULL,
+  namespace TEXT DEFAULT 'default',
+  content TEXT NOT NULL,
+  type TEXT DEFAULT 'semantic',
+  embedding TEXT,           -- inline vector (no separate embeddings table)
+  embedding_model TEXT DEFAULT 'local',
+  embedding_dimensions INTEGER,
+  tags TEXT,
+  metadata TEXT,
+  owner_id TEXT,
+  created_at INTEGER,
+  updated_at INTEGER,
+  expires_at INTEGER,
+  last_accessed_at INTEGER,
+  access_count INTEGER DEFAULT 0,
+  status TEXT DEFAULT 'active',
+  UNIQUE(namespace, key)
+);
+```
+
+## Namespaces
+
+| Namespace | Indexed By | Key Patterns | Purpose |
+|-----------|-----------|--------------|---------|
+| `code-map` | `generate-code-map.mjs` | `file:{path}`, `dir:{path}`, `type-index:{n}` | Codebase navigation |
+| `guidance` | `index-guidance.mjs` | `chunk-guidance-*` | Governance docs, shipped guidance |
+| `patterns` | `index-patterns.mjs` | `pattern:file:{path}`, `pattern:service:{name}`, `pattern:route:{path}`, `pattern:error:{path}` | Per-file code patterns |
+| `tests` | `index-tests.mjs` | test-related keys | Test structure and patterns |
+
+## MCP Tools
+
+| Tool | Use |
+|------|-----|
+| `mcp__moflo__memory_search` | Semantic search (HNSW-accelerated) |
+| `mcp__moflo__memory_store` | Store/upsert entries |
+| `mcp__moflo__memory_delete` | Delete single entry by key |
+| `mcp__moflo__memory_stats` | Namespace counts and HNSW index status |
+| `mcp__moflo__memory_list` | List keys in a namespace |
+
+## Reindex Commands
+
+```bash
+# Full reindex (guidance → code-map → tests → patterns → pretrain → HNSW rebuild)
+node bin/index-all.mjs
+
+# Individual indexers
+node bin/index-guidance.mjs
+node bin/generate-code-map.mjs
+node bin/index-tests.mjs
+node bin/index-patterns.mjs          # --force for full reindex, --stats for counts
+
+# Rebuild embeddings (vectorize all entries)
+node bin/build-embeddings.mjs
+```
+
+## Purging a Namespace
+
+Use sql.js directly (no sqlite3 binary on this machine):
+
+```js
+node --input-type=module -e "
+import initSqlJs from 'sql.js';
+import { readFileSync, writeFileSync } from 'fs';
+
+const SQL = await initSqlJs();
+const buf = readFileSync('.swarm/memory.db');
+const db = new SQL.Database(buf);
+
+// Check counts before
+const before = db.exec('SELECT namespace, COUNT(*) FROM memory_entries GROUP BY namespace');
+console.log('BEFORE:', JSON.stringify(before[0]?.values));
+
+// Purge a namespace (embeddings are inline — no separate table to clean)
+db.run(\"DELETE FROM memory_entries WHERE namespace = 'code-map'\");
+
+const after = db.exec('SELECT namespace, COUNT(*) FROM memory_entries GROUP BY namespace');
+console.log('AFTER:', JSON.stringify(after[0]?.values));
+
+writeFileSync('.swarm/memory.db', Buffer.from(db.export()));
+db.close();
+"
+```
+
+After purging, reindex the namespace and rebuild embeddings:
+```bash
+node bin/generate-code-map.mjs   # or whichever indexer owns the namespace
+node bin/build-embeddings.mjs
+```
+
+## Auto-Reindex on Session Start
+
+`hooks.mjs` spawns `index-all.mjs` on session start, which runs the full chain. This only works if `index-all.mjs` is in the `scriptFiles` array in `session-start-launcher.mjs` (see docs/BUILD.md § "Adding New Scripts").
+
+## Troubleshooting
+
+- **Empty search results:** Run `mcp__moflo__memory_stats` to check entry counts. If a namespace is empty, run its indexer.
+- **Stale embeddings:** Run `node bin/build-embeddings.mjs` — it skips entries that already have embeddings unless content changed.
+- **Full reset:** Delete `.swarm/memory.db` and run `node bin/index-all.mjs && node bin/build-embeddings.mjs`.

package/.claude/scripts/session-start-launcher.mjs

@@ -8,7 +8,7 @@
  */
 
 import { spawn } from 'child_process';
-import { existsSync, readFileSync, copyFileSync, unlinkSync, readdirSync } from 'fs';
+import { existsSync, readFileSync, writeFileSync, copyFileSync, unlinkSync, readdirSync, mkdirSync } from 'fs';
 import { resolve, dirname } from 'path';
 import { fileURLToPath } from 'url';
 
@@ -32,7 +32,6 @@ function fireAndForget(cmd, args, label) {
 }
 
 // ── 2. Reset workflow state for new session ──────────────────────────────────
-import { writeFileSync, mkdirSync } from 'fs';
 const stateDir = resolve(projectRoot, '.claude');
 const stateFile = resolve(stateDir, 'workflow-state.json');
 try {
@@ -112,7 +111,7 @@ try {
         const scriptFiles = [
           'hooks.mjs', 'session-start-launcher.mjs', 'index-guidance.mjs',
           'build-embeddings.mjs', 'generate-code-map.mjs', 'semantic-search.mjs',
-          'index-tests.mjs',
+          'index-tests.mjs', 'index-all.mjs',
         ];
         for (const file of scriptFiles) {
           syncFile(resolve(binDir, file), resolve(scriptsDir, file), `.claude/scripts/${file}`);
@@ -164,19 +163,21 @@ try {
         }
       }
 
-      // Sync guidance bootstrap file (moflo-bootstrap.md)
-      const shippedBootstrap = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped/agent-bootstrap.md');
-      const legacyBootstrap = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/agent-bootstrap.md');
-      const bootstrapSrc = existsSync(shippedBootstrap) ? shippedBootstrap : legacyBootstrap;
+      // Sync all shipped guidance files from node_modules/moflo/.claude/guidance/shipped/
       const guidanceDir = resolve(projectRoot, '.claude/guidance');
-      const bootstrapDest = resolve(guidanceDir, 'moflo-bootstrap.md');
-      if (existsSync(bootstrapSrc)) {
+      const shippedDir = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped');
+      if (existsSync(shippedDir)) {
         try {
           if (!existsSync(guidanceDir)) mkdirSync(guidanceDir, { recursive: true });
-          const header = '<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/agent-bootstrap.md -->\n\n';
-          const content = readFileSync(bootstrapSrc, 'utf-8');
-          writeFileSync(bootstrapDest, header + content);
-          currentManifest.push('.claude/guidance/moflo-bootstrap.md');
+          const shippedFiles = readdirSync(shippedDir).filter(f => f.endsWith('.md'));
+          for (const file of shippedFiles) {
+            const src = resolve(shippedDir, file);
+            const dest = resolve(guidanceDir, `moflo-${file}`);
+            const header = `<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/shipped/${file} -->\n\n`;
+            const content = readFileSync(src, 'utf-8');
+            writeFileSync(dest, header + content);
+            currentManifest.push(`.claude/guidance/moflo-${file}`);
+          }
         } catch { /* non-fatal */ }
       }
 
@@ -206,19 +207,22 @@ try {
   // Non-fatal — scripts will still work, just may be stale
 }
 
-// ── 3b. Ensure guidance bootstrap file exists (even without version change) ──
-// Subagents need this file on disk for direct reads without memory search.
+// ── 3b. Ensure shipped guidance files exist (even without version change) ──
+// Subagents need these files on disk for direct reads without memory search.
 try {
-  const shippedBs = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped/agent-bootstrap.md');
-  const legacyBs = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/agent-bootstrap.md');
-  const bootstrapSrc = existsSync(shippedBs) ? shippedBs : legacyBs;
   const guidanceDir = resolve(projectRoot, '.claude/guidance');
-  const bootstrapDest = resolve(guidanceDir, 'moflo-bootstrap.md');
-  if (existsSync(bootstrapSrc) && !existsSync(bootstrapDest)) {
-    if (!existsSync(guidanceDir)) mkdirSync(guidanceDir, { recursive: true });
-    const header = '<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/agent-bootstrap.md -->\n\n';
-    const content = readFileSync(bootstrapSrc, 'utf-8');
-    writeFileSync(bootstrapDest, header + content);
+  const shippedDir = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped');
+  if (existsSync(shippedDir)) {
+    const shippedFiles = readdirSync(shippedDir).filter(f => f.endsWith('.md'));
+    for (const file of shippedFiles) {
+      const dest = resolve(guidanceDir, `moflo-${file}`);
+      if (!existsSync(dest)) {
+        if (!existsSync(guidanceDir)) mkdirSync(guidanceDir, { recursive: true });
+        const header = `<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/shipped/${file} -->\n\n`;
+        const content = readFileSync(resolve(shippedDir, file), 'utf-8');
+        writeFileSync(dest, header + content);
+      }
+    }
   }
 } catch { /* non-fatal */ }
 

package/bin/session-start-launcher.mjs

@@ -8,7 +8,7 @@
  */
 
 import { spawn } from 'child_process';
-import { existsSync, readFileSync, copyFileSync, unlinkSync, readdirSync } from 'fs';
+import { existsSync, readFileSync, writeFileSync, copyFileSync, unlinkSync, readdirSync, mkdirSync } from 'fs';
 import { resolve, dirname } from 'path';
 import { fileURLToPath } from 'url';
 
@@ -32,7 +32,6 @@ function fireAndForget(cmd, args, label) {
 }
 
 // ── 2. Reset workflow state for new session ──────────────────────────────────
-import { writeFileSync, mkdirSync } from 'fs';
 const stateDir = resolve(projectRoot, '.claude');
 const stateFile = resolve(stateDir, 'workflow-state.json');
 try {
@@ -112,7 +111,7 @@ try {
         const scriptFiles = [
           'hooks.mjs', 'session-start-launcher.mjs', 'index-guidance.mjs',
           'build-embeddings.mjs', 'generate-code-map.mjs', 'semantic-search.mjs',
-          'index-tests.mjs',
+          'index-tests.mjs', 'index-all.mjs',
         ];
         for (const file of scriptFiles) {
           syncFile(resolve(binDir, file), resolve(scriptsDir, file), `.claude/scripts/${file}`);
@@ -164,19 +163,21 @@ try {
         }
       }
 
-      // Sync guidance bootstrap file (moflo-bootstrap.md)
-      const shippedBootstrap = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped/agent-bootstrap.md');
-      const legacyBootstrap = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/agent-bootstrap.md');
-      const bootstrapSrc = existsSync(shippedBootstrap) ? shippedBootstrap : legacyBootstrap;
+      // Sync all shipped guidance files from node_modules/moflo/.claude/guidance/shipped/
       const guidanceDir = resolve(projectRoot, '.claude/guidance');
-      const bootstrapDest = resolve(guidanceDir, 'moflo-bootstrap.md');
-      if (existsSync(bootstrapSrc)) {
+      const shippedDir = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped');
+      if (existsSync(shippedDir)) {
         try {
           if (!existsSync(guidanceDir)) mkdirSync(guidanceDir, { recursive: true });
-          const header = '<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/agent-bootstrap.md -->\n\n';
-          const content = readFileSync(bootstrapSrc, 'utf-8');
-          writeFileSync(bootstrapDest, header + content);
-          currentManifest.push('.claude/guidance/moflo-bootstrap.md');
+          const shippedFiles = readdirSync(shippedDir).filter(f => f.endsWith('.md'));
+          for (const file of shippedFiles) {
+            const src = resolve(shippedDir, file);
+            const dest = resolve(guidanceDir, `moflo-${file}`);
+            const header = `<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/shipped/${file} -->\n\n`;
+            const content = readFileSync(src, 'utf-8');
+            writeFileSync(dest, header + content);
+            currentManifest.push(`.claude/guidance/moflo-${file}`);
+          }
         } catch { /* non-fatal */ }
       }
 
@@ -206,19 +207,22 @@ try {
   // Non-fatal — scripts will still work, just may be stale
 }
 
-// ── 3b. Ensure guidance bootstrap file exists (even without version change) ──
-// Subagents need this file on disk for direct reads without memory search.
+// ── 3b. Ensure shipped guidance files exist (even without version change) ──
+// Subagents need these files on disk for direct reads without memory search.
 try {
-  const shippedBs = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped/agent-bootstrap.md');
-  const legacyBs = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/agent-bootstrap.md');
-  const bootstrapSrc = existsSync(shippedBs) ? shippedBs : legacyBs;
   const guidanceDir = resolve(projectRoot, '.claude/guidance');
-  const bootstrapDest = resolve(guidanceDir, 'moflo-bootstrap.md');
-  if (existsSync(bootstrapSrc) && !existsSync(bootstrapDest)) {
-    if (!existsSync(guidanceDir)) mkdirSync(guidanceDir, { recursive: true });
-    const header = '<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/agent-bootstrap.md -->\n\n';
-    const content = readFileSync(bootstrapSrc, 'utf-8');
-    writeFileSync(bootstrapDest, header + content);
+  const shippedDir = resolve(projectRoot, 'node_modules/moflo/.claude/guidance/shipped');
+  if (existsSync(shippedDir)) {
+    const shippedFiles = readdirSync(shippedDir).filter(f => f.endsWith('.md'));
+    for (const file of shippedFiles) {
+      const dest = resolve(guidanceDir, `moflo-${file}`);
+      if (!existsSync(dest)) {
+        if (!existsSync(guidanceDir)) mkdirSync(guidanceDir, { recursive: true });
+        const header = `<!-- AUTO-GENERATED by moflo session-start. Do not edit — changes will be overwritten. -->\n<!-- Source: node_modules/moflo/.claude/guidance/shipped/${file} -->\n\n`;
+        const content = readFileSync(resolve(shippedDir, file), 'utf-8');
+        writeFileSync(dest, header + content);
+      }
+    }
   }
 } catch { /* non-fatal */ }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "moflo",
-  "version": "4.8.38",
+  "version": "4.8.40",
   "description": "MoFlo — AI agent orchestration for Claude Code. Forked from ruflo/claude-flow with patches applied to source, plus feature-level orchestration.",
   "main": "dist/index.js",
   "type": "module",
@@ -89,7 +89,7 @@
     "@types/bcrypt": "^5.0.2",
     "@types/node": "^20.19.37",
     "eslint": "^8.0.0",
-    "moflo": "^4.8.22",
+    "moflo": "^4.8.39",
     "tsx": "^4.21.0",
     "typescript": "^5.9.3",
     "vitest": "^4.0.0"

package/src/@claude-flow/cli/dist/src/version.js

@@ -2,5 +2,5 @@
  * Auto-generated by build. Do not edit manually.
  * Source of truth: root package.json → scripts/sync-version.mjs
  */
-export const VERSION = '4.8.38';
+export const VERSION = '4.8.40';
 //# sourceMappingURL=version.js.map

package/src/@claude-flow/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@moflo/cli",
-  "version": "4.8.38",
+  "version": "4.8.40",
   "type": "module",
   "description": "MoFlo CLI — AI agent orchestration with specialized agents, swarm coordination, MCP server, self-learning hooks, and vector memory for Claude Code",
   "main": "dist/src/index.js",

package/.claude/guidance/shipped/guidance-memory-strategy.md

@@ -1,262 +0,0 @@
-# Guidance & Memory Tuning Strategy
-
-**Purpose:** How to build and tune a RAG-based guidance system using moflo's semantic search, embedding pipeline, and indexing. Reference when creating guidance documents, troubleshooting search quality, or extending the system.
-
----
-
-## Problem Statement
-
-Claude Code agents need project-specific knowledge — coding rules, architecture patterns, entity templates, testing conventions — delivered at the right moment. Without a retrieval system, agents either miss critical rules or require massive CLAUDE.md files that waste context window tokens.
-
-**Goals:**
-- Agents find relevant guidance automatically via semantic search
-- Subagents spawned by the coordinator inherit memory access
-- Search quality is high enough that agents don't need to read whole files
-- The system survives `npm install` (indexing runs on session start)
-
----
-
-## Architecture
-
-Three layers: embedding generation, vector storage, and search.
-
-```
-Source Files (.claude/guidance/*.md, docs/*.md)
-         |
-         v
-index-guidance.mjs --- Chunk on ## headers, build RAG links
-         |                (prev/next, siblings, parent/child, context overlap)
-         v
-.swarm/memory.db ----- SQLite (entries + metadata + embedding vectors)
-         |
-         v
-build-embeddings.mjs - Generate 384-dim vectors per entry
-         |                (Xenova/all-MiniLM-L6-v2 neural, or domain-aware hash fallback)
-         v
-RuVector (@ruvector/core) -- HNSW index infrastructure
-         v
-Search layer ---------- Three access paths:
-                          1. MCP tools (mcp__moflo__memory_search) -- preferred
-                          2. CLI (npx flo memory search) -- fallback
-                          3. Script (semantic-search.mjs) -- detailed output
-```
-
-**Key files:**
-
-| File | Role |
-|------|------|
-| `.claude/guidance/*.md` | Guidance documents (source of truth) |
-| `bin/index-guidance.mjs` | Chunks documents, stores in SQLite with RAG metadata |
-| `bin/build-embeddings.mjs` | Generates vector embeddings (neural or hash) |
-| `.swarm/memory.db` | SQLite database with entries, metadata, embeddings |
-| `@ruvector/core` | HNSW vector index, WASM fallback, SIMD operations |
-
----
-
-## Guidance Document Optimization Rules
-
-These rules determine how well your guidance documents retrieve via semantic search:
-
-### 1. Every file needs a Purpose line
-
-Add `**Purpose:**` as the first meaningful line after the title. Claude checks this first for relevance scoring. Without it, the chunk has no summary signal.
-
-### 2. H2 headings are the primary retrieval signal
-
-The indexer splits on `##`. Each heading becomes the chunk title, prepended to searchable content. Domain-specific keywords in headings dramatically improve recall.
-
-**Bad:** `## Overview`, `## Rules`, `## Pattern`
-**Good:** `## Soft Delete Rules`, `## JWT Authentication Pattern`, `## Database Entity Migration`
-
-### 3. Ideal chunk size: 1000-4000 characters
-
-Below 50 chars the chunk is dropped. Above 6000 the indexer force-splits on paragraphs, which breaks mid-thought. The sweet spot produces focused embeddings.
-
-### 4. Self-contained chunks
-
-Each H2 section must answer a question without needing the rest of the document. Include: the rule, a code example, and a cross-reference.
-
-### 5. Tables over prose
-
-Claude parses structured data more accurately than paragraphs. DO/DON'T tables, field reference tables, and command tables all retrieve better.
-
-### 6. Cross-references create a navigation graph
-
-The RAG indexer stores `prevChunk`/`nextChunk`/`siblings` metadata. Cross-references between documents let Claude follow chains: `core.md -> coding-rules.md -> database.md`.
-
-### 7. No decorative formatting
-
-ASCII boxes, excessive emoji, rhetorical questions, and motivational text all waste tokens without improving retrieval or comprehension.
-
----
-
-## Embedding Pipeline
-
-### Embedding Models
-
-| Model | Quality | Speed | When Used |
-|-------|---------|-------|-----------|
-| `Xenova/all-MiniLM-L6-v2` | High (true semantic) | ~3s for 1000 entries | Primary — `build-embeddings.mjs` uses this |
-| `domain-aware-hash-v1` | Good (domain clustering) | <1s for 1000 entries | Fallback when Transformers.js unavailable |
-
-**Neural embeddings (Xenova/all-MiniLM-L6-v2):**
-- Uses `@xenova/transformers` with ONNX WASM runtime
-- 384-dimensional vectors, L2-normalized
-- True semantic understanding — "soft delete" matches "mark as deleted" without keyword overlap
-- Loaded lazily on first use, cached for subsequent queries
-- Ships with moflo; no additional install needed
-
-**Domain-aware hash embeddings (fallback):**
-- Custom SimHash-style algorithm with 12 domain clusters
-- Domain clusters group related terms: `database` (orm, postgresql, entity, schema...), `frontend` (react, component, css...), `testing` (vitest, mock, expect...), etc.
-- Multi-position hashing with bigram/trigram features
-- Good at keyword-level matching but misses semantic paraphrases
-- No external dependencies — always available
-
-### The Embedding Alignment Problem
-
-**Critical rule:** Query embeddings MUST match stored embeddings. Computing cosine similarity between vectors from different models produces meaningless scores.
-
-Both the search scripts and the MCP memory tools auto-detect the stored embedding model:
-
-```javascript
-// Check what model stored entries predominantly use
-const modelCheck = db.prepare(
-  `SELECT embedding_model, COUNT(*) as cnt FROM memory_entries
-   WHERE status = 'active' AND embedding IS NOT NULL
-   GROUP BY embedding_model ORDER BY cnt DESC LIMIT 1`
-).get();
-
-// If stored embeddings are neural, use neural for query too
-```
-
-Search also **filters out entries with mismatched `embedding_model`** — if the query uses neural embeddings, hash-embedded entries are skipped (and vice versa).
-
-### Domain Cluster Tuning
-
-The hash fallback's domain clusters can be extended with project-specific terms. Add terms to the relevant cluster in the hash embedding function to improve keyword-level matching for your domain:
-
-| Cluster | Example Terms |
-|---------|--------------|
-| `database` | your ORM, database engine, schema terms |
-| `frontend` | UI framework, component library terms |
-| `backend` | DI container, API framework terms |
-| `testing` | test framework, assertion library terms |
-| `security` | auth system, permission model terms |
-
----
-
-## RAG Indexing Pipeline
-
-### How `index-guidance.mjs` Works
-
-1. **Scan** configured directories for `.md` files
-2. **Hash check** — Skip files whose content hash hasn't changed (unless `--force`)
-3. **Store full document** as `doc-{prefix}-{name}` (for complete retrieval)
-4. **Chunk on `##` headers** — Each H2 section becomes a separate entry
-5. **H3 subsections** become child chunks with parent H2 as context prefix
-6. **Force-split** sections over 4000 chars on paragraph boundaries
-7. **Build RAG metadata** for every chunk:
-
-| Metadata Field | Purpose |
-|---------------|---------|
-| `parentDoc` | Link back to full document |
-| `prevChunk` / `nextChunk` | Sequential navigation |
-| `siblings` | All chunk keys from same document |
-| `hierarchicalParent` / `hierarchicalChildren` | H2->H3 relationships |
-| `contextBefore` / `contextAfter` | 20% overlapping text from adjacent chunks |
-
-8. **Prepend context** — Each chunk's searchable content includes overlap from neighbors
-9. **Stale cleanup** — After indexing, remove entries for files that no longer exist on disk
-10. **Background embedding** — Spawn `build-embeddings.mjs` in background to generate vectors
-
-### Configuring Indexed Directories
-
-In `moflo.yaml`:
-
-```yaml
-guidance:
-  directories:
-    - .claude/guidance
-    - docs/guides
-```
-
-Default directories (when no config): `.claude/guidance`, `docs/guides`
-
-Moflo also automatically indexes its own bundled guidance from `node_modules/moflo/.claude/guidance/` when installed as a library in a consumer project.
-
----
-
-## Lessons Learned
-
-### Document Optimization
-
-1. **`**Purpose:**` lines are critical** — They're the single highest-impact addition for retrieval quality.
-2. **Headings are embeddings** — In a chunk-per-section system, the heading IS the embedding's primary signal. Generic headings are nearly useless.
-3. **Tables retrieve better than prose** — Claude parses structured data with higher accuracy.
-4. **Cross-references are the RAG graph** — Isolated documents can't be navigated.
-5. **Chunk size matters** — A 10,000-char section produces a diluted embedding. Splitting into focused sections triples the chance of matching specific queries.
-
-### Embedding Pipeline
-
-6. **Query embeddings MUST match stored embeddings** — This is the single most critical rule. Auto-detect and match.
-7. **Domain clusters need project-specific terms** — Generic NLP clusters miss project-specific terminology. Adding terms to domain clusters dramatically improves keyword-level matching.
-8. **Filter mismatched entries during search** — Mixed databases need explicit filtering by `embedding_model`.
-
----
-
-## Replication Guide
-
-To set up this system in a new project using moflo:
-
-### 1. Install Moflo
-
-```bash
-npm install moflo
-npx flo init
-```
-
-### 2. Create Guidance Documents
-
-Create `.claude/guidance/` directory with markdown files following the optimization rules above:
-- Every file has `**Purpose:**` line
-- H2 sections with domain keywords in headings
-- Tables for structured rules
-- Cross-references between related docs
-- 1000-4000 char sections
-
-### 3. Configure Indexing
-
-In `moflo.yaml`:
-
-```yaml
-guidance:
-  directories:
-    - .claude/guidance
-    - docs/guides
-
-auto_index:
-  guidance: true
-  code_map: true
-```
-
-### 4. Index and Verify
-
-```bash
-# Index documents
-npx flo-index --force
-
-# Test search quality
-npx flo memory search --query "your domain query" --namespace guidance
-
-# Verify from Claude Code via MCP
-# mcp__moflo__memory_search query="your domain query" namespace="guidance"
-```
-
----
-
-## See Also
-
-- `.claude/guidance/memory-strategy.md` - Memory architecture and search commands
-- `.claude/guidance/agent-bootstrap.md` - Subagent bootstrap guide
-- `.claude/guidance/moflo.md` - Full CLI/MCP reference