moflo 4.8.21 → 4.8.23

package/.claude/guidance/shipped/guidance-memory-strategy.md

@@ -1,262 +1,262 @@
-# Guidance & Memory Tuning Strategy
-
-**Purpose:** How to build and tune a RAG-based guidance system using moflo's semantic search, embedding pipeline, and indexing. Reference when creating guidance documents, troubleshooting search quality, or extending the system.
-
----
-
-## Problem Statement
-
-Claude Code agents need project-specific knowledge — coding rules, architecture patterns, entity templates, testing conventions — delivered at the right moment. Without a retrieval system, agents either miss critical rules or require massive CLAUDE.md files that waste context window tokens.
-
-**Goals:**
-- Agents find relevant guidance automatically via semantic search
-- Subagents spawned by the coordinator inherit memory access
-- Search quality is high enough that agents don't need to read whole files
-- The system survives `npm install` (indexing runs on session start)
-
----
-
-## Architecture
-
-Three layers: embedding generation, vector storage, and search.
-
-```
-Source Files (.claude/guidance/*.md, docs/*.md)
-         |
-         v
-index-guidance.mjs --- Chunk on ## headers, build RAG links
-         |                (prev/next, siblings, parent/child, context overlap)
-         v
-.swarm/memory.db ----- SQLite (entries + metadata + embedding vectors)
-         |
-         v
-build-embeddings.mjs - Generate 384-dim vectors per entry
-         |                (Xenova/all-MiniLM-L6-v2 neural, or domain-aware hash fallback)
-         v
-RuVector (@ruvector/core) -- HNSW index infrastructure
-         v
-Search layer ---------- Three access paths:
-                          1. MCP tools (mcp__moflo__memory_search) -- preferred
-                          2. CLI (npx flo memory search) -- fallback
-                          3. Script (semantic-search.mjs) -- detailed output
-```
-
-**Key files:**
-
-| File | Role |
-|------|------|
-| `.claude/guidance/*.md` | Guidance documents (source of truth) |
-| `bin/index-guidance.mjs` | Chunks documents, stores in SQLite with RAG metadata |
-| `bin/build-embeddings.mjs` | Generates vector embeddings (neural or hash) |
-| `.swarm/memory.db` | SQLite database with entries, metadata, embeddings |
-| `@ruvector/core` | HNSW vector index, WASM fallback, SIMD operations |
-
----
-
-## Guidance Document Optimization Rules
-
-These rules determine how well your guidance documents retrieve via semantic search:
-
-### 1. Every file needs a Purpose line
-
-Add `**Purpose:**` as the first meaningful line after the title. Claude checks this first for relevance scoring. Without it, the chunk has no summary signal.
-
-### 2. H2 headings are the primary retrieval signal
-
-The indexer splits on `##`. Each heading becomes the chunk title, prepended to searchable content. Domain-specific keywords in headings dramatically improve recall.
-
-**Bad:** `## Overview`, `## Rules`, `## Pattern`
-**Good:** `## Soft Delete Rules`, `## JWT Authentication Pattern`, `## Database Entity Migration`
-
-### 3. Ideal chunk size: 1000-4000 characters
-
-Below 50 chars the chunk is dropped. Above 6000 the indexer force-splits on paragraphs, which breaks mid-thought. The sweet spot produces focused embeddings.
-
-### 4. Self-contained chunks
-
-Each H2 section must answer a question without needing the rest of the document. Include: the rule, a code example, and a cross-reference.
-
-### 5. Tables over prose
-
-Claude parses structured data more accurately than paragraphs. DO/DON'T tables, field reference tables, and command tables all retrieve better.
-
-### 6. Cross-references create a navigation graph
-
-The RAG indexer stores `prevChunk`/`nextChunk`/`siblings` metadata. Cross-references between documents let Claude follow chains: `core.md -> coding-rules.md -> database.md`.
-
-### 7. No decorative formatting
-
-ASCII boxes, excessive emoji, rhetorical questions, and motivational text all waste tokens without improving retrieval or comprehension.
-
----
-
-## Embedding Pipeline
-
-### Embedding Models
-
-| Model | Quality | Speed | When Used |
-|-------|---------|-------|-----------|
-| `Xenova/all-MiniLM-L6-v2` | High (true semantic) | ~3s for 1000 entries | Primary — `build-embeddings.mjs` uses this |
-| `domain-aware-hash-v1` | Good (domain clustering) | <1s for 1000 entries | Fallback when Transformers.js unavailable |
-
-**Neural embeddings (Xenova/all-MiniLM-L6-v2):**
-- Uses `@xenova/transformers` with ONNX WASM runtime
-- 384-dimensional vectors, L2-normalized
-- True semantic understanding — "soft delete" matches "mark as deleted" without keyword overlap
-- Loaded lazily on first use, cached for subsequent queries
-- Ships with moflo; no additional install needed
-
-**Domain-aware hash embeddings (fallback):**
-- Custom SimHash-style algorithm with 12 domain clusters
-- Domain clusters group related terms: `database` (orm, postgresql, entity, schema...), `frontend` (react, component, css...), `testing` (vitest, mock, expect...), etc.
-- Multi-position hashing with bigram/trigram features
-- Good at keyword-level matching but misses semantic paraphrases
-- No external dependencies — always available
-
-### The Embedding Alignment Problem
-
-**Critical rule:** Query embeddings MUST match stored embeddings. Computing cosine similarity between vectors from different models produces meaningless scores.
-
-Both the search scripts and the MCP memory tools auto-detect the stored embedding model:
-
-```javascript
-// Check what model stored entries predominantly use
-const modelCheck = db.prepare(
-  `SELECT embedding_model, COUNT(*) as cnt FROM memory_entries
-   WHERE status = 'active' AND embedding IS NOT NULL
-   GROUP BY embedding_model ORDER BY cnt DESC LIMIT 1`
-).get();
-
-// If stored embeddings are neural, use neural for query too
-```
-
-Search also **filters out entries with mismatched `embedding_model`** — if the query uses neural embeddings, hash-embedded entries are skipped (and vice versa).
-
-### Domain Cluster Tuning
-
-The hash fallback's domain clusters can be extended with project-specific terms. Add terms to the relevant cluster in the hash embedding function to improve keyword-level matching for your domain:
-
-| Cluster | Example Terms |
-|---------|--------------|
-| `database` | your ORM, database engine, schema terms |
-| `frontend` | UI framework, component library terms |
-| `backend` | DI container, API framework terms |
-| `testing` | test framework, assertion library terms |
-| `security` | auth system, permission model terms |
-
----
-
-## RAG Indexing Pipeline
-
-### How `index-guidance.mjs` Works
-
-1. **Scan** configured directories for `.md` files
-2. **Hash check** — Skip files whose content hash hasn't changed (unless `--force`)
-3. **Store full document** as `doc-{prefix}-{name}` (for complete retrieval)
-4. **Chunk on `##` headers** — Each H2 section becomes a separate entry
-5. **H3 subsections** become child chunks with parent H2 as context prefix
-6. **Force-split** sections over 4000 chars on paragraph boundaries
-7. **Build RAG metadata** for every chunk:
-
-| Metadata Field | Purpose |
-|---------------|---------|
-| `parentDoc` | Link back to full document |
-| `prevChunk` / `nextChunk` | Sequential navigation |
-| `siblings` | All chunk keys from same document |
-| `hierarchicalParent` / `hierarchicalChildren` | H2->H3 relationships |
-| `contextBefore` / `contextAfter` | 20% overlapping text from adjacent chunks |
-
-8. **Prepend context** — Each chunk's searchable content includes overlap from neighbors
-9. **Stale cleanup** — After indexing, remove entries for files that no longer exist on disk
-10. **Background embedding** — Spawn `build-embeddings.mjs` in background to generate vectors
-
-### Configuring Indexed Directories
-
-In `moflo.yaml`:
-
-```yaml
-guidance:
-  directories:
-    - .claude/guidance
-    - docs/guides
-```
-
-Default directories (when no config): `.claude/guidance`, `docs/guides`
-
-Moflo also automatically indexes its own bundled guidance from `node_modules/moflo/.claude/guidance/` when installed as a library in a consumer project.
-
----
-
-## Lessons Learned
-
-### Document Optimization
-
-1. **`**Purpose:**` lines are critical** — They're the single highest-impact addition for retrieval quality.
-2. **Headings are embeddings** — In a chunk-per-section system, the heading IS the embedding's primary signal. Generic headings are nearly useless.
-3. **Tables retrieve better than prose** — Claude parses structured data with higher accuracy.
-4. **Cross-references are the RAG graph** — Isolated documents can't be navigated.
-5. **Chunk size matters** — A 10,000-char section produces a diluted embedding. Splitting into focused sections triples the chance of matching specific queries.
-
-### Embedding Pipeline
-
-6. **Query embeddings MUST match stored embeddings** — This is the single most critical rule. Auto-detect and match.
-7. **Domain clusters need project-specific terms** — Generic NLP clusters miss project-specific terminology. Adding terms to domain clusters dramatically improves keyword-level matching.
-8. **Filter mismatched entries during search** — Mixed databases need explicit filtering by `embedding_model`.
-
----
-
-## Replication Guide
-
-To set up this system in a new project using moflo:
-
-### 1. Install Moflo
-
-```bash
-npm install moflo
-npx flo init
-```
-
-### 2. Create Guidance Documents
-
-Create `.claude/guidance/` directory with markdown files following the optimization rules above:
-- Every file has `**Purpose:**` line
-- H2 sections with domain keywords in headings
-- Tables for structured rules
-- Cross-references between related docs
-- 1000-4000 char sections
-
-### 3. Configure Indexing
-
-In `moflo.yaml`:
-
-```yaml
-guidance:
-  directories:
-    - .claude/guidance
-    - docs/guides
-
-auto_index:
-  guidance: true
-  code_map: true
-```
-
-### 4. Index and Verify
-
-```bash
-# Index documents
-npx flo-index --force
-
-# Test search quality
-npx flo memory search --query "your domain query" --namespace guidance
-
-# Verify from Claude Code via MCP
-# mcp__moflo__memory_search query="your domain query" namespace="guidance"
-```
-
----
-
-## See Also
-
-- `.claude/guidance/memory-strategy.md` - Memory architecture and search commands
-- `.claude/guidance/agent-bootstrap.md` - Subagent bootstrap guide
-- `.claude/guidance/moflo.md` - Full CLI/MCP reference
+# Guidance & Memory Tuning Strategy
+
+**Purpose:** How to build and tune a RAG-based guidance system using moflo's semantic search, embedding pipeline, and indexing. Reference when creating guidance documents, troubleshooting search quality, or extending the system.
+
+---
+
+## Problem Statement
+
+Claude Code agents need project-specific knowledge — coding rules, architecture patterns, entity templates, testing conventions — delivered at the right moment. Without a retrieval system, agents either miss critical rules or require massive CLAUDE.md files that waste context window tokens.
+
+**Goals:**
+- Agents find relevant guidance automatically via semantic search
+- Subagents spawned by the coordinator inherit memory access
+- Search quality is high enough that agents don't need to read whole files
+- The system survives `npm install` (indexing runs on session start)
+
+---
+
+## Architecture
+
+Three layers: embedding generation, vector storage, and search.
+
+```
+Source Files (.claude/guidance/*.md, docs/*.md)
+         |
+         v
+index-guidance.mjs --- Chunk on ## headers, build RAG links
+         |                (prev/next, siblings, parent/child, context overlap)
+         v
+.swarm/memory.db ----- SQLite (entries + metadata + embedding vectors)
+         |
+         v
+build-embeddings.mjs - Generate 384-dim vectors per entry
+         |                (Xenova/all-MiniLM-L6-v2 neural, or domain-aware hash fallback)
+         v
+RuVector (@ruvector/core) -- HNSW index infrastructure
+         v
+Search layer ---------- Three access paths:
+                          1. MCP tools (mcp__moflo__memory_search) -- preferred
+                          2. CLI (npx flo memory search) -- fallback
+                          3. Script (semantic-search.mjs) -- detailed output
+```
+
+**Key files:**
+
+| File | Role |
+|------|------|
+| `.claude/guidance/*.md` | Guidance documents (source of truth) |
+| `bin/index-guidance.mjs` | Chunks documents, stores in SQLite with RAG metadata |
+| `bin/build-embeddings.mjs` | Generates vector embeddings (neural or hash) |
+| `.swarm/memory.db` | SQLite database with entries, metadata, embeddings |
+| `@ruvector/core` | HNSW vector index, WASM fallback, SIMD operations |
+
+---
+
+## Guidance Document Optimization Rules
+
+These rules determine how well your guidance documents retrieve via semantic search:
+
+### 1. Every file needs a Purpose line
+
+Add `**Purpose:**` as the first meaningful line after the title. Claude checks this first for relevance scoring. Without it, the chunk has no summary signal.
+
+### 2. H2 headings are the primary retrieval signal
+
+The indexer splits on `##`. Each heading becomes the chunk title, prepended to searchable content. Domain-specific keywords in headings dramatically improve recall.
+
+**Bad:** `## Overview`, `## Rules`, `## Pattern`
+**Good:** `## Soft Delete Rules`, `## JWT Authentication Pattern`, `## Database Entity Migration`
+
+### 3. Ideal chunk size: 1000-4000 characters
+
+Below 50 chars the chunk is dropped. Above 6000 the indexer force-splits on paragraphs, which breaks mid-thought. The sweet spot produces focused embeddings.
+
+### 4. Self-contained chunks
+
+Each H2 section must answer a question without needing the rest of the document. Include: the rule, a code example, and a cross-reference.
+
+### 5. Tables over prose
+
+Claude parses structured data more accurately than paragraphs. DO/DON'T tables, field reference tables, and command tables all retrieve better.
+
+### 6. Cross-references create a navigation graph
+
+The RAG indexer stores `prevChunk`/`nextChunk`/`siblings` metadata. Cross-references between documents let Claude follow chains: `core.md -> coding-rules.md -> database.md`.
+
+### 7. No decorative formatting
+
+ASCII boxes, excessive emoji, rhetorical questions, and motivational text all waste tokens without improving retrieval or comprehension.
+
+---
+
+## Embedding Pipeline
+
+### Embedding Models
+
+| Model | Quality | Speed | When Used |
+|-------|---------|-------|-----------|
+| `Xenova/all-MiniLM-L6-v2` | High (true semantic) | ~3s for 1000 entries | Primary — `build-embeddings.mjs` uses this |
+| `domain-aware-hash-v1` | Good (domain clustering) | <1s for 1000 entries | Fallback when Transformers.js unavailable |
+
+**Neural embeddings (Xenova/all-MiniLM-L6-v2):**
+- Uses `@xenova/transformers` with ONNX WASM runtime
+- 384-dimensional vectors, L2-normalized
+- True semantic understanding — "soft delete" matches "mark as deleted" without keyword overlap
+- Loaded lazily on first use, cached for subsequent queries
+- Ships with moflo; no additional install needed
+
+**Domain-aware hash embeddings (fallback):**
+- Custom SimHash-style algorithm with 12 domain clusters
+- Domain clusters group related terms: `database` (orm, postgresql, entity, schema...), `frontend` (react, component, css...), `testing` (vitest, mock, expect...), etc.
+- Multi-position hashing with bigram/trigram features
+- Good at keyword-level matching but misses semantic paraphrases
+- No external dependencies — always available
+
+### The Embedding Alignment Problem
+
+**Critical rule:** Query embeddings MUST match stored embeddings. Computing cosine similarity between vectors from different models produces meaningless scores.
+
+Both the search scripts and the MCP memory tools auto-detect the stored embedding model:
+
+```javascript
+// Check what model stored entries predominantly use
+const modelCheck = db.prepare(
+  `SELECT embedding_model, COUNT(*) as cnt FROM memory_entries
+   WHERE status = 'active' AND embedding IS NOT NULL
+   GROUP BY embedding_model ORDER BY cnt DESC LIMIT 1`
+).get();
+
+// If stored embeddings are neural, use neural for query too
+```
+
+Search also **filters out entries with mismatched `embedding_model`** — if the query uses neural embeddings, hash-embedded entries are skipped (and vice versa).
+
+### Domain Cluster Tuning
+
+The hash fallback's domain clusters can be extended with project-specific terms. Add terms to the relevant cluster in the hash embedding function to improve keyword-level matching for your domain:
+
+| Cluster | Example Terms |
+|---------|--------------|
+| `database` | your ORM, database engine, schema terms |
+| `frontend` | UI framework, component library terms |
+| `backend` | DI container, API framework terms |
+| `testing` | test framework, assertion library terms |
+| `security` | auth system, permission model terms |
+
+---
+
+## RAG Indexing Pipeline
+
+### How `index-guidance.mjs` Works
+
+1. **Scan** configured directories for `.md` files
+2. **Hash check** — Skip files whose content hash hasn't changed (unless `--force`)
+3. **Store full document** as `doc-{prefix}-{name}` (for complete retrieval)
+4. **Chunk on `##` headers** — Each H2 section becomes a separate entry
+5. **H3 subsections** become child chunks with parent H2 as context prefix
+6. **Force-split** sections over 4000 chars on paragraph boundaries
+7. **Build RAG metadata** for every chunk:
+
+| Metadata Field | Purpose |
+|---------------|---------|
+| `parentDoc` | Link back to full document |
+| `prevChunk` / `nextChunk` | Sequential navigation |
+| `siblings` | All chunk keys from same document |
+| `hierarchicalParent` / `hierarchicalChildren` | H2->H3 relationships |
+| `contextBefore` / `contextAfter` | 20% overlapping text from adjacent chunks |
+
+8. **Prepend context** — Each chunk's searchable content includes overlap from neighbors
+9. **Stale cleanup** — After indexing, remove entries for files that no longer exist on disk
+10. **Background embedding** — Spawn `build-embeddings.mjs` in background to generate vectors
+
+### Configuring Indexed Directories
+
+In `moflo.yaml`:
+
+```yaml
+guidance:
+  directories:
+    - .claude/guidance
+    - docs/guides
+```
+
+Default directories (when no config): `.claude/guidance`, `docs/guides`
+
+Moflo also automatically indexes its own bundled guidance from `node_modules/moflo/.claude/guidance/` when installed as a library in a consumer project.
+
+---
+
+## Lessons Learned
+
+### Document Optimization
+
+1. **`**Purpose:**` lines are critical** — They're the single highest-impact addition for retrieval quality.
+2. **Headings are embeddings** — In a chunk-per-section system, the heading IS the embedding's primary signal. Generic headings are nearly useless.
+3. **Tables retrieve better than prose** — Claude parses structured data with higher accuracy.
+4. **Cross-references are the RAG graph** — Isolated documents can't be navigated.
+5. **Chunk size matters** — A 10,000-char section produces a diluted embedding. Splitting into focused sections triples the chance of matching specific queries.
+
+### Embedding Pipeline
+
+6. **Query embeddings MUST match stored embeddings** — This is the single most critical rule. Auto-detect and match.
+7. **Domain clusters need project-specific terms** — Generic NLP clusters miss project-specific terminology. Adding terms to domain clusters dramatically improves keyword-level matching.
+8. **Filter mismatched entries during search** — Mixed databases need explicit filtering by `embedding_model`.
+
+---
+
+## Replication Guide
+
+To set up this system in a new project using moflo:
+
+### 1. Install Moflo
+
+```bash
+npm install moflo
+npx flo init
+```
+
+### 2. Create Guidance Documents
+
+Create `.claude/guidance/` directory with markdown files following the optimization rules above:
+- Every file has `**Purpose:**` line
+- H2 sections with domain keywords in headings
+- Tables for structured rules
+- Cross-references between related docs
+- 1000-4000 char sections
+
+### 3. Configure Indexing
+
+In `moflo.yaml`:
+
+```yaml
+guidance:
+  directories:
+    - .claude/guidance
+    - docs/guides
+
+auto_index:
+  guidance: true
+  code_map: true
+```
+
+### 4. Index and Verify
+
+```bash
+# Index documents
+npx flo-index --force
+
+# Test search quality
+npx flo memory search --query "your domain query" --namespace guidance
+
+# Verify from Claude Code via MCP
+# mcp__moflo__memory_search query="your domain query" namespace="guidance"
+```
+
+---
+
+## See Also
+
+- `.claude/guidance/memory-strategy.md` - Memory architecture and search commands
+- `.claude/guidance/agent-bootstrap.md` - Subagent bootstrap guide
+- `.claude/guidance/moflo.md` - Full CLI/MCP reference