moflo 4.3.1 → 4.6.0

package/.claude/guidance/agent-bootstrap.md

@@ -33,7 +33,7 @@ If you have MCP tools available (check for `mcp__claude-flow__*`), use them dire
 ### Option B: CLI via Bash
 
 ```bash
-npx moflo memory search --query "[describe your task]" --namespace guidance --limit 5
+npx flo memory search --query "[describe your task]" --namespace guidance --limit 5
 ```
 
 | Your task involves... | Search namespace | Example query |
@@ -111,7 +111,7 @@ mcp__claude-flow__memory_store
 
 ### CLI Fallback:
 ```bash
-npx moflo memory store --namespace patterns --key "brief-descriptive-key" --value "1-2 sentence insight"
+npx flo memory store --namespace patterns --key "brief-descriptive-key" --value "1-2 sentence insight"
 ```
 
 **Store:** Solutions to tricky bugs, patterns that worked, gotchas, workarounds

package/.claude/guidance/guidance-memory-strategy.md

@@ -0,0 +1,262 @@
+# Guidance & Memory Tuning Strategy
+
+**Purpose:** How to build and tune a RAG-based guidance system using moflo's semantic search, embedding pipeline, and indexing. Reference when creating guidance documents, troubleshooting search quality, or extending the system.
+
+---
+
+## Problem Statement
+
+Claude Code agents need project-specific knowledge — coding rules, architecture patterns, entity templates, testing conventions — delivered at the right moment. Without a retrieval system, agents either miss critical rules or require massive CLAUDE.md files that waste context window tokens.
+
+**Goals:**
+- Agents find relevant guidance automatically via semantic search
+- Subagents spawned by the coordinator inherit memory access
+- Search quality is high enough that agents don't need to read whole files
+- The system survives `npm install` (indexing runs on session start)
+
+---
+
+## Architecture
+
+Three layers: embedding generation, vector storage, and search.
+
+```
+Source Files (.claude/guidance/*.md, docs/*.md)
+         |
+         v
+index-guidance.mjs --- Chunk on ## headers, build RAG links
+         |                (prev/next, siblings, parent/child, context overlap)
+         v
+.swarm/memory.db ----- SQLite (entries + metadata + embedding vectors)
+         |
+         v
+build-embeddings.mjs - Generate 384-dim vectors per entry
+         |                (Xenova/all-MiniLM-L6-v2 neural, or domain-aware hash fallback)
+         v
+RuVector (@ruvector/core) -- HNSW index infrastructure
+         v
+Search layer ---------- Three access paths:
+                          1. MCP tools (mcp__claude-flow__memory_search) -- preferred
+                          2. CLI (npx flo memory search) -- fallback
+                          3. Script (semantic-search.mjs) -- detailed output
+```
+
+**Key files:**
+
+| File | Role |
+|------|------|
+| `.claude/guidance/*.md` | Guidance documents (source of truth) |
+| `bin/index-guidance.mjs` | Chunks documents, stores in SQLite with RAG metadata |
+| `bin/build-embeddings.mjs` | Generates vector embeddings (neural or hash) |
+| `.swarm/memory.db` | SQLite database with entries, metadata, embeddings |
+| `@ruvector/core` | HNSW vector index, WASM fallback, SIMD operations |
+
+---
+
+## Guidance Document Optimization Rules
+
+These rules determine how well your guidance documents retrieve via semantic search:
+
+### 1. Every file needs a Purpose line
+
+Add `**Purpose:**` as the first meaningful line after the title. Claude checks this first for relevance scoring. Without it, the chunk has no summary signal.
+
+### 2. H2 headings are the primary retrieval signal
+
+The indexer splits on `##`. Each heading becomes the chunk title, prepended to searchable content. Domain-specific keywords in headings dramatically improve recall.
+
+**Bad:** `## Overview`, `## Rules`, `## Pattern`
+**Good:** `## Soft Delete Rules`, `## JWT Authentication Pattern`, `## Database Entity Migration`
+
+### 3. Ideal chunk size: 1000-4000 characters
+
+Below 50 chars the chunk is dropped. Above 6000 the indexer force-splits on paragraphs, which breaks mid-thought. The sweet spot produces focused embeddings.
+
+### 4. Self-contained chunks
+
+Each H2 section must answer a question without needing the rest of the document. Include: the rule, a code example, and a cross-reference.
+
+### 5. Tables over prose
+
+Claude parses structured data more accurately than paragraphs. DO/DON'T tables, field reference tables, and command tables all retrieve better.
+
+### 6. Cross-references create a navigation graph
+
+The RAG indexer stores `prevChunk`/`nextChunk`/`siblings` metadata. Cross-references between documents let Claude follow chains: `core.md -> coding-rules.md -> database.md`.
+
+### 7. No decorative formatting
+
+ASCII boxes, excessive emoji, rhetorical questions, and motivational text all waste tokens without improving retrieval or comprehension.
+
+---
+
+## Embedding Pipeline
+
+### Embedding Models
+
+| Model | Quality | Speed | When Used |
+|-------|---------|-------|-----------|
+| `Xenova/all-MiniLM-L6-v2` | High (true semantic) | ~3s for 1000 entries | Primary — `build-embeddings.mjs` uses this |
+| `domain-aware-hash-v1` | Good (domain clustering) | <1s for 1000 entries | Fallback when Transformers.js unavailable |
+
+**Neural embeddings (Xenova/all-MiniLM-L6-v2):**
+- Uses `@xenova/transformers` with ONNX WASM runtime
+- 384-dimensional vectors, L2-normalized
+- True semantic understanding — "soft delete" matches "mark as deleted" without keyword overlap
+- Loaded lazily on first use, cached for subsequent queries
+- Ships with moflo; no additional install needed
+
+**Domain-aware hash embeddings (fallback):**
+- Custom SimHash-style algorithm with 12 domain clusters
+- Domain clusters group related terms: `database` (orm, postgresql, entity, schema...), `frontend` (react, component, css...), `testing` (vitest, mock, expect...), etc.
+- Multi-position hashing with bigram/trigram features
+- Good at keyword-level matching but misses semantic paraphrases
+- No external dependencies — always available
+
+### The Embedding Alignment Problem
+
+**Critical rule:** Query embeddings MUST match stored embeddings. Computing cosine similarity between vectors from different models produces meaningless scores.
+
+Both the search scripts and the MCP memory tools auto-detect the stored embedding model:
+
+```javascript
+// Check what model stored entries predominantly use
+const modelCheck = db.prepare(
+  `SELECT embedding_model, COUNT(*) as cnt FROM memory_entries
+   WHERE status = 'active' AND embedding IS NOT NULL
+   GROUP BY embedding_model ORDER BY cnt DESC LIMIT 1`
+).get();
+
+// If stored embeddings are neural, use neural for query too
+```
+
+Search also **filters out entries with mismatched `embedding_model`** — if the query uses neural embeddings, hash-embedded entries are skipped (and vice versa).
+
+### Domain Cluster Tuning
+
+The hash fallback's domain clusters can be extended with project-specific terms. Add terms to the relevant cluster in the hash embedding function to improve keyword-level matching for your domain:
+
+| Cluster | Example Terms |
+|---------|--------------|
+| `database` | your ORM, database engine, schema terms |
+| `frontend` | UI framework, component library terms |
+| `backend` | DI container, API framework terms |
+| `testing` | test framework, assertion library terms |
+| `security` | auth system, permission model terms |
+
+---
+
+## RAG Indexing Pipeline
+
+### How `index-guidance.mjs` Works
+
+1. **Scan** configured directories for `.md` files
+2. **Hash check** — Skip files whose content hash hasn't changed (unless `--force`)
+3. **Store full document** as `doc-{prefix}-{name}` (for complete retrieval)
+4. **Chunk on `##` headers** — Each H2 section becomes a separate entry
+5. **H3 subsections** become child chunks with parent H2 as context prefix
+6. **Force-split** sections over 4000 chars on paragraph boundaries
+7. **Build RAG metadata** for every chunk:
+
+| Metadata Field | Purpose |
+|---------------|---------|
+| `parentDoc` | Link back to full document |
+| `prevChunk` / `nextChunk` | Sequential navigation |
+| `siblings` | All chunk keys from same document |
+| `hierarchicalParent` / `hierarchicalChildren` | H2->H3 relationships |
+| `contextBefore` / `contextAfter` | 20% overlapping text from adjacent chunks |
+
+8. **Prepend context** — Each chunk's searchable content includes overlap from neighbors
+9. **Stale cleanup** — After indexing, remove entries for files that no longer exist on disk
+10. **Background embedding** — Spawn `build-embeddings.mjs` in background to generate vectors
+
+### Configuring Indexed Directories
+
+In `moflo.yaml`:
+
+```yaml
+guidance:
+  directories:
+    - .claude/guidance
+    - docs/guides
+```
+
+Default directories (when no config): `.claude/guidance`, `docs/guides`
+
+Moflo also automatically indexes its own bundled guidance from `node_modules/moflo/.claude/guidance/` when installed as a library in a consumer project.
+
+---
+
+## Lessons Learned
+
+### Document Optimization
+
+1. **`**Purpose:**` lines are critical** — They're the single highest-impact addition for retrieval quality.
+2. **Headings are embeddings** — In a chunk-per-section system, the heading IS the embedding's primary signal. Generic headings are nearly useless.
+3. **Tables retrieve better than prose** — Claude parses structured data with higher accuracy.
+4. **Cross-references are the RAG graph** — Isolated documents can't be navigated.
+5. **Chunk size matters** — A 10,000-char section produces a diluted embedding. Splitting into focused sections triples the chance of matching specific queries.
+
+### Embedding Pipeline
+
+6. **Query embeddings MUST match stored embeddings** — This is the single most critical rule. Auto-detect and match.
+7. **Domain clusters need project-specific terms** — Generic NLP clusters miss project-specific terminology. Adding terms to domain clusters dramatically improves keyword-level matching.
+8. **Filter mismatched entries during search** — Mixed databases need explicit filtering by `embedding_model`.
+
+---
+
+## Replication Guide
+
+To set up this system in a new project using moflo:
+
+### 1. Install Moflo
+
+```bash
+npm install moflo
+npx flo init
+```
+
+### 2. Create Guidance Documents
+
+Create `.claude/guidance/` directory with markdown files following the optimization rules above:
+- Every file has `**Purpose:**` line
+- H2 sections with domain keywords in headings
+- Tables for structured rules
+- Cross-references between related docs
+- 1000-4000 char sections
+
+### 3. Configure Indexing
+
+In `moflo.yaml`:
+
+```yaml
+guidance:
+  directories:
+    - .claude/guidance
+    - docs/guides
+
+auto_index:
+  guidance: true
+  code_map: true
+```
+
+### 4. Index and Verify
+
+```bash
+# Index documents
+npx flo-index --force
+
+# Test search quality
+npx flo memory search --query "your domain query" --namespace guidance
+
+# Verify from Claude Code via MCP
+# mcp__claude-flow__memory_search query="your domain query" namespace="guidance"
+```
+
+---
+
+## See Also
+
+- `.claude/guidance/memory-strategy.md` - Memory architecture and search commands
+- `.claude/guidance/agent-bootstrap.md` - Subagent bootstrap guide
+- `.claude/guidance/moflo.md` - Full CLI/MCP reference

package/.claude/guidance/memory-strategy.md

@@ -0,0 +1,204 @@
+# Memory & Semantic Search Strategy
+
+**Purpose:** How memory, embeddings, and semantic search work in moflo. Reference when debugging memory issues, understanding the search pipeline, or configuring memory for a consumer project.
+
+---
+
+## Architecture Overview
+
+```
+Source Files (.claude/guidance/, docs/)
+         |
+         v
+index-guidance.mjs (chunking, RAG linking)
+         |
+         v
+.swarm/memory.db (SQLite - entries + metadata)
+         |
+         v
+build-embeddings.mjs (384-dim neural or hash vectors)
+         |
+         v
+Search layer (cosine similarity - MCP, CLI, or script)
+```
+
+---
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `.swarm/memory.db` | SQLite database with all entries, embeddings, metadata |
+| `.swarm/code-map-hash.txt` | SHA-256 hash for incremental code map skip |
+| `.claude-flow/neural/patterns.json` | ReasoningBank learned patterns |
+| `bin/build-embeddings.mjs` | Generates 384-dim embeddings |
+| `bin/index-guidance.mjs` | Indexes guidance files with RAG linking |
+| `bin/generate-code-map.mjs` | Generates structural code map (projects, dirs, types, interfaces) |
+
+---
+
+## Embedding Strategy
+
+**Primary model:** `Xenova/all-MiniLM-L6-v2` (384-dim, neural — used by `build-embeddings.mjs`)
+**Fallback model:** `domain-aware-hash-v1` (384-dim, hash — used when Transformers.js unavailable)
+
+**Critical rule:** Query embeddings MUST match stored embeddings. Both the search scripts and MCP tools auto-detect the stored model and generate matching query vectors. Cross-model cosine similarity is meaningless.
+
+**Neural embeddings (primary):**
+- Uses `@xenova/transformers` with ONNX WASM runtime
+- True semantic understanding — "soft delete" matches "mark as deleted" without keyword overlap
+- ~3s for 1000 entries, loaded lazily and cached
+
+**Domain-aware hash (fallback):**
+- 12 domain clusters with project-specific terms
+- SimHash-style word encoding + bigram/trigram features
+- Good keyword-level matching, misses semantic paraphrases
+- No external dependencies — always available, <1s for 1000 entries
+
+See `guidance-memory-strategy.md` for full embedding pipeline details.
+
+---
+
+## Search Commands
+
+All methods auto-detect the stored embedding model and generate matching query vectors:
+
+**MCP (Preferred):** `mcp__claude-flow__memory_search` — `query: "your query", namespace: "guidance"`
+
+**CLI (Fallback):**
+```bash
+npx flo memory search --query "your query" --namespace guidance
+```
+
+**Search options:**
+
+| Flag | Default | Purpose |
+|------|---------|---------|
+| `--namespace` | all | Filter to specific namespace |
+| `--limit` | 5 | Number of results |
+| `--threshold` | 0.3 | Minimum similarity score |
+| `--json` | false | Output as JSON |
+
+### Code Map Search (for codebase navigation)
+
+When you need to find where a type, service, entity, or component lives — search `code-map` BEFORE using Glob/Grep:
+
+**MCP:** `mcp__claude-flow__memory_search` — `query: "payment service", namespace: "code-map"`
+
+**What code-map contains:**
+
+| Chunk prefix | What it answers |
+|--------------|-----------------|
+| `project:` | "What's in the api project?" |
+| `dir:` | "What types are in the entities directory?" |
+| `iface-map:` | "What implements IPaymentService?" |
+| `type-index:` | "Where is Service defined?" |
+
+**Regenerate:**
+```bash
+npx flo-codemap          # Incremental (skips if unchanged)
+npx flo-codemap --force  # Full rebuild
+```
+
+---
+
+## Database Schema
+
+```sql
+memory_entries (
+  id TEXT PRIMARY KEY,
+  key TEXT NOT NULL,           -- e.g., "chunk-guidance-core-0"
+  namespace TEXT,              -- "guidance", "patterns", "default"
+  content TEXT,                -- Full text content
+  embedding TEXT,              -- JSON array of 384 floats
+  embedding_model TEXT,        -- "Xenova/all-MiniLM-L6-v2" or "domain-aware-hash-v1"
+  embedding_dimensions INTEGER,-- 384
+  metadata TEXT,               -- JSON: parentDoc, chunkTitle, prevChunk, nextChunk, siblings
+  tags TEXT,                   -- JSON array
+  status TEXT,                 -- "active"
+  created_at INTEGER,
+  updated_at INTEGER
+)
+```
+
+---
+
+## RAG Linking (metadata fields)
+
+Each chunk includes navigation metadata:
+
+| Field | Purpose |
+|-------|---------|
+| `parentDoc` | Key of full document (e.g., `doc-guidance-core`) |
+| `prevChunk` | Previous chunk key for sequential reading |
+| `nextChunk` | Next chunk key |
+| `siblings` | All chunk keys from same document |
+| `hierarchicalParent` | H2 parent for H3 chunks |
+| `hierarchicalChildren` | H3 children for H2 chunks |
+| `contextBefore` | Overlapping text from previous chunk (20%) |
+| `contextAfter` | Overlapping text from next chunk (20%) |
+
+---
+
+## Namespaces
+
+| Namespace | Content | Notes |
+|-----------|---------|-------|
+| `guidance` | Indexed guidance and docs | Largest — includes bundled moflo guidance |
+| `code-map` | Structural codebase index (projects, directories, types, interfaces) | Search BEFORE Glob/Grep for navigation |
+| `patterns` | Learned patterns from sessions | Grows over time |
+| `default` | Misc stored data | Small |
+
+---
+
+## Session Start Indexing
+
+On every session start, moflo automatically runs three background indexers:
+
+| Indexer | Command | Namespace | What it does |
+|---------|---------|-----------|--------------|
+| Guidance | `npx flo-index` | `guidance` | Chunks markdown, builds RAG links, generates embeddings |
+| Code Map | `npx flo-codemap` | `code-map` | Scans source for types, interfaces, directories |
+| Learning | `npx flo-learn` | `patterns` | Pattern research on codebase |
+
+These are configured in the `SessionStart` hook in `.claude/settings.json` (set up by `npx flo init`).
+
+Indexing is incremental by default — files whose content hash hasn't changed are skipped. Use `--force` to reindex everything.
+
+---
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Search returns irrelevant results | Query/stored embedding model mismatch | Auto-detected now; verify with `--verbose` flag |
+| Low similarity scores | Query doesn't match domain terms | Include domain keywords in query |
+| "Vector: No" in list | Entry lacks embedding | Run `npx flo-index --force` |
+| Entries not found after adding file | Indexer hasn't run yet | Run `npx flo-index` or restart session |
+| Bundled moflo guidance not indexed | Not installed as dependency | Only indexes when `node_modules/moflo/.claude/guidance/` exists |
+
+---
+
+## Verification Commands
+
+```bash
+# Test semantic search
+npx flo memory search --query "database entity pattern" --namespace guidance
+
+# Force reindex all guidance
+npx flo-index --force
+
+# Force rebuild embeddings
+npx flo-index --force
+
+# Check entry count (requires better-sqlite3 or sql.js)
+npx flo memory list --namespace guidance --limit 0
+```
+
+---
+
+## See Also
+
+- `.claude/guidance/guidance-memory-strategy.md` - RAG system tuning guide
+- `.claude/guidance/agent-bootstrap.md` - Subagent bootstrap guide
+- `.claude/guidance/moflo.md` - Full CLI/MCP reference