aiwcli 0.12.2 → 0.12.3

package/dist/templates/cc-native/_cc-native/lib-ts/rlm/CLAUDE.md

@@ -0,0 +1,480 @@
+# RLM — Retrieval-Augmented Learning Memory
+
+## What is RLM?
+
+A two-tier retrieval system that lets you ask questions about past Claude Code sessions across all projects. Automatically indexes session transcripts and provides both keyword and semantic search.
+
+RLM stands for "Retrieval-augmented Learning Memory" — inspired by RAG (Retrieval-Augmented Generation) patterns but adapted for conversational session history. It turns your entire Claude Code session history into a searchable, queryable knowledge base.
+
+## User Guide
+
+### Commands
+
+**rlm:ask** — Answer a question about past work
+- Auto-builds indexes on first use (one-time setup)
+- Uses semantic search when Ollama + vectors available
+- Falls back to keyword search otherwise
+- Example: `/rlm:ask "How did we implement the plan review system?"`
+
+**rlm:overview** — Get a timeline summary
+- Group sessions by date/project/theme
+- Example: `/rlm:overview "hook development this week"`
+
+**rlm:index** — Force-rebuild indexes
+- Manual rebuild mechanism when auto-indexing fails
+- Accepts optional flags: `--limit=N`, `--project=<name>`
+- Example: `/rlm:index` or `/rlm:index --project=bridge`
+
+### First-Time Setup
+
+No manual setup required. On first use, `rlm:ask` will:
+1. **Auto-build JSON indexes** (takes 10-30s per 100 sessions)
+2. **Auto-detect Ollama** and use semantic search if available
+3. **Auto-pull model** if Ollama is running but `nomic-embed-text` is missing
+
+The first run might take 30-60 seconds as it builds indexes. Subsequent runs are fast (sub-second for keyword search, 2-5s for semantic search).
+
+### When to Use What
+
+| Use Case | Command |
+|----------|---------|
+| "How did we solve X?" | `rlm:ask` |
+| "What did we work on this week?" | `rlm:overview` |
+| "Find sessions about Y" | `rlm:ask` (returns sources table) |
+| "Indexes are stale/broken" | `rlm:index` (force rebuild) |
+
+### Search Quality: Keyword vs Semantic
+
+**Keyword Search** (always available, fast):
+- Uses weighted scoring: summary (3.0x), keywords (2.0x), files touched (1.5x), tool calls (1.0x)
+- Best for: Exact terms, file names, command names, specific error messages
+- Example: "transcript-indexer.ts" or "TaskCreate tool"
+
+**Semantic Search** (requires Ollama, higher quality):
+- Uses vector embeddings with KNN similarity
+- Best for: Conceptual queries, "how did we..." questions, approximate matches
+- Example: "error handling patterns" or "plan approval workflow"
+
+The system automatically picks the best available method. If Ollama isn't running, it gracefully falls back to keyword search.
+
+---
+
+## Architecture Overview (For Maintainers)
+
+### Two-Tier Pipeline
+
+**Tier 1: JSON Indexes (Always Available)**
+- Fast keyword-based search
+- Metadata extraction: summary, keywords, files touched, tool calls
+- Scoring with weighted factors (summary: 3.0, keywords: 2.0, files: 1.5, tools: 1.0)
+- Storage: `~/.claude/rlm-index/{project}/{session}.index.json`
+- Builder: `transcript-indexer.ts`
+- Searcher: `transcript-searcher.ts`
+
+**Tier 2: Vector Embeddings (Requires Ollama)**
+- Semantic similarity search using KNN
+- HyDE query expansion (optional, 20-45% recall improvement)
+- 6-stage retrieval pipeline: embed → search → summarize → rank → synthesize
+- Storage: `~/.claude/rlm-vectors.db` (SQLite + sqlite-vec)
+- Builder: `embedding-indexer.ts`
+- Pipeline: `retrieval-pipeline.ts`
+
+### HyDE (Hypothetical Document Embeddings)
+
+**What:** Generates 5 hypothetical responses to the query, embeds them, averages the vectors.
+
+**Why:** Improves recall by 20-45% (research-backed). The technique addresses the embedding space mismatch between short queries and long documents by generating synthetic document-like content from the query.
+
+**Status:** Opt-in via `cc-native.config.json` (`rlm.hyde.enabled: true`).
+
+**Cost:** Uses local Ollama (free) or Claude API (paid fallback).
+
+**How It Works:**
+1. User asks: "How did we implement hooks?"
+2. HyDE generates 5 hypothetical answers (e.g., "We implemented hooks using TypeScript...")
+3. Each hypothetical is embedded → 5 vectors
+4. Vectors are averaged → single query vector
+5. KNN search uses this averaged vector (closer to document space than raw query)
+
+Configuration in `cc-native.config.json`:
+```json
+"rlm": {
+  "hyde": {
+    "enabled": false,              // Opt-in
+    "provider": "ollama",          // "ollama" or "claude"
+    "ollamaModel": "qwen2.5:1.5b", // Fast model
+    "numResponses": 5,
+    "fallbackToQuery": true        // Graceful degradation
+  }
+}
+```
+
+### Code Organization
+
+```
+.aiwcli/_cc-native/lib-ts/rlm/
+├── types.ts                    # All types + constants
+├── transcript-indexer.ts       # Build JSON indexes
+├── transcript-searcher.ts      # Keyword search
+├── transcript-loader.ts        # Load transcript segments
+├── embedding-indexer.ts        # Build vector index
+├── vector-store.ts            # SQLite + sqlite-vec KNN
+├── retrieval-pipeline.ts       # 6-stage semantic pipeline
+├── ollama-client.ts           # Ollama API wrapper
+├── hyde.ts                    # HyDE query expansion
+├── logger.ts                  # RLM-specific logging
+└── CLAUDE.md                  # This file
+```
+
+### Data Flow
+
+```
+User Query
+  ↓
+rlm:ask command
+  ↓
+┌─ Auto-index check
+│  └─ transcript-indexer.ts (if needed)
+├─ Vector DB check
+│  ├─ YES + Ollama running → retrieval-pipeline.ts
+│  │   ├─ hyde.ts (optional query expansion)
+│  │   ├─ ollama-client.ts (embed query)
+│  │   ├─ vector-store.ts (KNN search)
+│  │   ├─ transcript-loader.ts (load segments)
+│  │   ├─ Parallel summarization (Haiku)
+│  │   ├─ AI ranking (Sonnet)
+│  │   └─ Synthesis (Sonnet)
+│  └─ NO Ollama → transcript-searcher.ts
+│      ├─ Load indexes
+│      ├─ Score against query
+│      └─ Return top N
+├─ transcript-loader.ts (load segments)
+├─ Parallel sub-agent analysis (1-5 agents)
+└─ Final synthesis with citations
+```
+
+### Index Schema Version
+
+Current: `CURRENT_SCHEMA_VERSION = 1` (in `types.ts`)
+
+When schema changes:
+1. Bump version constant
+2. Searcher automatically skips old indexes
+3. Indexer rebuilds on next run
+4. No manual migration needed
+
+The schema version is embedded in each `.index.json` file. The searcher checks this version and ignores indexes from older schema versions, forcing a rebuild.
+
+### JSON Index Structure
+
+Each session gets a `.index.json` file with this structure:
+
+```typescript
+interface TranscriptIndex {
+  schema_version: number;          // Current: 1
+  session_id: string;              // UUID
+  project: string;                 // Project name
+  source_path: string;             // Path to .jsonl transcript
+  created_at: string;              // ISO timestamp
+  summary: string;                 // AI-generated summary (1-2 sentences)
+  keywords: string[];              // Extracted keywords (5-10 terms)
+  files_touched: string[];         // File paths mentioned in session
+  tool_calls: string[];            // Tool names used (Read, Edit, Bash, etc.)
+  segments: Array<{                // Text chunks with line ranges
+    text: string;
+    lines: [number, number];
+  }>;
+}
+```
+
+### Vector Index Structure
+
+SQLite database (`~/.claude/rlm-vectors.db`) with two tables:
+
+**embeddings table:**
+```sql
+CREATE TABLE embeddings (
+  id INTEGER PRIMARY KEY,
+  session_id TEXT NOT NULL,
+  project TEXT NOT NULL,
+  source_path TEXT NOT NULL,
+  segment_index INTEGER NOT NULL,
+  text TEXT NOT NULL,
+  embedding BLOB NOT NULL,  -- Float32Array serialized
+  lines_start INTEGER,
+  lines_end INTEGER
+);
+```
+
+**metadata table:**
+```sql
+CREATE TABLE metadata (
+  session_id TEXT PRIMARY KEY,
+  project TEXT NOT NULL,
+  summary TEXT,
+  indexed_at TEXT NOT NULL
+);
+```
+
+The `sqlite-vec` extension provides KNN search via `vec_distance_cosine(embedding, query_vector)`.
+
+---
+
+## Troubleshooting
+
+### "No indexes found"
+- **Cause:** First run, or indexes deleted
+- **Fix:** Auto-builds on next `rlm:ask` run (10-30s delay)
+- **Manual fix:** `/rlm:index`
+
+### "Vector search unavailable"
+- **Cause:** Ollama not running or vector DB missing
+- **Impact:** Falls back to keyword search (still works, just less semantic)
+- **Fix:** Start Ollama (`ollama serve`), or let auto-indexing handle it on next semantic query
+
+### "Model not found: nomic-embed-text"
+- **Cause:** Ollama running but model not installed
+- **Fix:** Auto-pulls on next `rlm:ask` run (~400MB, 1-2 minutes), or manual: `ollama pull nomic-embed-text`
+
+### "HyDE disabled" (in logs)
+- **Cause:** `rlm.hyde.enabled: false` in config
+- **Impact:** Direct query embedding (still works, slightly lower recall ~20%)
+- **Fix:** Enable in `cc-native.config.json` if you want 20-45% recall improvement
+
+### Stale indexes
+- **Cause:** New sessions not indexed
+- **Fix:** Indexer checks source `.jsonl` mtime, auto-rebuilds if newer
+- **Manual fix:** `/rlm:index` to force full rebuild
+
+### SQLite errors
+- **Cause:** Corrupted `rlm-vectors.db`
+- **Fix:** Delete `~/.claude/rlm-vectors.db`, rebuild via `embedding-indexer.ts --batch`
+
+### Empty search results
+- **Possible causes:**
+  1. Query too specific (try broader terms)
+  2. Sessions not indexed yet (check `~/.claude/rlm-index/` exists)
+  3. Searching for recent work not yet persisted (Claude Code writes transcripts on session end)
+- **Fix:** Try different query terms, or run `/rlm:index` to ensure all sessions indexed
+
+---
+
+## Performance Tuning
+
+### Keyword Search (Fast)
+- **Average:** 50-200ms for 100 sessions
+- **No dependencies** (always available)
+- **Use for:** Quick session lookups, file/command searches, exact term matches
+
+### Vector Search (Slower, Higher Quality)
+- **Average:** 2-5s end-to-end (with HyDE: +1-3s)
+- **Requires:** Ollama + nomic-embed-text model
+- **Use for:** Semantic questions, "how did we...", "why did we...", conceptual queries
+
+### HyDE (Optional Quality Boost)
+- **Adds:** 1-3s (5 hypothetical responses generated)
+- **Improves recall:** 20-45% (research-backed)
+- **Recommended for:** Complex queries, abstract questions, when precision matters
+- **Disable for:** Speed-critical workflows, simple lookups
+
+### Parallel Agent Count
+
+The `rlm:ask` command spawns 1-5 parallel sub-agents based on search result count:
+- 1 result → 1 agent (minimal latency)
+- 3 results → 3 agents (balanced)
+- 10 results → 5 agents capped (prevents over-parallelization)
+
+This dynamic scaling ensures:
+- Fast response for focused queries (1-2 results)
+- Comprehensive analysis for broad queries (5+ results)
+- Bounded latency (never more than 5 concurrent agents)
+
+---
+
+## Configuration Reference
+
+See `cc-native.config.json`:
+
+```json
+"rlm": {
+  "hyde": {
+    "enabled": false,              // Opt-in for HyDE (20-45% recall boost)
+    "provider": "ollama",          // "ollama" (local, free) or "claude" (API, paid)
+    "ollamaModel": "qwen2.5:1.5b", // Model for hypothetical generation
+    "numResponses": 5,             // Hypotheticals to generate (more = better but slower)
+    "maxTokens": 200,              // Per response (keep short for speed)
+    "timeoutMs": 10000,            // Per response (fail fast on hung requests)
+    "fallbackToQuery": true,       // Degrade gracefully on HyDE failure
+    "fallbackToClaude": false      // Don't burn API credits on fallback (default)
+  }
+}
+```
+
+### Configuration Trade-offs
+
+| Setting | Value | Impact |
+|---------|-------|--------|
+| `hyde.enabled` | `true` | +20-45% recall, +1-3s latency |
+| `hyde.enabled` | `false` | Baseline recall, faster queries |
+| `hyde.numResponses` | `3` | Faster but lower recall boost (~15-25%) |
+| `hyde.numResponses` | `7` | Higher recall boost (~30-50%) but +2s latency |
+| `hyde.provider` | `ollama` | Free, requires local Ollama, ~2-3s for 5 responses |
+| `hyde.provider` | `claude` | Paid, always available, ~1-2s for 5 responses |
+
+---
+
+## Implementation Details
+
+### Auto-Indexing Logic
+
+The `rlm:ask` command checks for indexes before search:
+
+```bash
+INDEX_DIR="$HOME/.claude/rlm-index"
+if [ ! -d "$INDEX_DIR" ] || [ -z "$(ls -A $INDEX_DIR 2>/dev/null)" ]; then
+  # First run: build indexes
+  bun transcript-indexer.ts --batch
+fi
+```
+
+The indexer is **idempotent**:
+- Checks source `.jsonl` mtime vs index mtime
+- Skips up-to-date files
+- Only indexes new/modified sessions
+
+This means running `/rlm:index` multiple times is safe and fast (rebuilds only what changed).
+
+### Auto-Model-Pull Logic
+
+The `rlm:ask` command checks for the embedding model:
+
+```bash
+if curl -s http://localhost:11434/api/tags >/dev/null 2>&1; then
+  # Ollama running
+  if ! ollama list | grep -q "nomic-embed-text"; then
+    # Model missing: auto-pull
+    ollama pull nomic-embed-text
+  fi
+fi
+```
+
+The `ollama pull` command:
+- Downloads ~400MB model
+- Shows native Ollama progress bar
+- Blocks until complete (1-2 minutes on typical connection)
+- On failure, falls back to keyword search with warning
+
+### Graceful Degradation Path
+
+```
+rlm:ask query
+  ↓
+JSON indexes exist? ────NO──→ Auto-build → Continue
+  ↓ YES
+Vector DB exists? ────NO──→ transcript-searcher.ts (keyword)
+  ↓ YES
+Ollama running? ────NO──→ transcript-searcher.ts (keyword)
+  ↓ YES
+Model present? ────NO──→ Auto-pull → Continue or fallback
+  ↓ YES
+retrieval-pipeline.ts (semantic)
+```
+
+Every failure point has a fallback. The system never errors out due to missing infrastructure—it degrades to the best available method.
+
+---
+
+## Future Enhancements
+
+### Short-Term (Next 1-2 Versions)
+- **Auto-embed on session end** — Build vector index incrementally via `SessionEnd` hook
+- **Cross-project search** — Filter by multiple projects: `--projects=aiwcli,bridge-app`
+- **Date range filtering** — Native support: `--since=2025-01-01` or `--last-week`
+
+### Medium-Term (Next 3-6 Months)
+- **Reranking** — Use Cohere or Jina rerankers for better precision after KNN recall
+- **Cache summaries** — Avoid re-summarizing the same chunks across queries
+- **Query expansion** — Synonym/related term expansion for keyword search
+- **Incremental indexing** — Watch `.jsonl` files and index on change (file watcher hook)
+
+### Long-Term (Research / Experimental)
+- **Multi-modal embeddings** — Embed code, images, browser screenshots separately
+- **Graph-based retrieval** — Track file evolution, tool sequences, error→fix patterns
+- **Personalized ranking** — Learn from user's click/accept signals to improve relevance
+- **Cross-session context** — Link related sessions into threads/projects automatically
+
+---
+
+## FAQ
+
+### Why two tiers (JSON + vector)?
+
+**Availability:** JSON indexes have zero dependencies and always work. Vector search requires Ollama setup.
+
+**Speed:** Keyword search is 10-50x faster than vector search for simple lookups.
+
+**Quality:** Semantic search handles synonyms, rephrasing, and conceptual queries that keyword search misses.
+
+Having both tiers means the system is useful immediately (keyword) and scales up in quality when you install Ollama (semantic).
+
+### Why Ollama instead of OpenAI embeddings?
+
+**Cost:** Ollama is free and local. OpenAI charges per token.
+
+**Privacy:** Session transcripts stay on your machine.
+
+**Speed:** Local embeddings are ~10-30ms vs ~100-300ms for API round-trips.
+
+**Offline:** Works without internet once model is pulled.
+
+For production systems with budget, OpenAI embeddings (`text-embedding-3-small`) would be faster and higher quality. But for a personal dev tool, Ollama hits the sweet spot.
+
+### Why nomic-embed-text model?
+
+**Optimized for retrieval:** Trained specifically for semantic search (not chat).
+
+**Fast:** 137M parameters (~400MB), embeds in 10-30ms on CPU.
+
+**Quality:** Competitive with OpenAI's `text-embedding-ada-002` on MTEB benchmarks.
+
+**License:** Apache 2.0 (commercially usable).
+
+Alternative models (e.g., `all-MiniLM-L6-v2`) are smaller but lower quality. Larger models (e.g., `gte-large`) are higher quality but slower. `nomic-embed-text` is the best speed/quality trade-off for this use case.
+
+### Why SQLite for vector storage?
+
+**Simplicity:** Single-file database, no server setup.
+
+**Portability:** Copy `rlm-vectors.db` to backup/share entire index.
+
+**Performance:** sqlite-vec (KNN extension) is fast enough for 1000s of documents.
+
+**Compatibility:** Works everywhere SQLite works (all platforms, no dependencies).
+
+For larger deployments (10K+ sessions), consider ChromaDB, Pinecone, or Weaviate. But for personal use, SQLite is simpler and faster to set up.
+
+### How accurate is semantic search?
+
+**Precision:** ~70-85% (depends on query quality and corpus size)
+
+**Recall:** ~60-80% baseline, 80-95% with HyDE enabled
+
+**Comparison to keyword:** Semantic search handles paraphrasing better, keyword search handles exact terms better. Best results come from using both (semantic for recall, keyword for precision).
+
+The system doesn't expose both simultaneously yet—it auto-picks one. Future versions may support hybrid search (combine both scores).
+
+---
+
+## Changelog
+
+### v1.0.0 (2025-02-15)
+- **Command consolidation:** 6 commands → 3 commands (ask, overview, index)
+- **Auto-indexing:** Transparent index building on first use
+- **Auto-model-pull:** Automatic nomic-embed-text download if missing
+- **Unified interface:** Single `rlm:ask` for all Q&A workflows
+- **Documentation:** Comprehensive CLAUDE.md with architecture + user guide
+
+### Pre-v1.0 (2025-02-14)
+- Initial implementation with 6 separate commands
+- Manual index building required
+- Separate commands for search/retrieve/recall (confusing UX)