@intent-systems/nexus 2026.1.5-3

package/skills/onboarding/docs/skill-deep-dives/qmd.md

@@ -0,0 +1,742 @@
+# Skill Deep-Dive: qmd
+
+**Skill**: `qmd`
+**Category**: Code/Productivity (Local Search & Indexing)
+**Setup Difficulty**: 🟡 Medium
+**Time Estimate**: ~10 minutes
+**Platform**: Cross-platform (macOS, Windows, Linux)
+
+---
+
+## What It Enables
+
+| Capability | Description | Example |
+|------------|-------------|---------|
+| BM25 search | Fast keyword search across indexed files | `qmd search "authentication"` |
+| Vector search | Semantic search (meaning-based) | `qmd vsearch "how to login"` |
+| Hybrid search | Combined BM25 + vector + rerank | `qmd query "OAuth flow"` |
+| Collection management | Index specific directories with file masks | `qmd collection add ~/docs --name docs --mask "**/*.md"` |
+| Document retrieval | Get specific document with context | `qmd get docs/api.md:10 -l 40` |
+| MCP mode | Model Context Protocol server | `qmd mcp` |
+| Ollama integration | Local embeddings and reranking | Uses Ollama for vector generation |
+
+---
+
+## Dependencies
+
+### Hard Dependencies (Required)
+- [ ] `node` + `npm` — For installing qmd
+- [ ] `git` — For npm package installation from GitHub
+
+### Soft Dependencies (Recommended)
+- [ ] `ollama` — For vector search and reranking (optional but recommended)
+- [ ] Ollama models — For embeddings (e.g., `nomic-embed-text`)
+
+### This Skill Enables
+- [ ] Semantic code search — Find code by meaning, not just keywords
+- [ ] Documentation indexing — Search project docs quickly
+- [ ] Knowledge base — Index personal notes, wikis, etc.
+- [ ] AI context — Provide relevant docs to AI agents via MCP
+
+---
+
+## Installation
+
+### Prerequisites Check
+
+```bash
+# Check for Node.js
+which node && node --version || echo "❌ Need Node.js"
+
+# Check for npm
+which npm && npm --version || echo "❌ Need npm"
+
+# Check for git
+which git && echo "✅ git available"
+
+# Optional: Check for Ollama (for vector search)
+which ollama && echo "✅ Ollama available" || echo "⚠️  Ollama not installed (vector search disabled)"
+```
+
+### Installation Steps
+
+```bash
+# Install qmd from GitHub
+npm install -g https://github.com/tobi/qmd
+
+# Verify installation
+qmd --version
+
+# Check installation location
+which qmd
+```
+
+### Verification
+
+```bash
+# Verify qmd is accessible
+qmd --version
+
+# Check default config location
+ls ~/.cache/qmd && echo "✅ Cache directory exists" || echo "Will be created on first use"
+
+# Test Ollama connection (optional)
+if which ollama > /dev/null; then
+    curl -s http://localhost:11434/api/tags > /dev/null && echo "✅ Ollama reachable" || echo "⚠️  Start Ollama: ollama serve"
+fi
+```
+
+---
+
+## Configuration
+
+### Step 1: Start Ollama (Optional - For Vector Search)
+
+```bash
+# Check if Ollama is running
+curl -s http://localhost:11434/api/tags > /dev/null && echo "✅ Ollama running"
+
+# If not running, start it
+ollama serve &
+
+# Pull embedding model
+ollama pull nomic-embed-text
+
+# Pull reranking model (optional)
+ollama pull mixedbread-ai/mxbai-rerank-large-v1
+```
+
+### Step 2: Set Ollama URL (If Non-Default)
+
+```bash
+# Default is http://localhost:11434
+# To use different URL:
+export OLLAMA_URL="http://your-ollama-server:11434"
+
+# Add to shell profile for persistence
+echo 'export OLLAMA_URL="http://localhost:11434"' >> ~/.zshrc
+source ~/.zshrc
+```
+
+### Step 3: Add Your First Collection
+
+```bash
+# Index Markdown files in a directory
+qmd collection add ~/Documents/notes --name notes --mask "**/*.md"
+
+# Index code in a project
+qmd collection add ~/projects/myapp --name myapp --mask "**/*.{js,ts,py}"
+
+# Index multiple file types
+qmd collection add ~/nexus --name nexus --mask "**/*.{md,js,ts,yaml,json}"
+
+# List collections
+qmd collection list
+```
+
+### Step 4: Build Index
+
+```bash
+# Update index for all collections
+qmd update
+
+# This will:
+# 1. Scan directories for matching files
+# 2. Extract text content
+# 3. Build BM25 index
+# 4. Generate embeddings (if Ollama available)
+
+# Check status
+qmd status
+```
+
+### Verification
+
+```bash
+# Check collections
+qmd collection list
+
+# Check index status
+qmd status
+
+# Test search
+qmd search "test query"
+
+# Test vector search (requires Ollama)
+qmd vsearch "test query"
+```
+
+---
+
+## User Actions Required
+
+No manual system permissions required! All operations are file-based.
+
+---
+
+## Common Failure Points
+
+### Failure: "command not found: qmd"
+
+**Symptom**: Shell can't find qmd after installation
+
+**Cause**: npm global bin directory not in PATH
+
+**Recovery**:
+```bash
+# Find npm global bin directory
+npm config get prefix
+
+# Add to PATH (if not already)
+# For npm prefix at /usr/local:
+export PATH="/usr/local/bin:$PATH"
+
+# Or for user-local npm:
+export PATH="$HOME/.npm-global/bin:$PATH"
+
+# Add to shell profile
+echo 'export PATH="$HOME/.npm-global/bin:$PATH"' >> ~/.zshrc
+source ~/.zshrc
+
+# Verify
+which qmd
+```
+
+**Prevention**: Ensure npm global bin is in PATH before installing
+
+---
+
+### Failure: "Ollama connection refused"
+
+**Symptom**: Vector search fails with connection error
+
+**Cause**: Ollama not running
+
+**Recovery**:
+```bash
+# Check if Ollama is running
+curl -s http://localhost:11434/api/tags
+
+# If not, start Ollama
+ollama serve &
+
+# Wait a moment for startup
+sleep 2
+
+# Test again
+curl -s http://localhost:11434/api/tags
+
+# Try vector search
+qmd vsearch "test query"
+```
+
+**Prevention**: Start Ollama before using vector search features
+
+---
+
+### Failure: "No collections found"
+
+**Symptom**: Search returns no results
+
+**Cause**: No collections added or index not built
+
+**Recovery**:
+```bash
+# Check collections
+qmd collection list
+
+# If empty, add a collection
+qmd collection add ~/Documents --name docs --mask "**/*.md"
+
+# Update index
+qmd update
+
+# Verify
+qmd status
+
+# Try search again
+qmd search "test"
+```
+
+**Prevention**: Add collections and build index before searching
+
+---
+
+### Failure: "Index out of date"
+
+**Symptom**: Recent files not appearing in search results
+
+**Cause**: Index not updated after file changes
+
+**Recovery**:
+```bash
+# Rebuild index
+qmd update
+
+# For large collections, this may take a few minutes
+# Watch progress in terminal
+
+# Verify
+qmd status
+```
+
+**Prevention**: Run `qmd update` regularly or schedule via cron
+
+---
+
+### Failure: Vector search returns no results
+
+**Symptom**: `qmd vsearch` returns empty results
+
+**Cause**: Embeddings not generated (Ollama not available during indexing)
+
+**Recovery**:
+```bash
+# Ensure Ollama is running
+ollama serve &
+
+# Pull embedding model if missing
+ollama pull nomic-embed-text
+
+# Rebuild index with embeddings
+qmd update
+
+# Try vector search again
+qmd vsearch "your query"
+```
+
+**Prevention**: Install and start Ollama before first indexing
+
+---
+
+## Automation Opportunities
+
+| Step | Automatable? | Tool | Notes |
+|------|--------------|------|-------|
+| Install qmd | ✅ Yes | bash | `npm install -g https://github.com/tobi/qmd` |
+| Start Ollama | ✅ Yes | bash | `ollama serve &` |
+| Add collections | ✅ Yes | bash | `qmd collection add ...` |
+| Build index | ✅ Yes | bash | `qmd update` |
+| Search | ✅ Yes | bash | `qmd search "query"` |
+| Schedule updates | ✅ Yes | cron | Periodic `qmd update` |
+| MCP server | ✅ Yes | bash | `qmd mcp` |
+
+### Automation Script: Full Setup
+
+```bash
+#!/bin/bash
+# Complete qmd setup automation
+
+set -e
+
+echo "Installing qmd..."
+npm install -g https://github.com/tobi/qmd
+
+echo "Checking for Ollama..."
+if ! which ollama > /dev/null; then
+    echo "⚠️  Ollama not installed. Vector search will be disabled."
+    echo "Install with: curl -fsSL https://ollama.com/install.sh | sh"
+else
+    echo "Starting Ollama..."
+    ollama serve &
+    sleep 2
+
+    echo "Pulling embedding model..."
+    ollama pull nomic-embed-text
+fi
+
+echo "Adding collections..."
+# Customize these for your directories
+qmd collection add ~/Documents --name docs --mask "**/*.md"
+qmd collection add ~/nexus --name nexus --mask "**/*.{md,yaml,json}"
+qmd collection add ~/projects --name projects --mask "**/*.{js,ts,py,go}"
+
+echo "Building index..."
+qmd update
+
+echo "✅ qmd setup complete!"
+echo ""
+echo "Try it:"
+echo "  qmd search 'your query'"
+echo "  qmd vsearch 'semantic query'"
+echo "  qmd status"
+```
+
+### Scheduled Index Updates
+
+```bash
+# Add to cron or use nexus cron skill
+# Update index every hour
+0 * * * * /usr/local/bin/qmd update
+
+# Or daily at 3 AM
+0 3 * * * /usr/local/bin/qmd update
+```
+
+---
+
+## Wow Moments
+
+| Moment | What Happens | When to Demo |
+|--------|--------------|--------------|
+| Semantic search magic | Search "how to authenticate" finds code even if it doesn't contain those words | After first index build |
+| Instant results | Search 10k+ files in milliseconds | When user has large codebase |
+| Hybrid search | Combines keyword + semantic + rerank for best results | When showing advanced features |
+| MCP integration | AI agents can search your docs automatically | When using with AI tools |
+| Cross-project search | Find similar code patterns across all projects | When demonstrating power |
+
+### Demo Script
+
+```
+Agent: "Let me show you qmd's power. First, I've indexed your documents and code:"
+
+qmd status
+
+Agent: "Now let's do a traditional keyword search:"
+
+qmd search "authentication"
+
+Agent: "That's fast, but watch what happens with semantic search. I'll search by meaning, not exact words:"
+
+qmd vsearch "how do users log in"
+
+Agent: "Notice it found authentication code even though I didn't use the word 'authentication'? That's semantic search - it understands meaning."
+
+Agent: "And the most powerful feature - hybrid search combines both approaches and reranks results:"
+
+qmd query "OAuth implementation"
+
+Agent: "This gives you the best of both worlds: keyword precision + semantic understanding."
+```
+
+---
+
+## Usage Examples
+
+### Adding Collections
+
+```bash
+# Basic: Add directory with default mask (all files)
+qmd collection add ~/Documents --name docs
+
+# With file mask: Only Markdown files
+qmd collection add ~/notes --name notes --mask "**/*.md"
+
+# Multiple file types
+qmd collection add ~/projects/app --name app --mask "**/*.{js,ts,jsx,tsx}"
+
+# Exclude directories (use shell glob patterns)
+qmd collection add ~/code --name code --mask "**/*.py" --exclude "**/node_modules/**"
+
+# List all collections
+qmd collection list
+
+# Remove a collection
+qmd collection remove docs
+```
+
+### Updating Index
+
+```bash
+# Update all collections
+qmd update
+
+# Update specific collection (if supported)
+# qmd update --collection docs
+
+# Force full rebuild
+rm -rf ~/.cache/qmd
+qmd update
+```
+
+### Searching
+
+```bash
+# BM25 keyword search (fast, exact matching)
+qmd search "authentication"
+qmd search "OAuth flow"
+qmd search "API endpoint"
+
+# Vector semantic search (requires Ollama)
+qmd vsearch "how to login"
+qmd vsearch "error handling patterns"
+qmd vsearch "database connection setup"
+
+# Hybrid search (BM25 + vector + rerank - best results)
+qmd query "authentication implementation"
+qmd query "error handling"
+qmd query "API design patterns"
+```
+
+### Document Retrieval
+
+```bash
+# Get specific document
+qmd get docs/api.md
+
+# Get with line number and context (10 lines before/after line 50)
+qmd get docs/api.md:50 -l 10
+
+# Get by search result ID
+qmd search "authentication" | grep "docs/auth.md"
+qmd get docs/auth.md:42 -l 20
+```
+
+### Status and Info
+
+```bash
+# Check index status
+qmd status
+
+# List collections
+qmd collection list
+
+# Check Ollama connection
+curl http://localhost:11434/api/tags
+```
+
+---
+
+## MCP Mode (Model Context Protocol)
+
+qmd can run as an MCP server, allowing AI tools to search your indexed content:
+
+```bash
+# Start MCP server
+qmd mcp
+
+# MCP server listens for requests from AI tools
+# Tools like Claude Desktop can connect and search your docs
+```
+
+### MCP Configuration for Claude Desktop
+
+Add to Claude Desktop configuration:
+
+```json
+{
+  "mcpServers": {
+    "qmd": {
+      "command": "qmd",
+      "args": ["mcp"],
+      "env": {
+        "OLLAMA_URL": "http://localhost:11434"
+      }
+    }
+  }
+}
+```
+
+---
+
+## Integration with Other Skills
+
+| Skill | Integration | Notes |
+|-------|-------------|-------|
+| **ollama** | Embeddings and reranking | Required for vector search |
+| **cron** | Schedule index updates | Keep index fresh automatically |
+| **comms** | Index communication history | Search across all messages |
+| **aix** | Index AI sessions | Search past AI conversations |
+| **nexus-cloud** | Sync qmd index | Share index across devices |
+
+---
+
+## Testing Checklist
+
+- [ ] Fresh install: `npm install -g https://github.com/tobi/qmd`
+- [ ] Version check: `qmd --version`
+- [ ] Ollama running: `curl http://localhost:11434/api/tags`
+- [ ] Embedding model pulled: `ollama list | grep nomic-embed-text`
+- [ ] Collection added: `qmd collection add ~/test --name test`
+- [ ] Index built: `qmd update`
+- [ ] Status shows data: `qmd status`
+- [ ] BM25 search works: `qmd search "test"`
+- [ ] Vector search works: `qmd vsearch "test"`
+- [ ] Hybrid search works: `qmd query "test"`
+- [ ] Document retrieval works: `qmd get <doc-path>`
+- [ ] MCP mode starts: `qmd mcp`
+- [ ] Scheduled update works: Test cron job
+
+---
+
+## Notes for Onboarding Agent
+
+- **Ollama is key**: Emphasize vector search requires Ollama (optional but recommended)
+- **Index updates**: Remind users to run `qmd update` after adding files
+- **When to recommend**:
+  - User has large codebase or documentation
+  - User wants semantic search capabilities
+  - User uses AI tools that support MCP
+  - User needs to search notes, wikis, or knowledge bases
+- **When NOT to recommend**:
+  - User has small, well-organized files (grep is enough)
+  - User doesn't need semantic search
+  - User uncomfortable installing npm packages from GitHub
+- **Value proposition**: "Search your code and docs by meaning, not just keywords"
+- **Time estimate**:
+  - Installation: 2 minutes
+  - Ollama setup: 5 minutes (if not installed)
+  - First indexing: 5-15 minutes (depends on collection size)
+  - Configuration: 3 minutes
+- **Performance note**: Initial indexing can take time for large collections; subsequent updates are incremental
+- **Storage**: Index stored in `~/.cache/qmd/` (can be large for big collections)
+
+---
+
+## Architecture: How It Works
+
+### Indexing Pipeline
+
+```
+Files on disk
+  ↓
+qmd collection add (define what to index)
+  ↓
+qmd update
+  ↓
+1. Scan directories with file masks
+2. Extract text from files
+3. Build BM25 inverted index
+4. Generate embeddings (Ollama)
+5. Store in SQLite database (~/.cache/qmd/)
+  ↓
+Index ready for search
+```
+
+### Search Pipeline
+
+```
+User query
+  ↓
+qmd search (BM25 only)
+  ↓
+1. Tokenize query
+2. Search inverted index
+3. Rank by TF-IDF
+  ↓
+Return results
+
+OR
+
+User query
+  ↓
+qmd vsearch (Vector only)
+  ↓
+1. Generate query embedding (Ollama)
+2. Vector similarity search
+3. Rank by cosine similarity
+  ↓
+Return results
+
+OR
+
+User query
+  ↓
+qmd query (Hybrid)
+  ↓
+1. BM25 search → candidates
+2. Vector search → candidates
+3. Merge results
+4. Rerank with LLM (Ollama)
+  ↓
+Return best results
+```
+
+---
+
+## Performance Considerations
+
+### Indexing Performance
+
+- **Small collections** (<1k files): ~1-2 minutes
+- **Medium collections** (1k-10k files): ~5-10 minutes
+- **Large collections** (10k+ files): ~15-30 minutes
+- **Incremental updates**: Fast (only changed files)
+
+### Search Performance
+
+- **BM25 search**: <100ms for millions of documents
+- **Vector search**: ~100-500ms (depends on collection size)
+- **Hybrid search**: ~500ms-1s (includes reranking)
+
+### Storage
+
+- **BM25 index**: ~10-20% of original text size
+- **Vector embeddings**: ~4KB per document chunk
+- **Total**: Expect ~20-30% of original collection size
+
+---
+
+## Troubleshooting Guide
+
+### Search Returns Wrong Results
+
+```bash
+# Check if index is up to date
+qmd status
+
+# Rebuild index
+qmd update
+
+# Try different search methods
+qmd search "query"    # Exact keyword matching
+qmd vsearch "query"   # Semantic matching
+qmd query "query"     # Hybrid (best results)
+```
+
+### Slow Search Performance
+
+```bash
+# Check collection size
+qmd status
+
+# Consider splitting large collections
+qmd collection remove large-collection
+qmd collection add ~/large-dir/subdir1 --name subdir1
+qmd collection add ~/large-dir/subdir2 --name subdir2
+
+# Check Ollama performance
+time ollama run nomic-embed-text "test query"
+```
+
+### High Memory Usage
+
+```bash
+# Check cache size
+du -sh ~/.cache/qmd
+
+# Clear cache and rebuild
+rm -rf ~/.cache/qmd
+qmd update
+
+# Reduce collection size (add more exclusions)
+qmd collection add ~/code --name code --mask "**/*.py" --exclude "**/venv/**" --exclude "**/node_modules/**"
+```
+
+---
+
+## Advanced: Custom Embedding Models
+
+```bash
+# Use different embedding model
+export OLLAMA_EMBED_MODEL="mxbai-embed-large"
+
+# Pull model
+ollama pull mxbai-embed-large
+
+# Rebuild index with new embeddings
+rm -rf ~/.cache/qmd
+qmd update
+```
+
+---
+
+## Changelog
+
+| Date | Change |
+|------|--------|
+| 2026-01-12 | Initial deep-dive created |
+
+---
+
+*Template version: 1.0*