mdcontext 0.0.1 → 0.1.0

package/research/semantic-search/032-research-semantic-search.md

@@ -0,0 +1,427 @@
+# Research Task Analysis: Embedding Models, RAG Alternatives, and Vector Search
+
+Analysis of research documents against current mdcontext implementation.
+
+Date: January 2026
+
+---
+
+## Documents Analyzed
+
+1. `002-research-embedding-models.md` - Embedding model comparison and recommendations
+2. `003-research-rag-alternatives.md` - RAG alternatives for improving semantic search
+3. `004-research-vector-search.md` - Vector search patterns and techniques
+
+---
+
+## Implemented (No Action Needed)
+
+### 1. Dimension Reduction (512 dimensions)
+
+**Research Recommendation:** Reduce OpenAI embeddings from 1536 to 512 dimensions for 67% storage reduction with minimal quality loss.
+
+**Current Implementation:** Already implemented in `src/embeddings/openai-provider.ts`:
+
+```typescript
+const response = await this.client.embeddings.create({
+  model: this.model,
+  input: batch,
+  dimensions: 512, // Already using reduced dimensions
+});
+```
+
+**Status:** Implemented
+
+---
+
+### 2. HNSW Vector Index
+
+**Research Recommendation:** Stay with HNSW for documentation corpora (<100K sections).
+
+**Current Implementation:** Using `hnswlib-node` with cosine similarity in `src/embeddings/vector-store.ts`:
+
+```typescript
+this.index = new HierarchicalNSW.HierarchicalNSW("cosine", this.dimensions);
+this.index.initIndex(10000, 16, 200, 100); // M=16, efConstruction=200
+```
+
+**Status:** Implemented
+
+---
+
+### 3. EmbeddingProvider Interface
+
+**Research Recommendation:** Create provider abstraction for embedding models.
+
+**Current Implementation:** `src/embeddings/types.ts` defines a clean provider interface:
+
+```typescript
+export interface EmbeddingProvider {
+  readonly name: string;
+  readonly dimensions: number;
+  embed(texts: string[]): Promise<EmbeddingResult>;
+}
+```
+
+**Status:** Implemented (foundation ready for additional providers)
+
+---
+
+### 4. Path Pattern Filtering (Post-filter)
+
+**Research Recommendation:** Implement metadata filtering for search results.
+
+**Current Implementation:** `pathPattern` option implemented in `semanticSearch()` as post-filtering.
+
+**Status:** Implemented (basic)
+
+---
+
+### 5. Document Context in Embeddings
+
+**Research Recommendation:** Include document title and parent section in embedding text.
+
+**Current Implementation:** Already in `src/embeddings/semantic-search.ts`:
+
+```typescript
+const generateEmbeddingText = (
+  section,
+  content,
+  documentTitle,
+  parentHeading,
+) => {
+  parts.push(`# ${section.heading}`);
+  if (parentHeading) parts.push(`Parent section: ${parentHeading}`);
+  parts.push(`Document: ${documentTitle}`);
+  parts.push(content);
+  // ...
+};
+```
+
+**Status:** Implemented
+
+---
+
+## Task Candidates
+
+### 1. Add Hybrid Search (BM25 + Semantic)
+
+**Priority:** High
+
+**Description:**
+Implement hybrid search combining BM25 keyword search with semantic search, using Reciprocal Rank Fusion (RRF) to merge results.
+
+**Why It Matters:**
+
+- Research shows 15-30% recall improvement over single-method retrieval
+- Handles exact term matching (API names, error codes, identifiers) that pure semantic search misses
+- Current keyword search exists separately but isn't integrated with semantic search
+
+**Implementation Notes:**
+
+- Add `wink-bm25-text-search` dependency
+- Build BM25 index alongside vector index during `mdcontext embed`
+- Add `--mode hybrid` option to search command
+- Implement RRF fusion (~50 lines of code)
+
+**Current Gap:**
+
+- Keyword search (`src/search/searcher.ts`) and semantic search (`src/embeddings/semantic-search.ts`) are separate codepaths
+- No fusion mechanism exists
+
+**Estimated Effort:** 2-3 days
+
+---
+
+### 2. Add Local Embedding Provider (Ollama)
+
+**Priority:** High
+
+**Description:**
+Implement an Ollama-based embedding provider using `nomic-embed-text-v1.5` for offline semantic search.
+
+**Why It Matters:**
+
+- Enables offline semantic search (major feature gap)
+- Zero ongoing API costs
+- Quality matches or exceeds OpenAI text-embedding-3-small
+- Privacy-sensitive use cases
+
+**Implementation Notes:**
+
+- Create `src/embeddings/ollama-provider.ts` implementing `EmbeddingProvider`
+- Add provider selection via config or `--provider` CLI flag
+- Default to OpenAI for backward compatibility
+- nomic-embed-text supports Matryoshka (dimension flexibility)
+
+**Models to Support:**
+
+1. `nomic-embed-text` - Best overall fit (fast, 8K context, Matryoshka)
+2. `mxbai-embed-large` - Higher quality option
+3. `bge-m3` - Multilingual option
+
+**Estimated Effort:** 2-3 days
+
+---
+
+### 3. Add Cross-Encoder Re-ranking
+
+**Priority:** Medium
+
+**Description:**
+Add optional re-ranking of top-N semantic search results using a cross-encoder model.
+
+**Why It Matters:**
+
+- 20-35% accuracy improvement in retrieval precision
+- Cross-encoders capture fine-grained relevance that bi-encoders miss
+- Can be opt-in to avoid latency when not needed
+
+**Implementation Notes:**
+
+- Add `@xenova/transformers` dependency for Transformers.js
+- Use `ms-marco-MiniLM-L-6-v2` model (22.7M params, 2-5ms/pair)
+- Re-rank top-20 candidates to top-10
+- Add `--rerank` flag to search command
+
+**Alternative:** Cohere Rerank API for simpler integration (adds cost)
+
+**Estimated Effort:** 2-3 days
+
+---
+
+### 4. Add Dynamic efSearch (Quality Modes)
+
+**Priority:** Medium
+
+**Description:**
+Allow users to control search quality/speed tradeoff via HNSW efSearch parameter at query time.
+
+**Why It Matters:**
+
+- Zero dependency changes
+- Immediate quality/speed improvements
+- Low risk
+
+**Implementation Notes:**
+
+- Add `--quality` flag: `fast` (64), `balanced` (100), `thorough` (256)
+- efSearch is already configurable at query time in hnswlib-node
+- Update search functions to accept quality parameter
+
+**Current State:**
+
+```typescript
+this.index.initIndex(10000, 16, 200, 100); // efSearch=100 (implicit)
+```
+
+**Estimated Effort:** 0.5 days
+
+---
+
+### 5. Add Configurable HNSW Parameters
+
+**Priority:** Low
+
+**Description:**
+Expose HNSW build parameters (M, efConstruction) via configuration for users with specific needs.
+
+**Why It Matters:**
+
+- Users with large corpora may want to tune for speed
+- Users needing maximum recall can increase parameters
+- Enables benchmarking different configurations
+
+**Current Hardcoded Values:**
+
+```typescript
+M: 16; // Max connections per node
+efConstruction: 200; // Construction-time search width
+```
+
+**Recommended Configurations:**
+
+- Quality-focused: M=24, efConstruction=256
+- Speed-focused: M=12, efConstruction=128
+
+**Estimated Effort:** 1 day
+
+---
+
+### 6. Add Query Preprocessing
+
+**Priority:** Low
+
+**Description:**
+Add basic query preprocessing before embedding to reduce noise.
+
+**Why It Matters:**
+
+- 2-5% precision improvement
+- Simple implementation
+
+**Implementation:**
+
+```typescript
+function preprocessQuery(query: string): string {
+  return query
+    .toLowerCase()
+    .replace(/[^\w\s]/g, " ")
+    .replace(/\s+/g, " ")
+    .trim();
+}
+```
+
+**Estimated Effort:** 1-2 hours
+
+---
+
+### 7. Add Heading Match Boost
+
+**Priority:** Low
+
+**Description:**
+Boost search results when query terms appear in section headings.
+
+**Why It Matters:**
+
+- Significant for navigation queries ("installation guide", "API reference")
+- Simple scoring adjustment
+
+**Implementation:**
+
+```typescript
+function adjustScore(result, query): number {
+  const queryTerms = query.toLowerCase().split(/\s+/);
+  const headingLower = result.heading.toLowerCase();
+  const headingMatches = queryTerms.filter((t) =>
+    headingLower.includes(t),
+  ).length;
+  return result.similarity + headingMatches * 0.05;
+}
+```
+
+**Estimated Effort:** 2-4 hours
+
+---
+
+### 8. Add HyDE Query Expansion (Optional)
+
+**Priority:** Low
+
+**Description:**
+Implement Hypothetical Document Embeddings for complex queries - generate a hypothetical answer with LLM, then search using that embedding.
+
+**Why It Matters:**
+
+- 10-30% retrieval improvement on ambiguous queries
+- Bridges semantic gap between short questions and detailed documents
+
+**Considerations:**
+
+- Adds LLM call (cost, latency)
+- Should be opt-in for complex queries only
+- Works poorly if LLM lacks domain knowledge
+
+**Estimated Effort:** 1-2 days
+
+---
+
+### 9. Fix Dimension Mismatch in Provider
+
+**Priority:** Medium
+
+**Description:**
+The OpenAI provider reports incorrect dimensions (1536/3072) while actually using 512.
+
+**Current Issue:**
+
+```typescript
+// In openai-provider.ts
+this.dimensions = this.model === "text-embedding-3-large" ? 3072 : 1536;
+// But actual API call uses:
+dimensions: 512;
+```
+
+This mismatch could cause issues if other code relies on `provider.dimensions`.
+
+**Fix:** Update dimension reporting to match actual API parameter.
+
+**Estimated Effort:** 0.5 hours
+
+---
+
+### 10. Add Alternative API Provider (Voyage AI)
+
+**Priority:** Low
+
+**Description:**
+Add Voyage AI as an alternative embedding provider for users wanting better quality at similar cost.
+
+**Why It Matters:**
+
+- voyage-3.5-lite: Same price as OpenAI ($0.02/1M), but better quality
+- 32K token context (4x OpenAI)
+- Free 200M tokens for testing
+
+**Estimated Effort:** 1 day
+
+---
+
+## Skip (Not Applicable)
+
+| Recommendation           | Reason to Skip                                                        |
+| ------------------------ | --------------------------------------------------------------------- |
+| ColBERT Late Interaction | Overkill for documentation corpus sizes; requires Python service      |
+| SPLADE Sparse Retrieval  | BM25 + semantic hybrid likely sufficient; adds complexity             |
+| GraphRAG                 | Overkill for documentation search                                     |
+| Fine-tuned Embeddings    | Requires training infrastructure; general models work well            |
+| IVF/DiskANN Indexes      | HNSW sufficient for typical documentation sizes (<100K sections)      |
+| LLM-based Re-ranking     | Cross-encoders provide similar quality without LLM cost/latency       |
+| Self-RAG                 | Beyond current scope; more relevant for RAG pipelines with generation |
+
+---
+
+## Summary
+
+| Category        | Count             |
+| --------------- | ----------------- |
+| Implemented     | 5 items           |
+| Task Candidates | 10 items          |
+| Skipped         | 7 recommendations |
+
+### Priority Matrix
+
+| Priority   | Tasks                                                                     |
+| ---------- | ------------------------------------------------------------------------- |
+| **High**   | Hybrid Search (BM25+RRF), Local Embedding Provider (Ollama)               |
+| **Medium** | Cross-Encoder Re-ranking, Dynamic efSearch, Fix Dimension Mismatch        |
+| **Low**    | HNSW Config, Query Preprocessing, Heading Boost, HyDE, Voyage AI Provider |
+
+### Recommended Implementation Order
+
+**Phase 1: Quick Wins (1 week)**
+
+1. Fix dimension mismatch in provider
+2. Add dynamic efSearch (quality modes)
+3. Add query preprocessing
+4. Add heading match boost
+
+**Phase 2: Hybrid Search (1-2 weeks)**
+
+1. Integrate BM25 library
+2. Build BM25 index during embed
+3. Implement RRF fusion
+4. Add `--mode hybrid` to CLI
+
+**Phase 3: Local/Offline (1-2 weeks)**
+
+1. Implement Ollama provider
+2. Add provider selection CLI
+3. Test with nomic-embed-text
+
+**Phase 4: Advanced (2 weeks)**
+
+1. Cross-encoder re-ranking (Transformers.js)
+2. HyDE query expansion (optional)
+3. Alternative API providers (Voyage AI)

package/research/task-management-2026/00-synthesis-recommendations.md

@@ -0,0 +1,295 @@
+# Task Management for Ralph: Synthesis & Recommendations
+
+**Date:** January 21, 2026
+**Purpose:** Recommend a task management approach for capturing intent before AI agent (ralph) processing
+
+---
+
+## Executive Summary
+
+**For capturing task intent in an AI agent workflow, use a file-based SPEC.md pattern stored in Git.** This approach offers the best balance of human usability, LLM compatibility, and zero friction. External tools like Linear or GitHub Issues add unnecessary complexity for the core problem of "capture intent quickly so ralph can act on it." The SPEC.md pattern is already proven in the codebase, requires no new tooling, and aligns with the emerging industry standard of spec-driven development. Reserve external tools (Linear, GitHub Issues) for team coordination and visibility, not for the AI agent interface.
+
+---
+
+## Context
+
+### The Problem
+
+We need a system for capturing task intent before processing through "ralph" (an AI agent orchestration system). This is specifically about the **input interface** to the agent, not project management at large.
+
+### Requirements
+
+| Requirement                 | Priority | Notes                                   |
+| --------------------------- | -------- | --------------------------------------- |
+| Capture task intent quickly | Critical | Friction kills adoption                 |
+| LLM-friendly format         | Critical | Ralph must parse and understand it      |
+| Git-native                  | High     | Version control, audit trail, offline   |
+| Low friction for humans     | High     | Must be faster than "just do it myself" |
+| Scales to many tasks        | Medium   | Backlog accumulation, parallel work     |
+
+### Constraint
+
+Ralph operates on files in a repository. Whatever system we choose must either:
+
+1. **Live in the repo** (file-based), or
+2. **Be accessible via MCP/API** (external tool)
+
+---
+
+## Options Matrix
+
+### Top 4 Approaches Evaluated
+
+| Approach            | Quick Capture | LLM-Friendly | Git-Native | Low Friction | Scales | Overall |
+| ------------------- | ------------- | ------------ | ---------- | ------------ | ------ | ------- |
+| **SPEC.md Pattern** | A             | A            | A          | A            | B      | **A**   |
+| **Backlog.md**      | B             | A            | A          | B            | A      | **A-**  |
+| **Linear**          | B             | B            | C          | A            | A      | **B+**  |
+| **GitHub Issues**   | C             | B            | B          | B            | A      | **B**   |
+
+### Detailed Analysis
+
+#### 1. SPEC.md Pattern (File-Based Specifications)
+
+**What it is:** Markdown files in the repo (e.g., `SPEC.md`, `TODO.md`, `PLAN.md`) that define task intent. Already emerging as the industry standard for AI-assisted development in 2026.
+
+**Pros:**
+
+- Zero dependencies - works with any editor, any AI tool
+- Direct context injection into LLM prompts (no API calls)
+- Git versioning provides audit trail and rollback for free
+- Human and AI edit the same artifact
+- Token-efficient format (markdown is LLM-native)
+- Already using something similar in the codebase
+
+**Cons:**
+
+- No built-in visualization (Kanban, timelines)
+- Requires discipline to maintain structure
+- Doesn't scale well for team-wide coordination across multiple projects
+
+**Best for:** Individual or small team AI coding workflows where the developer is the primary user.
+
+#### 2. Backlog.md
+
+**What it is:** Purpose-built tool for AI-human collaboration. Each task is a separate `.md` file in a `.backlog/` directory. CLI and web interface available.
+
+**Pros:**
+
+- Designed specifically for AI agent workflows
+- Git-native with task IDs referencing commits/branches
+- Terminal Kanban for visualization
+- Works with Claude Code, Cursor, and MCP-compatible tools
+- Open source, no vendor lock-in
+
+**Cons:**
+
+- Adds a tool/CLI dependency
+- Slightly more structure than bare SPEC.md
+- Newer tool with smaller community
+
+**Best for:** Teams wanting more structure than plain markdown but still git-native.
+
+#### 3. Linear
+
+**What it is:** The leading AI-native project management tool. Treats AI agents as "full teammates" with native integrations to Cursor, Copilot, and Claude.
+
+**Pros:**
+
+- Best-in-class AI agent integration (agents as assignable teammates)
+- Excellent API and MCP servers
+- Team visibility and coordination features
+- Fast, keyboard-first UI
+
+**Cons:**
+
+- External service (not in repo)
+- Requires MCP/API integration for ralph to access
+- Adds latency to task capture (open tool, create issue)
+- Paid at scale ($8/user/month)
+- Another tool to maintain
+
+**Best for:** Team coordination, roadmap visibility, when multiple people need to see task status.
+
+#### 4. GitHub Issues
+
+**What it is:** GitHub's built-in issue tracker, now with Copilot coding agent that can be assigned issues directly.
+
+**Pros:**
+
+- Already using GitHub
+- Copilot can autonomously work on issues
+- No new tool adoption
+- Free
+
+**Cons:**
+
+- Slow to create issues (web UI friction)
+- Not optimized for quick intent capture
+- Less LLM-friendly than markdown files
+- Requires API calls for ralph to read
+
+**Best for:** When you need the issue-to-PR workflow with Copilot, team-visible bug tracking.
+
+---
+
+## Recommendation
+
+### Primary: SPEC.md Pattern
+
+**Use file-based specifications as the primary interface between humans and ralph.**
+
+#### Rationale
+
+1. **Lowest friction for capture:** Open file, write intent, save. Done. No context switching to another tool.
+
+2. **Best LLM compatibility:** Markdown is the native language of LLMs. No translation layer, no API calls, no token overhead from structured formats.
+
+3. **Already proven:** The research shows spec-driven development is the emerging standard. GitHub Spec-Kit, JetBrains Junie, Amazon Kiro - all use this pattern.
+
+4. **Git-native by default:** Every change is versioned. Branching allows parallel task exploration. History shows what was attempted and why.
+
+5. **Zero new tooling:** Works today with the existing setup. No adoption curve, no dependencies to maintain.
+
+6. **Context window optimization:** Files can be selectively loaded. Large backlogs don't need to be sent to the LLM - only the relevant spec.
+
+#### Proposed File Structure
+
+```
+.ralph/
+  BACKLOG.md          # Quick task capture (one-liner ideas)
+  active/
+    feature-x.spec.md # Detailed spec for in-progress work
+    bug-fix-y.spec.md
+  completed/          # Archived specs (reference/learning)
+  templates/
+    feature.spec.md   # Template for new features
+    bugfix.spec.md    # Template for bug fixes
+```
+
+#### Spec File Format
+
+```markdown
+# [Task Title]
+
+## Intent
+
+[One paragraph: What do we want to accomplish and why?]
+
+## Success Criteria
+
+- [ ] Criterion 1
+- [ ] Criterion 2
+- [ ] Criterion 3
+
+## Context
+
+[Any relevant background, links, constraints]
+
+## Notes
+
+[Optional: implementation hints, considerations]
+```
+
+### Secondary: Linear for Team Visibility
+
+**If you need team coordination or external visibility, sync completed specs to Linear.**
+
+- Use Linear for roadmap and sprint planning
+- Use Linear for stakeholder visibility
+- Keep ralph's input interface file-based
+- Consider automation: completed specs could auto-create Linear issues for tracking
+
+### Why Not Linear as Primary?
+
+Linear is excellent, but it adds friction to the core workflow:
+
+1. **Context switch:** You're in your editor, you have an idea, you have to open Linear
+2. **API dependency:** Ralph needs MCP/API calls instead of file reads
+3. **Not the source of truth:** The code is in git, the spec should be too
+4. **Overkill for capture:** Linear is optimized for team coordination, not quick intent capture
+
+Linear makes sense for _visibility and coordination_, not for _AI agent input_.
+
+### Why Not Backlog.md?
+
+Backlog.md is well-designed and could work. However:
+
+1. **Adds a dependency** for marginal benefit over plain markdown
+2. **Learning curve** for the CLI and conventions
+3. **SPEC.md is simpler** and already aligns with industry patterns
+
+If the team grows or the backlog becomes complex, Backlog.md is a good upgrade path.
+
+---
+
+## Implementation Path
+
+### Phase 1: Establish the Pattern (Day 1)
+
+1. Create `.ralph/` directory structure
+2. Create `BACKLOG.md` for quick capture
+3. Create spec templates
+4. Document the pattern in CLAUDE.md
+
+### Phase 2: Integrate with Ralph (Week 1)
+
+1. Update ralph to look for specs in `.ralph/active/`
+2. Implement spec parsing (extract intent, success criteria)
+3. Add success criteria tracking (ralph marks criteria complete)
+4. Move completed specs to `.ralph/completed/`
+
+### Phase 3: Optimize (Ongoing)
+
+1. Refine templates based on what works
+2. Consider auto-sync to Linear for visibility (if needed)
+3. Add tooling for quick spec creation (CLI or editor shortcuts)
+
+### Example Workflow
+
+```bash
+# Human captures intent quickly
+echo "# Add dark mode support
+
+## Intent
+Users want dark mode to reduce eye strain during night coding sessions.
+
+## Success Criteria
+- [ ] Toggle in settings
+- [ ] Persists across sessions
+- [ ] Respects system preference by default
+" > .ralph/active/dark-mode.spec.md
+
+# Ralph picks it up and executes
+ralph process .ralph/active/dark-mode.spec.md
+```
+
+---
+
+## Decision Summary
+
+| Decision                | Choice                              | Confidence |
+| ----------------------- | ----------------------------------- | ---------- |
+| Primary task capture    | SPEC.md files in `.ralph/`          | High       |
+| Format                  | Markdown with structured sections   | High       |
+| Location                | Git repository                      | High       |
+| Team coordination       | Linear (optional, secondary)        | Medium     |
+| External issue tracking | GitHub Issues (for bugs, community) | Medium     |
+
+---
+
+## Sources
+
+This synthesis is based on four research documents:
+
+1. **01-ai-workflow-tools.md** - Survey of AI-native task management tools (Linear, Taskade, Backlog.md, Plane.so, etc.)
+2. **02-agent-framework-patterns.md** - How AI agent frameworks (LangGraph, CrewAI, Claude Code) handle task persistence
+3. **03-lightweight-file-based.md** - File-based approaches (SPEC.md, Beads, todo.txt, etc.)
+4. **04-established-tools-ai-features.md** - AI features in GitHub, Linear, Jira, Notion
+
+Key external sources informing this recommendation:
+
+- [GitHub Spec-Kit](https://github.com/github/spec-kit/blob/main/spec-driven.md)
+- [Addy Osmani on specs for AI agents](https://addyosmani.com/blog/good-spec/)
+- [Steve Yegge on coding agent memory](https://steve-yegge.medium.com/introducing-beads-a-coding-agent-memory-system-637d7d92514a)
+- [Thoughtworks on Spec-Driven Development](https://www.thoughtworks.com/en-us/insights/blog/agile-engineering-practices/spec-driven-development-unpacking-2025-new-engineering-practices)