mdcontext 0.0.1 → 0.2.0

package/docs/020-current-implementation.md

@@ -0,0 +1,364 @@
+# mdcontext Semantic Search: Current Implementation
+
+This document describes the current semantic search implementation in mdcontext, covering architecture, components, data flow, and known limitations.
+
+## Overview
+
+mdcontext provides semantic search capabilities that allow users to search markdown documentation by meaning rather than exact text matching. The system uses OpenAI's text-embedding-3-small model to generate vector embeddings and HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor search.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                           CLI Layer                                      │
+│  src/cli/commands/search.ts                                             │
+│  - Mode detection (semantic vs keyword)                                 │
+│  - Auto-index prompt for missing embeddings                             │
+│  - Result formatting and display                                        │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               │
+┌──────────────────────────────▼──────────────────────────────────────────┐
+│                        Semantic Search Layer                             │
+│  src/embeddings/semantic-search.ts                                      │
+│  - Cost estimation (estimateEmbeddingCost)                              │
+│  - Embedding generation (buildEmbeddings)                               │
+│  - Query execution (semanticSearch, semanticSearchWithContent)          │
+│  - Statistics (getEmbeddingStats)                                       │
+└─────────────┬─────────────────────────────────────┬─────────────────────┘
+              │                                     │
+┌─────────────▼───────────────┐     ┌───────────────▼─────────────────────┐
+│      Embedding Provider      │     │           Vector Store              │
+│  src/embeddings/             │     │  src/embeddings/vector-store.ts     │
+│    openai-provider.ts        │     │  - HNSW index (hnswlib-node)        │
+│  - OpenAI API integration    │     │  - Cosine similarity search         │
+│  - text-embedding-3-small    │     │  - Binary index persistence         │
+│  - Batch processing (100)    │     │  - Metadata JSON storage            │
+└──────────────────────────────┘     └─────────────────────────────────────┘
+```
+
+## Components
+
+### 1. Embedding Provider (`src/embeddings/openai-provider.ts`)
+
+**Current Provider**: OpenAI `text-embedding-3-small`
+
+| Property   | Value                    |
+| ---------- | ------------------------ |
+| Model      | `text-embedding-3-small` |
+| Dimensions | 1536                     |
+| Batch Size | 100 texts per API call   |
+| Cost       | $0.02 per 1M tokens      |
+
+**Interface**:
+
+```typescript
+interface EmbeddingProvider {
+  readonly name: string; // e.g., "openai:text-embedding-3-small"
+  readonly dimensions: number; // 1536 for small, 3072 for large
+  embed(texts: string[]): Promise<EmbeddingResult>;
+}
+
+interface EmbeddingResult {
+  readonly embeddings: readonly number[][];
+  readonly tokensUsed: number;
+  readonly cost: number;
+}
+```
+
+**Supported Models**:
+
+- `text-embedding-3-small` (default): 1536 dimensions, $0.02/1M tokens
+- `text-embedding-3-large`: 3072 dimensions, $0.13/1M tokens
+- `text-embedding-ada-002` (legacy): 1536 dimensions, $0.10/1M tokens
+
+### 2. Vector Store (`src/embeddings/vector-store.ts`)
+
+**Implementation**: HNSW via `hnswlib-node`
+
+| Parameter        | Value    | Description                                  |
+| ---------------- | -------- | -------------------------------------------- |
+| Space            | `cosine` | Cosine similarity distance metric            |
+| Initial Capacity | 10,000   | Auto-resizes by 2x when full                 |
+| M                | 16       | Max connections per node (default)           |
+| efConstruction   | 200      | Construction-time search width               |
+| efSearch         | 100      | Query-time search width (implicit from init) |
+
+**Storage Format**:
+
+- `vectors.bin`: Binary HNSW index file
+- `vectors.meta.json`: Metadata including entries, costs, timestamps
+
+**Vector Entry Structure**:
+
+```typescript
+interface VectorEntry {
+  readonly id: string; // Section ID
+  readonly sectionId: string; // Same as id
+  readonly documentPath: string; // Relative path to document
+  readonly heading: string; // Section heading text
+  readonly embedding: readonly number[]; // 1536-dimensional vector
+}
+```
+
+**Similarity Calculation**:
+
+- HNSW stores cosine distance (1 - similarity)
+- Search returns `similarity = 1 - distance`
+- Results filtered by threshold (default: 0.35)
+
+### 3. Semantic Search (`src/embeddings/semantic-search.ts`)
+
+**Text Generation for Embeddings**:
+
+Each section is embedded with contextual metadata:
+
+```
+# {heading}
+Parent section: {parentHeading}  // if nested
+Document: {documentTitle}
+
+{full section content}
+```
+
+**Filtering**:
+
+- Sections with < 10 tokens are skipped
+- Exclude patterns can filter by document path
+
+**Search Flow**:
+
+1. Load vector store from disk
+2. Embed query using same provider
+3. kNN search with limit \* 2 (over-fetch for filtering)
+4. Apply path pattern filter if specified
+5. Return top `limit` results above threshold
+
+### 4. CLI Search Command (`src/cli/commands/search.ts`)
+
+**Mode Detection Priority**:
+
+1. `--mode semantic` or `--mode keyword` (explicit)
+2. `--keyword` flag (force keyword)
+3. Boolean/phrase pattern detected (`AND`, `OR`, `NOT`, `"quoted"`)
+4. Regex pattern detected (special characters)
+5. Embeddings available → semantic
+6. No embeddings → keyword
+
+**Auto-Index Behavior**:
+
+- If semantic mode requested but no embeddings exist:
+  - Estimate time/cost
+  - If < 10 seconds: auto-create silently
+  - Otherwise: prompt user for choice
+
+**Default Search Threshold**: 0.35 (raised from 0.3 to filter low-quality matches)
+
+## Data Flow
+
+### Building Embeddings
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 1. Load Indexes                                                          │
+│    - documents.json (document metadata)                                  │
+│    - sections.json (section index with line numbers)                     │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 2. Group Sections by Document                                            │
+│    - Skip sections < 10 tokens                                           │
+│    - Apply exclude patterns                                              │
+│    - Track parent headings for context                                   │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 3. Read File Content                                                     │
+│    - For each document, read file                                        │
+│    - Extract section content by line numbers                             │
+│    - Generate embedding text with metadata                               │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 4. Generate Embeddings                                                   │
+│    - Send all texts to OpenAI API (batched by 100)                       │
+│    - Track token usage and cost                                          │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 5. Build HNSW Index                                                      │
+│    - Add vectors with sequential integer IDs                             │
+│    - Map section IDs to index positions                                  │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 6. Persist to Disk                                                       │
+│    - vectors.bin (HNSW binary)                                           │
+│    - vectors.meta.json (metadata + entries)                              │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+### Query Execution
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 1. Query Input                                                           │
+│    "How do I configure authentication?"                                  │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 2. Load Vector Store                                                     │
+│    - Read vectors.bin into HNSW index                                    │
+│    - Load metadata from vectors.meta.json                                │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 3. Embed Query                                                           │
+│    - Single API call to OpenAI                                           │
+│    - Returns 1536-dim vector                                             │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 4. HNSW kNN Search                                                       │
+│    - Find k nearest neighbors (cosine similarity)                        │
+│    - Over-fetch: request limit * 2                                       │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 5. Post-Processing                                                       │
+│    - Filter by similarity threshold (default: 0.35)                      │
+│    - Filter by path pattern (if specified)                               │
+│    - Truncate to requested limit                                         │
+└──────────────────────────────┬──────────────────────────────────────────┘
+                               ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 6. Return Results                                                        │
+│    [{sectionId, documentPath, heading, similarity}, ...]                │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+## Storage Files
+
+Located in `.mdcontext/` directory:
+
+| File                | Format | Contents                                       |
+| ------------------- | ------ | ---------------------------------------------- |
+| `vectors.bin`       | Binary | HNSW index (hnswlib native format)             |
+| `vectors.meta.json` | JSON   | Entry metadata, costs, timestamps              |
+| `documents.json`    | JSON   | Document index (title, path, stats)            |
+| `sections.json`     | JSON   | Section index (headings, line numbers, tokens) |
+
+## Current Limitations and Gaps
+
+### 1. ~~Single Provider Lock-in~~ RESOLVED (ALP-215)
+
+- **RESOLVED**: Multiple embedding providers now supported (OpenAI, Ollama, LM Studio, OpenRouter)
+- **Impact**: Users can choose local providers for offline capability and cost savings
+- **Code Location**: `provider-factory.ts` handles provider selection based on config
+
+### 2. No Incremental Updates
+
+- **Issue**: `buildEmbeddings` either builds all or skips entirely
+- **Impact**: Adding one document requires re-embedding everything (with `--force`)
+- **Workaround**: Cache hit detection skips if any embeddings exist
+
+### 3. Fixed HNSW Parameters
+
+- **Issue**: HNSW parameters (M=16, efConstruction=200) are hardcoded
+- **Impact**: No tuning for different corpus sizes or quality/speed tradeoffs
+- **Code Location**: `vector-store.ts:94`
+
+### 4. No Hybrid Search
+
+- **Issue**: Semantic and keyword search are mutually exclusive
+- **Impact**: Can't combine exact matches with semantic similarity
+- **Workaround**: Mode auto-detection helps, but no fusion ranking
+
+### 5. No Re-ranking
+
+- **Issue**: Results are pure cosine similarity, no re-ranking
+- **Impact**: May miss contextually relevant results that rank lower in embedding space
+- **Alternative**: Cross-encoder re-ranking could improve precision
+
+### 6. Section-Level Granularity Only
+
+- **Issue**: Embeddings are per-section, no paragraph or sentence chunking
+- **Impact**: Large sections may have diluted embeddings; queries may match subsections better
+- **Tradeoff**: Current approach preserves document structure
+
+### 7. No Query Expansion
+
+- **Issue**: Queries are embedded as-is
+- **Impact**: Synonyms, abbreviations, and related terms may not match
+- **Opportunity**: HyDE or query reformulation could help
+
+### 8. Limited Metadata Filtering
+
+- **Issue**: Only path pattern filtering supported
+- **Impact**: Can't filter by date, author, tags, or other metadata
+- **Code Location**: `semanticSearch` has `pathPattern` option only
+
+### 9. No Batch Query Support
+
+- **Issue**: Each search embeds query individually
+- **Impact**: Multiple searches incur repeated API calls
+- **Opportunity**: Query batching could reduce latency
+
+### 10. Memory Usage
+
+- **Issue**: Entire HNSW index loaded into memory
+- **Impact**: Large corpora may hit memory limits
+- **Note**: Not a problem for typical documentation sizes
+
+## Cost Analysis
+
+For a typical documentation corpus (~1000 sections, ~500K tokens):
+
+| Operation         | Tokens  | Cost       |
+| ----------------- | ------- | ---------- |
+| Initial embedding | ~500K   | ~$0.01     |
+| Per query         | ~50-100 | ~$0.000002 |
+
+The cost is dominated by initial embedding creation. Query costs are negligible.
+
+## Performance Characteristics
+
+| Metric          | Typical Value                  |
+| --------------- | ------------------------------ |
+| Embedding build | ~1.5s per 100 sections         |
+| Query latency   | ~200-500ms (API call dominant) |
+| Index load time | ~50-100ms for 1000 vectors     |
+| Memory usage    | ~10MB per 1000 vectors         |
+
+## Configuration
+
+Current configuration is largely hardcoded. Key values:
+
+```typescript
+// Embedding
+model: 'text-embedding-3-small'     // openai-provider.ts
+batchSize: 100                       // openai-provider.ts
+minTokens: 10                        // semantic-search.ts (skip small sections)
+
+// Vector store
+space: 'cosine'                      // vector-store.ts
+initialCapacity: 10000               // vector-store.ts
+M: 16                                // vector-store.ts
+efConstruction: 200                  // vector-store.ts
+
+// Search
+defaultLimit: 10                     // search.ts
+defaultThreshold: 0.35               // search.ts
+autoIndexThreshold: 10 seconds       // search.ts
+```
+
+## Type Definitions
+
+Full type definitions are in `src/embeddings/types.ts`:
+
+- `EmbeddingProvider`: Provider interface
+- `EmbeddingResult`: Embed response
+- `VectorEntry`: Stored vector with metadata
+- `VectorIndex`: Full index schema
+- `SemanticSearchOptions`: Search parameters
+- `SemanticSearchResult`: Search result item
+- `EmbedError`: Error types (RateLimit, ApiKey, Network, Unknown)

package/docs/021-DOGFOODING-FINDINGS.md

@@ -0,0 +1,175 @@
+# mdcontext Dogfooding Findings
+
+**Date:** 2025-01-19
+**Method:** 6 autonomous agents explored documentation directories using only mdcontext CLI
+**Target directories:** `./docs`, `./doc.llm`, `./docs.amorphic` (in `/Users/alphab/Dev/LLM/DEV/TMP/ralph`)
+
+---
+
+## Executive Summary
+
+**Verdict: YES - mdcontext is useful**, with the `context` command being the standout feature delivering 80-99% token reduction. However, several issues limit the tool's effectiveness for discovery workflows.
+
+---
+
+## What Works Well
+
+### 1. Context Command (Killer Feature)
+
+- Consistently achieved 80-99% token reduction across all test directories
+- Multi-file assembly with budget allocation works as designed
+- `--brief` mode effective for quick overviews
+- JSON output useful for programmatic consumption
+
+### 2. Tree Command
+
+- Excellent for initial codebase discovery
+- Clean hierarchical output
+- Good starting point before diving into specific files
+
+### 3. Help System
+
+- Well-polished with examples
+- Subcommand help (`mdcontext context --help`) informative
+- Agents successfully learned the tool from `--help` alone
+
+---
+
+## Issues Found
+
+### Critical: `index` Command Returns "0 Documents"
+
+**Symptom:**
+
+```bash
+mdcontext index ./docs
+# Output: "Indexed: 0 documents"
+```
+
+**Expected:** Should index markdown files in the directory.
+
+**Impact:** Breaks the discovery workflow. Users can't find content without working index.
+
+**Frequency:** Inconsistent - sometimes works, sometimes doesn't.
+
+---
+
+### High: `search` Only Matches Headings
+
+**Symptom:**
+
+```bash
+mdcontext search "authentication" ./docs
+# Returns: Only matches if "authentication" appears in a heading
+```
+
+**Expected:** Search should find content anywhere in documents.
+
+**Impact:** Major limitation for discovery. Users searching for concepts/keywords get no results if those words aren't in headings.
+
+**Note:** Full semantic search requires an embedding provider (OpenAI, Ollama, LM Studio, or OpenRouter) + `--embed` flag. See [CONFIG.md](./CONFIG.md) for free local options. Structural search should at least search content.
+
+---
+
+### Medium: Token Budget Sometimes Exceeded
+
+**Symptom:**
+
+```bash
+mdcontext context --tokens 500 --brief file.md
+# Output may exceed 500 tokens
+```
+
+**Expected:** Output should respect token budget.
+
+**Impact:** Unpredictable context sizes when assembling for LLM consumption.
+
+---
+
+### Low: `stats` Command Minimal Without Embeddings
+
+**Symptom:**
+
+```bash
+mdcontext stats ./docs
+# Shows basic counts only
+```
+
+**Impact:** Limited usefulness without embeddings enabled.
+
+---
+
+## Recommendations
+
+### P0 - Fix Index Command
+
+The "0 documents" issue breaks the primary discovery workflow. Investigate:
+
+- Directory path resolution
+- File extension filtering
+- Silent failures in indexing process
+
+### P1 - Expand Structural Search
+
+Make structural search (`--structural` or default without embeddings) search document content, not just headings:
+
+- Full-text regex matching
+- Content snippet in results
+- Line number references
+
+### P2 - Enforce Token Budgets
+
+Ensure `--tokens` flag is respected:
+
+- Truncate output if necessary
+- Warn user if content exceeds budget
+- Consider separate flags for hard vs soft limits
+
+### P3 - Improve Stats Without Embeddings
+
+Show useful stats even without embeddings:
+
+- Document count, total tokens, avg tokens/doc
+- Section depth analysis
+- File size distribution
+
+---
+
+## Test Matrix
+
+| Agent   | Directory       | Commands Used          | Verdict                         |
+| ------- | --------------- | ---------------------- | ------------------------------- |
+| a3caa1b | ./docs          | tree, context, search  | YES with caveats                |
+| a199309 | ./docs          | tree, context, index   | YES - context justifies tool    |
+| a7857e0 | ./docs          | help, context, tree    | YES - direction is good         |
+| a71de5c | ./doc.llm       | index, search, context | Partially - search/index issues |
+| a4ec1e1 | ./docs (ralph)  | tree, context, stats   | YES - solves real problem       |
+| a96ff96 | ./docs.amorphic | tree, context, index   | YES - context is valuable       |
+
+---
+
+## Raw Findings Summary
+
+### What Agents Tried That Failed
+
+1. `mdcontext index <dir>` → "0 documents"
+2. `mdcontext search "keyword" <dir>` → No results (keyword in content, not heading)
+3. `mdcontext stats <dir>` → Minimal output without embeddings
+
+### What Agents Found Valuable
+
+1. `mdcontext context --brief file.md` → Instant useful summary
+2. `mdcontext tree <dir>` → Quick structure overview
+3. `mdcontext context file1.md file2.md --tokens 1000` → Multi-file assembly
+4. `mdcontext --help` / `mdcontext <cmd> --help` → Self-discovery worked
+
+---
+
+## Suggested Task Scope
+
+A follow-up task should address:
+
+1. **Index reliability** - Debug why index returns 0 documents
+2. **Content search** - Extend structural search beyond headings
+3. **Budget enforcement** - Ensure token limits are respected
+4. **Error messages** - Surface why operations "silently fail"

package/docs/BACKLOG.md

@@ -0,0 +1,80 @@
+# Backlog
+
+Ideas and improvements to revisit later.
+
+---
+
+## CLI: Schema-Based Argv Preprocessor
+
+**Date:** 2025-01-19
+**Priority:** Medium
+**Context:** Current `argv-preprocessor.ts` uses hardcoded flag lists which breaks on unknown flags.
+
+### Problem
+
+```bash
+mdcontext context --json docs/*.md --pretty -x 200
+# Error: ENOENT: no such file or directory, open '.../-x'
+```
+
+Unknown flags like `-x` get passed through as positional args (file paths) instead of being rejected with a clear error.
+
+### Proposed Solution
+
+Replace hardcoded `flagsWithValues` set with a schema-based approach:
+
+```typescript
+interface FlagSpec {
+  type: "boolean" | "string";
+}
+
+const schema: Record<string, FlagSpec> = {
+  "--json": { type: "boolean" },
+  "--output": { type: "string" },
+};
+
+function parse(argv: string[], schema: Record<string, FlagSpec>) {
+  const options: Record<string, any> = {};
+  const positionals: string[] = [];
+
+  for (let i = 0; i < argv.length; i++) {
+    const arg = argv[i];
+    if (arg.startsWith("-")) {
+      const spec = schema[arg];
+      if (!spec) throw new Error(`Unknown option: ${arg}`);
+      if (spec.type === "boolean") {
+        options[arg] = true;
+      } else {
+        const value = argv[i + 1];
+        if (!value || value.startsWith("-")) {
+          throw new Error(`Missing value for option: ${arg}`);
+        }
+        options[arg] = value;
+        i++;
+      }
+    } else {
+      positionals.push(arg);
+    }
+  }
+  return { options, positionals };
+}
+```
+
+### Benefits
+
+1. **Clear errors** - "Unknown option: -x" instead of cryptic file errors
+2. **Single source of truth** - Schema can align with Effect CLI definitions
+3. **Per-command schemas** - Each command declares its own flags
+4. **Maintainable** - Adding flags = adding to schema, not hunting through code
+
+### Implementation Notes
+
+- Could extract schema from existing Effect CLI option definitions
+- Or define shared schema that both preprocessor and CLI use
+- Consider generating schema from CLI definitions at build time
+
+### Related Files
+
+- `src/cli/argv-preprocessor.ts` - Current implementation
+- `src/cli/commands/*.ts` - Effect CLI command definitions
+- `src/cli/options.ts` - Shared options