mdcontext 0.0.1 → 0.2.0

package/research/semantic-search/multi-word-failure-reproduction.md

@@ -0,0 +1,171 @@
+# Multi-Word Semantic Search Failure Reproduction
+
+## Executive Summary
+
+After systematic testing, the reported "multi-word semantic search failure" is **NOT a failure of semantic search itself**, but rather a **threshold calibration issue**. The root causes are:
+
+1. **Single-word queries have low similarity scores** (30-40%) while multi-word queries have higher scores (50-70%)
+2. **Default threshold of 0.5** filters out both single-word AND semantically-distant multi-word queries
+3. **Queries with abstract/non-domain-specific terms** (e.g., "gaps missing omissions", "issue challenge gap") score below threshold
+4. **Domain-specific multi-word queries work well** (e.g., "failure automation" = 61%, "process orchestration" = 68%)
+
+## Test Methodology
+
+### Test Corpus
+
+Created a controlled test corpus in `src/__tests__/fixtures/semantic-search/multi-word-corpus/` with 6 markdown files covering:
+- failure-automation.md - Failure detection and automated recovery
+- job-context.md - Job execution context and metadata
+- error-handling.md - Error handling patterns
+- configuration-management.md - Config management practices
+- distributed-systems.md - Distributed systems architecture
+- process-orchestration.md - Workflow orchestration patterns
+
+**Corpus Statistics:**
+- 6 documents
+- 67 sections
+- 52 embedded vectors
+- ~4,725 tokens
+
+### Index Command
+
+```bash
+node dist/cli/main.js index src/__tests__/fixtures/semantic-search/multi-word-corpus --embed --force
+```
+
+## Test Results
+
+### Multi-Word Domain-Specific Queries (DEFAULT THRESHOLD 0.5)
+
+| Query | Results | Top Match | Top Score |
+|-------|---------|-----------|-----------|
+| "failure automation" | 7 | failure-automation.md: Best Practices | 61.6% |
+| "job context" | 4 | job-context.md: What is Job Context? | 60.4% |
+| "error handling" | 7 | error-handling.md: Introduction | 63.7% |
+| "configuration management" | 8 | configuration-management.md: Overview | 69.5% |
+| "distributed systems" | 4 | distributed-systems.md: What Are... | 61.0% |
+| "process orchestration" | 8 | process-orchestration.md: Introduction | 68.0% |
+
+**Finding:** Multi-word queries with domain-specific terms **WORK WELL** with default threshold.
+
+### Single-Word Queries (DEFAULT THRESHOLD 0.5)
+
+| Query | Results | Notes |
+|-------|---------|-------|
+| "failure" | 0 | Below 0.5 threshold |
+| "automation" | 0 | Below 0.5 threshold |
+| "context" | 0 | Below 0.5 threshold |
+| "error" | 0 | Below 0.5 threshold |
+
+### Single-Word Queries (THRESHOLD 0.3)
+
+| Query | Results | Top Match | Top Score |
+|-------|---------|-----------|-----------|
+| "failure" | 10 | failure-automation.md: Failure Isolation | 39.1% |
+| "automation" | 10 | (similar) | ~35% |
+| "error" | 10 | error-handling.md: Programming Errors | 49.1% |
+
+**Finding:** Single-word queries have inherently **LOW similarity scores** (30-49%) due to:
+1. Short query embeddings lack semantic context
+2. Embedding model produces less distinctive vectors for single words
+3. Cosine similarity between short and long vectors is compressed
+
+### Abstract/Generic Multi-Word Queries (DEFAULT THRESHOLD 0.5)
+
+| Query | Results | Notes |
+|-------|---------|-------|
+| "issue challenge gap" | 0 | Abstract terms, no domain match |
+| "gaps missing omissions" | 0 | Meta-language about content, not content itself |
+
+### Abstract Queries (THRESHOLD 0.3)
+
+| Query | Results | Top Match | Top Score |
+|-------|---------|-----------|-----------|
+| "issue challenge gap" | 10 | distributed-systems.md: Consistency vs Availability | 40.8% |
+| "gaps missing omissions" | 3 | error-handling.md: Programming Errors | 35.0% |
+
+**Finding:** Abstract/meta-language queries score **30-40%** - below default threshold but findable with lower threshold.
+
+### Hybrid Search Results
+
+| Query | Hybrid Results | Primary Source |
+|-------|---------------|----------------|
+| "failure automation" | 7 | Semantic (RRF ~1.6) |
+| "job context" | 4 | Semantic (RRF ~1.6) |
+
+**Finding:** Hybrid search successfully combines semantic and keyword results, but the semantic component still uses the threshold filter.
+
+## Pattern Analysis
+
+### What Works (>50% similarity)
+- Multi-word queries with **domain-specific terms** directly present in content
+- Queries that form **coherent concepts** (e.g., "process orchestration")
+- Queries that match **document titles or major headings**
+
+### What Fails at Default Threshold
+- **Single words** - all score 30-49%
+- **Abstract meta-language** - "gaps", "issues", "challenges" without domain context
+- **Non-domain queries** searching indexed domain content
+- **Very short queries** (1-2 generic words)
+
+### Similarity Score Distribution
+
+```
+70%+ : Document title/heading exact concept matches
+60-70%: Multi-word domain queries matching content topics
+50-60%: Multi-word queries with partial concept overlap
+40-50%: Single words or abstract queries with some relevance
+30-40%: Tangentially related content
+<30% : Unrelated content (correctly filtered)
+```
+
+## Dogfooding Context
+
+The dogfooding agents reported semantic search as "unreliable for multi-word conceptual queries". Re-analysis shows:
+
+1. **No embeddings were built** during dogfooding (only keyword index existed)
+2. Semantic search was **unavailable** - falling back to keyword search
+3. Multi-word **keyword** searches like "failure automation" worked
+4. Multi-word keyword searches as **quoted phrases** returned 0 (expecting exact text)
+5. Abstract queries like "gaps missing omissions" correctly returned 0 (phrase not in content)
+
+The actual issue was:
+- **Semantic search unavailable** (no embeddings)
+- **Keyword phrase search** misunderstood (quoted = exact match)
+- **Abstract conceptual queries** don't match concrete content via keyword
+
+## Recommendations
+
+### For ALP-204 (Embedding Text Analysis)
+- Analyze how `generateEmbeddingText()` combines section context
+- Check if heading + parent + content provides enough semantic signal for short queries
+
+### For ALP-205 (Query Processing)
+- Query text is passed directly to embedding - no preprocessing
+- Consider query expansion for short queries
+
+### For ALP-206 (Vector Search Parameters)
+- Default threshold of 0.5 is **too high** for single-word queries
+- Consider adaptive thresholds based on query length
+- Consider returning top-K results regardless of threshold, then filtering
+
+### For ALP-207 (Solution Design)
+Key solutions to consider:
+1. **Adaptive threshold** - lower for short queries
+2. **Query expansion** - augment short queries with context
+3. **Better user feedback** - show "X results below threshold" message
+4. **Threshold documentation** - educate users on --threshold flag
+
+## Conclusion
+
+Multi-word semantic search **is working correctly** for domain-specific queries. The perceived "failure" is a combination of:
+1. No embeddings in dogfooding environment
+2. Threshold too high for short/abstract queries
+3. Confusion between keyword phrase search and semantic search
+4. Users expecting semantic search to understand meta-language about content
+
+The fix is NOT to change semantic search algorithm, but to:
+1. Calibrate default threshold appropriately
+2. Add query-length-aware threshold adjustment
+3. Improve error messages when no results found
+4. Consider hybrid search as default mode

package/research/semantic-search/query-processing-analysis.md

@@ -0,0 +1,207 @@
+# Query Processing Analysis
+
+## Executive Summary
+
+Query processing is **minimal and appropriate**. The query text is passed directly to the embedding API without modification. This is correct behavior for OpenAI's text-embedding-3-small model, which handles text normalization internally.
+
+The asymmetry between query format (plain text) and document format (text with metadata) does NOT cause issues - embedding models are designed for this asymmetric retrieval pattern.
+
+## Query Flow
+
+```
+User Input
+    │
+    ▼
+CLI Parser (search.ts)
+    │ query string unchanged
+    ▼
+semanticSearch(rootPath, query, options)
+    │ query string unchanged
+    ▼
+provider.embed([query])
+    │ passed directly to API
+    ▼
+OpenAI Embeddings API
+    │ returns 512-dimensional vector
+    ▼
+Vector Store search()
+    │ cosine similarity comparison
+    ▼
+Results filtered by threshold
+```
+
+## Code Trace
+
+### Entry Point: CLI
+
+```typescript
+// src/cli/commands/search.ts:53-55
+query: Args.text({ name: 'query' }).pipe(
+  Args.withDescription('Search query (natural language or regex pattern)'),
+),
+```
+
+The query enters as a raw text string, no preprocessing.
+
+### Search Mode Detection
+
+```typescript
+// src/cli/commands/search.ts:201-206
+} else if (isAdvancedQuery(query)) {
+  effectiveMode = 'keyword'
+  modeReason = 'boolean/phrase pattern detected'
+} else if (isRegexPattern(query)) {
+  effectiveMode = 'keyword'
+  modeReason = 'regex pattern detected'
+}
+```
+
+Queries with boolean operators (AND, OR, NOT) or quoted phrases are routed to keyword search. Plain multi-word queries go to semantic search.
+
+### Semantic Search Function
+
+```typescript
+// src/embeddings/semantic-search.ts:558-559
+// Embed the query
+const queryResult = yield* wrapEmbedding(provider.embed([query]))
+```
+
+**No preprocessing** - query is embedded exactly as received.
+
+### Embedding API Call
+
+```typescript
+// src/embeddings/openai-provider.ts:175-179
+const response = await this.client.embeddings.create({
+  model: this.model,
+  input: batch,  // query text passed directly
+  dimensions: 512,
+})
+```
+
+Query text goes directly to OpenAI API without modification.
+
+## Query vs Document Format Asymmetry
+
+### Document Embedding Format (from ALP-204)
+
+```
+# {heading}
+Parent section: {parentHeading}
+Document: {documentTitle}
+
+{content}
+```
+
+### Query Format
+
+```
+{raw query text}
+```
+
+### Analysis
+
+This asymmetry is **intentional and correct** for semantic search:
+
+1. **Embedding models handle asymmetry**: OpenAI's text-embedding models are trained on diverse text formats. They produce semantically meaningful vectors regardless of format.
+
+2. **Query expansion is not needed**: The embedding model understands "failure automation" conceptually - it doesn't need to see `# Failure Automation` format.
+
+3. **Document context helps disambiguation**: The heading/document metadata in indexed content helps distinguish between sections with similar content but different contexts.
+
+4. **Industry standard practice**: Most RAG systems use plain queries against enriched documents.
+
+## Query Variation Tests
+
+All variations produce semantically similar results:
+
+| Query | Top Result | Similarity |
+|-------|------------|------------|
+| "failure automation" | Best Practices | 61.6% |
+| "failure-automation" | Overview | 68.8% |
+| "Failure Automation" | Best Practices | 65.6% |
+| "automation for failures" | Overview | 70.3% |
+| "how to automate failure handling" | Best Practices | 66.4% |
+
+**Findings:**
+- Casing doesn't significantly affect results
+- Hyphenation produces slightly different top result
+- Word order matters but doesn't break search
+- Natural language queries work well
+
+## Threshold Analysis
+
+### Default Threshold Flow
+
+```
+CLI default: 0.45
+    │
+    ▼ (if CLI uses default)
+Config default: 0.5
+    │
+    ▼
+Effective threshold: 0.5
+```
+
+When user doesn't specify `--threshold`, the effective value is 0.5 from config.
+
+### Threshold Impact
+
+| Threshold | Single-word "failure" | Multi-word "failure automation" |
+|-----------|----------------------|--------------------------------|
+| 0.5 | 0 results | 7 results |
+| 0.3 | 10 results | 7+ results |
+| 0.1 | 10 results | 7+ results |
+
+The 0.5 threshold filters out low-similarity single-word matches while allowing relevant multi-word matches through.
+
+## Potential Query Enhancements (for ALP-207)
+
+While current processing is correct, potential improvements could include:
+
+### 1. Query Expansion for Short Queries
+
+```typescript
+// Hypothetical enhancement
+const enhancedQuery = query.split(' ').length <= 2
+  ? `Find content about: ${query}`
+  : query
+```
+
+### 2. Adaptive Threshold
+
+```typescript
+// Lower threshold for shorter queries
+const adaptiveThreshold = query.split(' ').length <= 1
+  ? 0.3
+  : options.threshold ?? 0.5
+```
+
+### 3. Hybrid by Default
+
+Short queries might benefit from hybrid mode being the default, leveraging both keyword and semantic signals.
+
+## Recommendations
+
+### No Changes Needed to Query Processing
+
+The current implementation is correct. The query flow is:
+- Clean (no unnecessary transformations)
+- Transparent (what you type is what gets embedded)
+- Flexible (users can adjust with --threshold)
+
+### Focus Areas for ALP-207
+
+1. **Threshold tuning** - Consider lowering default to 0.4 or making it adaptive
+2. **Better feedback** - Show "X results below threshold" when 0 results
+3. **Documentation** - Explain threshold behavior in help text
+4. **Hybrid default** - Consider hybrid mode as default for better coverage
+
+## Conclusion
+
+Query processing is implemented correctly. The perceived "multi-word query failures" are actually threshold calibration issues, not query processing bugs. The search correctly:
+
+1. Passes queries unchanged to embedding API (correct)
+2. Uses asymmetric retrieval (query vs enriched documents) (correct)
+3. Handles query variations semantically (working)
+4. Applies configurable threshold (working, but may need tuning)

package/research/semantic-search/root-cause-and-solution.md

@@ -0,0 +1,114 @@
+# Root Cause Analysis and Solution Design
+
+## Executive Summary
+
+**Root Cause**: The "multi-word semantic search failure" is a **threshold calibration issue**, not a search algorithm bug.
+
+**Key Findings**:
+1. Multi-word domain queries WORK correctly (60-70% similarity)
+2. Single-word queries score lower (30-40%) due to embedding model properties
+3. Default 0.5 threshold filters out short/abstract queries
+4. The dogfooding had no embeddings built - agents fell back to keyword search
+5. Embedding text format, query processing, and HNSW config are all correct
+
+**Solution**: Lower default threshold + improve user feedback for edge cases.
+
+## Synthesis of Diagnostic Findings
+
+### ALP-203: Reproduction Results
+
+| Query Type | Works at 0.5? | Score Range |
+|------------|---------------|-------------|
+| "failure automation" | YES | 54-62% |
+| "error handling" | YES | 53-64% |
+| "failure" (single) | NO | 31-39% |
+| "error" (single) | NO | 32-49% |
+| "gaps missing omissions" | NO | 30-35% |
+
+**Conclusion**: Multi-word domain queries work. Short/abstract queries fail threshold.
+
+### ALP-204: Embedding Text Analysis
+
+- Format is correct: `# heading\nParent: X\nDocument: Y\n\ncontent`
+- Follows industry best practices
+- No issues identified
+
+### ALP-205: Query Processing Analysis
+
+- Query passed unchanged to embedding API (correct)
+- Asymmetric retrieval (plain query vs enriched docs) is normal
+- Query variations all work correctly
+
+### ALP-206: Vector Search Analysis
+
+- HNSW parameters (M=16, efConstruction=200, efSearch=100) are optimal
+- Cosine distance correct for text embeddings
+- Threshold filtering is the only issue
+
+## Root Cause
+
+**Primary Cause**: The default similarity threshold (0.5) is too high for:
+1. Single-word queries (max ~49% similarity due to embedding model properties)
+2. Abstract/meta-language queries
+3. Non-domain-specific queries
+
+**NOT the cause**:
+- Embedding text format (correct)
+- Query processing (correct)
+- HNSW parameters (optimal)
+- Embedding model (working as expected)
+
+**Contributing Factor**: Dogfooding lacked embeddings, causing confusion about what was failing.
+
+## Solution Design
+
+### Recommended Approach: Threshold Tuning + UX Improvements
+
+#### 1. Lower Default Threshold to 0.35
+
+```typescript
+// src/config/schema.ts
+minSimilarity: Config.number('minSimilarity').pipe(Config.withDefault(0.35))
+```
+
+**Rationale**:
+- Captures single-word results (30-40% range)
+- Still filters irrelevant content (<30%)
+- Low risk - users can adjust with --threshold
+
+#### 2. Add "Below Threshold" Feedback
+
+When 0 results, show hint about lower-scored results:
+
+```
+Results: 0
+
+Note: 10 results found below 0.35 threshold (highest: 0.34)
+Tip: Use --threshold 0.3 to see more results
+```
+
+#### 3. Consider Hybrid Search as Default
+
+For queries without boolean operators, hybrid mode provides better coverage by combining semantic and keyword signals.
+
+## Implementation Plan for Phase 2
+
+1. **Lower default threshold** - Change config default from 0.5 to 0.35
+2. **Add below-threshold feedback** - Show hint when 0 results
+3. **Document threshold behavior** - Update README/help
+4. **Validate changes** - Re-run test corpus
+
+## Expected Outcomes
+
+| Metric | Before | After |
+|--------|--------|-------|
+| Single-word results at default | 0 | 10+ |
+| Multi-word results | 7+ | 7+ (unchanged) |
+
+## Conclusion
+
+The "multi-word semantic search failure" was misidentified. Multi-word queries work correctly. The issue is threshold calibration affecting single-word and abstract queries.
+
+**Recommended Solution**: Lower threshold to 0.35, add user feedback, improve documentation.
+
+**No algorithmic changes needed** to embedding generation, query processing, or vector search.

package/research/semantic-search/threshold-validation-report.md

@@ -0,0 +1,69 @@
+# Threshold Validation Report
+
+## Summary
+
+Validation confirms that lowering the default similarity threshold from 0.5 to 0.35 (ALP-208) **fixes single-word query failures** without regressing multi-word query performance.
+
+## Test Environment
+
+- **Test Corpus**: `src/__tests__/fixtures/semantic-search/multi-word-corpus/`
+- **Documents**: 6 markdown files (failure-automation, job-context, error-handling, configuration-management, distributed-systems, process-orchestration)
+- **Sections**: 52 embedded vectors
+- **Date**: 2026-01-26
+
+## Before/After Comparison
+
+### Single-Word Queries
+
+| Query | Before (0.5) | After (0.35) | Top Match | Top Score |
+|-------|-------------|--------------|-----------|-----------|
+| "failure" | 0 results | **6 results** | failure-automation.md: Failure Isolation | 39.0% |
+| "error" | 0 results | **7 results** | error-handling.md: Programming Errors | 49.1% |
+| "automation" | 0 results | **10 results** | failure-automation.md: Overview | 44.9% |
+| "context" | 0 results | **10 results** | job-context.md: What is Job Context? | 48.1% |
+
+**Improvement**: 100% of single-word queries now return relevant results.
+
+### Multi-Word Queries (Regression Check)
+
+| Query | Before (0.5) | After (0.35) | Top Match | Top Score |
+|-------|-------------|--------------|-----------|-----------|
+| "failure automation" | 7 results | 10 results | failure-automation.md: Best Practices | 61.5% |
+| "job context" | 4 results | 7 results | job-context.md: What is Job Context? | 60.4% |
+| "error handling" | 7 results | 10 results | error-handling.md: Introduction | 63.6% |
+| "configuration management" | 8 results | 10 results | configuration-management.md: Overview | 69.5% |
+| "distributed systems" | 4 results | 10 results | distributed-systems.md: What Are... | 60.9% |
+| "process orchestration" | 8 results | 10 results | process-orchestration.md: Introduction | 67.9% |
+
+**Finding**: No regression. Multi-word queries actually return MORE results (expected, since threshold is lower), with the same top matches and scores.
+
+## Success Criteria Validation
+
+- [x] **Single-word queries return results at default threshold** - All 4 test queries now return 6-10 results
+- [x] **Multi-word queries work as before (no regression)** - All 6 queries return results with same top matches
+- [x] **Quantitative improvement documented** - See tables above
+
+## Below-Threshold Feedback (ALP-209)
+
+The new feedback feature correctly reports results below threshold:
+
+```json
+{
+  "results": [...6 results...],
+  "belowThresholdCount": 14,
+  "belowThresholdHighest": 0.349
+}
+```
+
+This helps users understand that more content exists if they lower the threshold.
+
+## Conclusion
+
+The threshold change from 0.5 to 0.35 is validated as the correct fix:
+
+1. **Single-word queries now work** - Users can search for concepts like "failure", "error", "context"
+2. **Multi-word queries unaffected** - High-quality results with same top matches
+3. **User guidance in place** - Documentation (ALP-210) explains threshold behavior
+4. **Below-threshold feedback** - Users see when lowering threshold would help
+
+The root cause identified in ALP-207 (threshold too high for short queries scoring 30-40%) is confirmed fixed.

package/research/semantic-search/vector-search-analysis.md

@@ -0,0 +1,63 @@
+# Vector Search Parameters and Scoring Analysis
+
+## Executive Summary
+
+The HNSW vector search configuration is **appropriate and well-tuned**. The root cause of "0 results" is **NOT the vector search algorithm**, but the **similarity threshold filtering** applied after search.
+
+Key finding: Single-word queries have inherently lower similarity scores (30-40%) than multi-word queries (50-70%). The default 0.5 threshold filters out all single-word results.
+
+## HNSW Configuration
+
+### Current Parameters
+
+From `src/embeddings/vector-store.ts:98`:
+
+```typescript
+this.index.initIndex(10000, 16, 200, 100)
+//                   maxElements, M, efConstruction, efSearch
+```
+
+| Parameter | Value | Description | Assessment |
+|-----------|-------|-------------|------------|
+| maxElements | 10,000 | Initial capacity (auto-resizes) | Adequate |
+| M | 16 | Max connections per node | Good balance |
+| efConstruction | 200 | Construction-time search width | High quality |
+| efSearch | 100 | Query-time search width | Good recall |
+
+All parameters are well-tuned. No changes needed.
+
+## Similarity Score Analysis
+
+### Threshold Experiment
+
+Testing "failure" at different thresholds:
+
+| Threshold | Results | Top Score |
+|-----------|---------|-----------|
+| 0.0 | 10 | 39.1% |
+| 0.3 | 10 | 39.1% |
+| 0.4 | 0 | - |
+| 0.5 | 0 | - |
+
+### Score Distribution by Query Type
+
+| Query Type | Score Range | Results at 0.5 |
+|------------|-------------|----------------|
+| Single word | 31-49% | 0 |
+| Two-word domain | 54-70% | 7+ |
+| Natural language | 50-66% | 9 |
+
+## Root Cause
+
+The 0.5 default threshold filters out single-word results (max ~49%). This is threshold calibration, not a search algorithm issue.
+
+## Recommendations for ALP-207
+
+1. Lower default threshold to 0.3-0.4
+2. Consider adaptive threshold by query length
+3. Show "N results below threshold" message
+4. Make threshold more visible in docs
+
+## Conclusion
+
+Vector search works correctly. Focus ALP-207 on threshold tuning, not algorithmic changes.