mdcontext 0.1.0 → 0.2.0

package/research/semantic-search/threshold-validation-report.md

@@ -0,0 +1,69 @@
+# Threshold Validation Report
+
+## Summary
+
+Validation confirms that lowering the default similarity threshold from 0.5 to 0.35 (ALP-208) **fixes single-word query failures** without regressing multi-word query performance.
+
+## Test Environment
+
+- **Test Corpus**: `src/__tests__/fixtures/semantic-search/multi-word-corpus/`
+- **Documents**: 6 markdown files (failure-automation, job-context, error-handling, configuration-management, distributed-systems, process-orchestration)
+- **Sections**: 52 embedded vectors
+- **Date**: 2026-01-26
+
+## Before/After Comparison
+
+### Single-Word Queries
+
+| Query | Before (0.5) | After (0.35) | Top Match | Top Score |
+|-------|-------------|--------------|-----------|-----------|
+| "failure" | 0 results | **6 results** | failure-automation.md: Failure Isolation | 39.0% |
+| "error" | 0 results | **7 results** | error-handling.md: Programming Errors | 49.1% |
+| "automation" | 0 results | **10 results** | failure-automation.md: Overview | 44.9% |
+| "context" | 0 results | **10 results** | job-context.md: What is Job Context? | 48.1% |
+
+**Improvement**: 100% of single-word queries now return relevant results.
+
+### Multi-Word Queries (Regression Check)
+
+| Query | Before (0.5) | After (0.35) | Top Match | Top Score |
+|-------|-------------|--------------|-----------|-----------|
+| "failure automation" | 7 results | 10 results | failure-automation.md: Best Practices | 61.5% |
+| "job context" | 4 results | 7 results | job-context.md: What is Job Context? | 60.4% |
+| "error handling" | 7 results | 10 results | error-handling.md: Introduction | 63.6% |
+| "configuration management" | 8 results | 10 results | configuration-management.md: Overview | 69.5% |
+| "distributed systems" | 4 results | 10 results | distributed-systems.md: What Are... | 60.9% |
+| "process orchestration" | 8 results | 10 results | process-orchestration.md: Introduction | 67.9% |
+
+**Finding**: No regression. Multi-word queries actually return MORE results (expected, since threshold is lower), with the same top matches and scores.
+
+## Success Criteria Validation
+
+- [x] **Single-word queries return results at default threshold** - All 4 test queries now return 6-10 results
+- [x] **Multi-word queries work as before (no regression)** - All 6 queries return results with same top matches
+- [x] **Quantitative improvement documented** - See tables above
+
+## Below-Threshold Feedback (ALP-209)
+
+The new feedback feature correctly reports results below threshold:
+
+```json
+{
+  "results": [...6 results...],
+  "belowThresholdCount": 14,
+  "belowThresholdHighest": 0.349
+}
+```
+
+This helps users understand that more content exists if they lower the threshold.
+
+## Conclusion
+
+The threshold change from 0.5 to 0.35 is validated as the correct fix:
+
+1. **Single-word queries now work** - Users can search for concepts like "failure", "error", "context"
+2. **Multi-word queries unaffected** - High-quality results with same top matches
+3. **User guidance in place** - Documentation (ALP-210) explains threshold behavior
+4. **Below-threshold feedback** - Users see when lowering threshold would help
+
+The root cause identified in ALP-207 (threshold too high for short queries scoring 30-40%) is confirmed fixed.

package/research/semantic-search/vector-search-analysis.md

@@ -0,0 +1,63 @@
+# Vector Search Parameters and Scoring Analysis
+
+## Executive Summary
+
+The HNSW vector search configuration is **appropriate and well-tuned**. The root cause of "0 results" is **NOT the vector search algorithm**, but the **similarity threshold filtering** applied after search.
+
+Key finding: Single-word queries have inherently lower similarity scores (30-40%) than multi-word queries (50-70%). The default 0.5 threshold filters out all single-word results.
+
+## HNSW Configuration
+
+### Current Parameters
+
+From `src/embeddings/vector-store.ts:98`:
+
+```typescript
+this.index.initIndex(10000, 16, 200, 100)
+//                   maxElements, M, efConstruction, efSearch
+```
+
+| Parameter | Value | Description | Assessment |
+|-----------|-------|-------------|------------|
+| maxElements | 10,000 | Initial capacity (auto-resizes) | Adequate |
+| M | 16 | Max connections per node | Good balance |
+| efConstruction | 200 | Construction-time search width | High quality |
+| efSearch | 100 | Query-time search width | Good recall |
+
+All parameters are well-tuned. No changes needed.
+
+## Similarity Score Analysis
+
+### Threshold Experiment
+
+Testing "failure" at different thresholds:
+
+| Threshold | Results | Top Score |
+|-----------|---------|-----------|
+| 0.0 | 10 | 39.1% |
+| 0.3 | 10 | 39.1% |
+| 0.4 | 0 | - |
+| 0.5 | 0 | - |
+
+### Score Distribution by Query Type
+
+| Query Type | Score Range | Results at 0.5 |
+|------------|-------------|----------------|
+| Single word | 31-49% | 0 |
+| Two-word domain | 54-70% | 7+ |
+| Natural language | 50-66% | 9 |
+
+## Root Cause
+
+The 0.5 default threshold filters out single-word results (max ~49%). This is threshold calibration, not a search algorithm issue.
+
+## Recommendations for ALP-207
+
+1. Lower default threshold to 0.3-0.4
+2. Consider adaptive threshold by query length
+3. Show "N results below threshold" message
+4. Make threshold more visible in docs
+
+## Conclusion
+
+Vector search works correctly. Focus ALP-207 on threshold tuning, not algorithmic changes.

package/research/test-path-issues.md

@@ -0,0 +1,276 @@
+# Test Path Issues Analysis
+
+---
+**RESEARCH METADATA**
+
+- Analysis Date: 2026-01-24
+- Git Commit: 07c9e72ba01cda840046b96a1be4743a85e3d4c5
+- Status: ✅ Valid
+- Last Validated: 2026-01-24
+- Worktree: nancy-ALP-139
+- Index: [/research/INDEX.md](INDEX.md)
+
+**ACCURACY NOTE**
+
+Test path fix validation. Findings are accurate and current.
+---
+
+**Date:** 2026-01-24
+**Issue:** Cross-platform path handling in test suite
+**Related File:** `src/cli/cli.test.ts`
+
+## Summary
+
+This document analyzes the fix made to line 458 of `src/cli/cli.test.ts` and identifies other potential path-related issues across the test suite.
+
+## The Fix at Line 458
+
+### What Changed
+
+```typescript
+// Before:
+expect(output).toContain('/nonexistent/path.json')
+
+// After:
+expect(output).toContain('path.json')
+```
+
+### Test Context
+
+The test (lines 453-459) verifies that when a non-existent config file is specified, the error message includes information about the missing file:
+
+```typescript
+it('shows error for non-existent config file', async () => {
+  const output = await run('--config /nonexistent/path.json --help', {
+    expectError: true,
+  })
+  expect(output).toContain('Error: Config file not found')
+  expect(output).toContain('path.json')
+})
+```
+
+### Analysis of the Fix
+
+**Is this fix correct?**
+YES - The fix is correct for the following reasons:
+
+1. **Cross-platform compatibility**: The original test used a Unix-style absolute path (`/nonexistent/path.json`) which would fail on Windows where absolute paths look like `C:\nonexistent\path.json`.
+
+2. **Appropriate validation scope**: The test's purpose is to verify that:
+   - An error is shown for non-existent config files
+   - The error message mentions the filename
+
+   It does NOT need to validate that the full absolute path appears in the error message.
+
+3. **More resilient**: By checking only for `path.json` (the basename), the test becomes agnostic to:
+   - Path separators (forward vs backslash)
+   - Path normalization
+   - Absolute vs relative path representation in error messages
+
+**Is this sufficient?**
+YES - The fix is sufficient because:
+
+1. The test has two assertions:
+   - First checks for the error type: `'Error: Config file not found'`
+   - Second checks that the filename appears: `'path.json'`
+
+2. Together, these assertions validate the core requirement: that users can identify which config file was not found.
+
+3. The test doesn't need to verify exact path formatting, which may vary by OS and implementation.
+
+**Does the test still validate what it should?**
+YES - The test's purpose is to ensure:
+- The CLI detects non-existent config files
+- The error message is informative enough for users to identify the problem
+- Both requirements are still met after the fix
+
+## Other Path Issues Found
+
+### 1. Mock Path Variables in `src/cli/argv-preprocessor.test.ts`
+
+**Lines 9-10:**
+```typescript
+const node = '/usr/bin/node'
+const script = '/path/to/mdcontext'
+```
+
+**Issue:** Uses hardcoded Unix-style paths for test fixtures.
+
+**Impact:** LOW - These are mock values used only to construct argv arrays for testing flag parsing. They don't interact with the filesystem or get validated against actual paths.
+
+**Recommendation:** No fix needed. These are appropriate test fixtures because:
+- They represent typical argv[0] and argv[1] values
+- They're used purely as string identifiers in the test
+- The actual values don't matter for the flag parsing logic being tested
+
+### 2. Hardcoded Paths in Error Test Files
+
+Multiple test files use hardcoded Unix-style paths as test data:
+
+#### `src/summarize/budget-bugs.test.ts`
+- Lines 11, 69, 126, 182, 220, 246, 335, 367, 426, 483, 499, 528, 556, 589
+- Example: `path: '/test/file.md'`
+
+#### `src/summarize/summarizer.test.ts`
+- Lines 16, 55, 92, 132, 150, 192, 224, 252
+- Example: `path: '/test/file.md'`
+
+#### `src/summarize/verify-bugs.test.ts`
+- Lines 21, 71, 92, 108, 148, 180, 203
+- Example: `path: '/very/long/path/to/deeply/nested/directory/structure/file.md'`
+
+#### `src/errors/errors.test.ts`
+- Lines 45, 53, 62, 72, 81, 95, 119, 131, 155, 346, 367, 382, 389, 395, 413, 473, 518, 547, 559, 571, 622, 634, 742, 792, 814
+- Examples:
+  - `path: '/test/file.md'`
+  - `sourceFile: '/path/to/mdcontext.config.json'`
+
+**Issue:** Hardcoded Unix-style paths in test data.
+
+**Impact:** VERY LOW - These are test fixtures representing document paths in mock data structures. They are:
+- Not used for filesystem operations
+- Not validated against actual paths
+- Used purely as string identifiers in domain objects
+- Never checked with path-specific assertions
+
+**Recommendation:** No fix needed. These paths are appropriate because:
+1. They represent the `path` field in domain objects (DocumentSummary, Error objects, etc.)
+2. The actual format doesn't affect test validity
+3. In production, these paths would come from actual filesystem operations
+4. Using Unix-style paths as test data is a common convention
+
+### 3. Config Test Paths in `src/config/file-provider.test.ts`
+
+**Lines 233, 248:**
+```typescript
+paths: {
+  cacheDir: '/custom/cache',
+}
+// ...
+expect(result.paths.cacheDir).toBe('/custom/cache')
+```
+
+**Issue:** Hardcoded Unix-style absolute path.
+
+**Impact:** VERY LOW - This is testing configuration value passthrough, not filesystem operations.
+
+**Recommendation:** No fix needed. The test validates that:
+- Config values are correctly parsed and stored
+- The exact string value is preserved
+- The path format is irrelevant since it's testing data flow, not path validation
+
+### 4. Config Schema Test Paths in `src/config/schema.test.ts`
+
+**Lines 231-245:**
+```typescript
+const provider = ConfigProvider.fromMap(
+  new Map([
+    ['root', '/home/user/docs'],
+    ['configFile', './custom.config.json'],
+    ['cacheDir', '.cache/mdcontext'],
+  ]),
+)
+// ...
+expect(Option.getOrThrow(result.root)).toBe('/home/user/docs')
+expect(Option.getOrThrow(result.configFile)).toBe('./custom.config.json')
+expect(result.cacheDir).toBe('.cache/mdcontext')
+```
+
+**Issue:** Mix of Unix absolute and relative paths.
+
+**Impact:** VERY LOW - Tests configuration schema parsing, not path operations.
+
+**Recommendation:** No fix needed. These test the configuration system's ability to:
+- Accept various path formats
+- Preserve path values as provided
+- Map between config sources and domain objects
+
+## Patterns Using `path.join` (Correct Usage)
+
+The test suite correctly uses `path.join()` for constructing paths that interact with the filesystem:
+
+### `src/cli/cli.test.ts`
+```typescript
+const TEST_FIXTURE_DIR = path.join(process.cwd(), 'tests', 'fixtures', 'cli')
+const CLI = `node ${path.join(process.cwd(), 'dist', 'cli', 'main.js')}`
+```
+
+### `src/search/searcher.test.ts`
+```typescript
+const TEST_DIR = path.join(process.cwd(), 'tests', 'fixtures', 'search')
+const doc1Path = path.join(TEST_DIR, 'doc1.md')
+```
+
+These are correct because they:
+- Build actual filesystem paths
+- Use OS-appropriate separators
+- Work across Windows, macOS, and Linux
+
+## Recommendations
+
+### 1. No Additional Fixes Required
+
+The test suite does not have significant cross-platform path issues. The fix at line 458 was appropriate, but no similar issues exist elsewhere because:
+
+- Most hardcoded paths are test data, not filesystem paths
+- Actual filesystem operations use `path.join()` correctly
+- Test assertions check for meaningful content, not path formatting
+
+### 2. Best Practices for Future Tests
+
+When writing new tests:
+
+#### DO:
+- Use `path.join()` for constructing filesystem paths
+- Use `path.basename()` when asserting on filenames in error messages
+- Check for file/directory names rather than full paths in assertions
+- Use relative paths in test fixtures when possible
+
+#### DON'T:
+- Hardcode absolute paths with `/` or `\` separators for filesystem operations
+- Make assertions about exact path formatting in error messages
+- Assume path separator type in test expectations
+- Use platform-specific path conventions in assertions
+
+#### Example: Testing Error Messages
+```typescript
+// GOOD: Check for filename, not full path
+expect(output).toContain('config.json')
+
+// BAD: Assumes Unix path format
+expect(output).toContain('/path/to/config.json')
+
+// GOOD: For actual file operations
+const configPath = path.join(testDir, 'config.json')
+```
+
+#### Example: Test Data
+```typescript
+// OK: Mock data representing a document path
+const mockDoc: DocumentSummary = {
+  path: '/test/file.md',  // Not a real filesystem path
+  // ...
+}
+
+// GOOD: Actual filesystem test fixture
+const testFile = path.join(fixtureDir, 'file.md')
+```
+
+### 3. Testing Strategy
+
+The current approach is sound:
+- Mock data paths can remain Unix-style for consistency
+- Filesystem operations use `path.join()`
+- Error message assertions check for meaningful content, not formatting
+- Integration tests use real filesystem operations with proper path handling
+
+## Conclusion
+
+The fix to line 458 is **correct and sufficient**. No other cross-platform path issues exist in the test suite. The codebase follows best practices by:
+
+1. Using `path.join()` for actual filesystem paths
+2. Using hardcoded paths only as mock data
+3. Making assertions on meaningful content rather than path formatting
+4. Keeping tests platform-agnostic
+
+The test suite is well-designed for cross-platform compatibility.