mdcontext 0.0.1 → 0.1.0

package/BACKLOG.md

@@ -0,0 +1,338 @@
+# mdcontext Improvement Backlog
+
+> Generated from validation experiment with 11 AI agents across 3 strategies.
+> See `/reports/FINAL-SYNTHESIS.md` for full analysis.
+
+---
+
+## P0 - Critical (Blocking Agent Workflows)
+
+### 1. Boolean Query Operators
+
+**Problem:** Agents couldn't search for multi-term concepts like "architecture AND criticism" or "checkpoint NOT example". Phrases returned 0 results even when concepts were present separately.
+
+**Impact:** 3-5x more commands needed for workaround (multiple single-term searches + manual correlation). Strategy B agents rated tool lower (4.2/5) partly due to this.
+
+**Solution:** Add boolean operators to search command.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "auth AND criticism"` returns results containing both terms
+- [ ] `mdcontext search "checkpoint OR gate"` returns results containing either term
+- [ ] `mdcontext search "implementation NOT example"` excludes results with "example"
+- [ ] `mdcontext search --help` documents boolean syntax
+
+**Effort:** Medium
+
+**Sources:** B1, B2, B3, C3, C4, FINAL-SYNTHESIS
+
+---
+
+### 2. Graceful Embeddings Fallback
+
+**Problem:** Semantic searches returned 0 results without clear indication that embeddings weren't built. Agent A2 was confused by silent failures.
+
+**Impact:** Inconsistent behavior, wasted agent turns, reduced confidence. A2 rating would be 5/5 with better UX.
+
+**Solution:** Auto-detect embeddings state and provide clear feedback.
+
+**Acceptance Criteria:**
+- [ ] When embeddings don't exist, search shows: "Semantic search unavailable. Using structural search. Run `mdcontext index --embed` for semantic search."
+- [ ] OR: Auto-prompt on first search: "Enable semantic search? (requires ~30s indexing)"
+- [ ] Search output shows mode indicator: `[semantic]` or `[structural]`
+- [ ] `mdcontext stats` shows embeddings status: "Embeddings: Yes/No (run index --embed to enable)"
+
+**Effort:** Low-Medium
+
+**Sources:** A2, FINAL-SYNTHESIS
+
+---
+
+### 3. Section-Level Context Extraction
+
+**Problem:** Agents couldn't request context for a specific section without retrieving the entire file. When investigating a specific subsection, full-file context wastes tokens.
+
+**Impact:** Over-retrieval or multiple refined searches needed. Forces choosing between full context or aggressive summarization.
+
+**Solution:** Enable section-targeted context extraction.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext context file.md --section "Memory Model"` returns only that section
+- [ ] `mdcontext context file.md:5.3` returns section 5.3 (by number)
+- [ ] `mdcontext context file.md --section "Memory*"` supports glob patterns
+- [ ] Nested sections included by default, `--shallow` flag for top-level only
+
+**Effort:** Medium-High
+
+**Sources:** A1, A2, C2, C4, FINAL-SYNTHESIS
+
+---
+
+## P1 - High Priority (Significant UX Improvement)
+
+### 4. Search Result Context Lines
+
+**Problem:** Search results show line numbers but minimal surrounding context. Hard to evaluate relevance without reading full sections.
+
+**Impact:** Extra commands needed to fetch context around matches. Agents requested grep-like `-C` behavior.
+
+**Solution:** Add context lines around search matches.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "checkpoint" -C 3` shows 3 lines before/after each match
+- [ ] `mdcontext search "checkpoint" -B 2 -A 5` shows 2 before, 5 after
+- [ ] Context lines clearly delineated from match lines
+- [ ] Works with both structural and semantic search
+
+**Effort:** Low-Medium
+
+**Sources:** B2, C3, C5, FINAL-SYNTHESIS
+
+---
+
+### 5. Remove 10-Result Limit / Add Pagination
+
+**Problem:** Default 10 results per query makes it hard to see ALL occurrences of a theme. Agents needed multiple queries to ensure comprehensive coverage.
+
+**Impact:** Incomplete results for common terms, extra commands for pagination workarounds.
+
+**Solution:** Add flags for result limit control.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "workflow" --all` shows all matches (no limit)
+- [ ] `mdcontext search "workflow" -n 50` shows up to 50 results
+- [ ] `mdcontext search "workflow" --offset 10 -n 10` for pagination
+- [ ] Default remains 10 for quick searches
+
+**Effort:** Low
+
+**Sources:** B1, B3, C1, FINAL-SYNTHESIS
+
+---
+
+### 6. Truncation UX Improvement
+
+**Problem:** When output truncated, agents couldn't selectively access missing sections. Truncation note appeared at end, not clearly signaled upfront.
+
+**Impact:** Forces over-retrieval or multiple refined searches. 50-96% reduction sometimes excessive.
+
+**Solution:** Better truncation signaling and navigation.
+
+**Acceptance Criteria:**
+- [ ] Truncation warning appears at TOP of output: "Output truncated (showing 2000/8500 tokens). Use --full or --section to retrieve more."
+- [ ] `mdcontext context file.md --full` shows complete content (no truncation)
+- [ ] Truncated output shows which sections were included/excluded
+- [ ] `mdcontext context file.md --sections` lists available sections for targeted retrieval
+
+**Effort:** Medium
+
+**Sources:** A2, B2, C2, FINAL-SYNTHESIS
+
+---
+
+### 7. Phrase Search with Quotes
+
+**Problem:** No way to search for exact phrases. "context resumption" as two words found irrelevant matches.
+
+**Impact:** Reduced precision for multi-word concepts.
+
+**Solution:** Support quoted phrase search.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search '"context resumption"'` matches exact phrase only
+- [ ] `mdcontext search '"drift-free"'` matches hyphenated terms
+- [ ] Can combine with boolean: `mdcontext search '"context resumption" AND drift`
+- [ ] `mdcontext search --help` documents phrase syntax
+
+**Effort:** Medium
+
+**Sources:** B1, B2, FINAL-SYNTHESIS
+
+---
+
+## P2 - Medium Priority (Nice to Have)
+
+### 8. Cross-File Link Analysis
+
+**Problem:** No command to find which files reference which concepts. Backlinks/links commands showed 0 for most queries.
+
+**Impact:** Manual grep needed to understand document relationships. Particularly wanted by Strategy C agents for architectural investigation.
+
+**Solution:** Implement concept-based cross-file analysis.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext refs "Execution Context"` shows all files mentioning this concept
+- [ ] `mdcontext refs --graph` outputs dependency graph (mermaid/dot format)
+- [ ] `mdcontext backlinks file.md` shows files that link TO this file
+- [ ] Works with markdown links `[text](file.md)` and concept mentions
+
+**Effort:** High
+
+**Sources:** C1, C6, FINAL-SYNTHESIS
+
+---
+
+### 9. Neighborhood View Around Search Results
+
+**Problem:** Search results show isolated matches. Agents wanted to see adjacent sections for context without fetching entire file.
+
+**Impact:** Extra context commands needed to understand match surroundings.
+
+**Solution:** Add section-level context around search results.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "checkpoint" --context-sections 1` shows 1 section before/after each match
+- [ ] Section context clearly labeled with headers
+- [ ] Works with structural and semantic search
+
+**Effort:** Medium
+
+**Sources:** C2, C3, FINAL-SYNTHESIS
+
+---
+
+### 10. Search Mode Indicator
+
+**Problem:** Agents couldn't tell if search was using semantic or structural mode. Behavior varied based on embeddings presence.
+
+**Impact:** Confusion about why some searches worked differently than others.
+
+**Solution:** Always show search mode in output.
+
+**Acceptance Criteria:**
+- [ ] Search results header shows: `[semantic search]` or `[structural search]`
+- [ ] If semantic attempted but embeddings missing, show: `[structural search - embeddings not found]`
+- [ ] `mdcontext search --mode` flag to force mode: `--mode semantic` or `--mode structural`
+
+**Effort:** Low
+
+**Sources:** A2, FINAL-SYNTHESIS
+
+---
+
+### 11. Query Syntax Help
+
+**Problem:** Agents had to discover search syntax through trial and error. No examples in help.
+
+**Impact:** Wasted turns on failed queries, inconsistent usage patterns.
+
+**Solution:** Improve search help with examples.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search --help` includes example section:
+  ```
+  EXAMPLES:
+    mdcontext search "auth"                    # Single term (structural)
+    mdcontext search "how to deploy"           # Semantic (if embeddings exist)
+    mdcontext search "auth AND deploy"         # Boolean AND
+    mdcontext search '"context resumption"'    # Exact phrase
+    mdcontext search "impl NOT test" -C 3      # Exclude term, show context
+  ```
+- [ ] `mdcontext search --examples` shows extended examples with explanations
+
+**Effort:** Low
+
+**Sources:** B2, B3, FINAL-SYNTHESIS
+
+---
+
+## P3 - Low Priority (Future Enhancements)
+
+### 12. Multi-File Glob Context
+
+**Problem:** Agents had to run context command on each file separately for batch operations.
+
+**Impact:** More commands needed for comprehensive extraction.
+
+**Solution:** Support glob patterns in context command.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext context "docs/**/*.md"` extracts context from all matching files
+- [ ] `mdcontext context "docs/*.md" -t 5000` applies token budget across all files
+- [ ] Output clearly shows which content came from which file
+
+**Effort:** Medium
+
+**Sources:** FINAL-SYNTHESIS
+
+---
+
+### 13. Saved Queries / Aliases
+
+**Problem:** Agents repeatedly ran similar complex queries.
+
+**Impact:** Command duplication, typo risk.
+
+**Solution:** Allow saving common query patterns.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext alias add arch-issues "search 'architecture AND (problem OR issue OR concern)'"`
+- [ ] `mdcontext arch-issues` runs saved query
+- [ ] Aliases stored in `.mdcontext/aliases.json`
+
+**Effort:** Medium
+
+**Sources:** FINAL-SYNTHESIS
+
+---
+
+### 14. Relevance Score Display
+
+**Problem:** Search results showed matches but not how relevant each match was.
+
+**Impact:** Harder to prioritize which results to investigate first.
+
+**Solution:** Show relevance/similarity scores.
+
+**Acceptance Criteria:**
+- [ ] Semantic search shows similarity score: `[0.87] docs/ARCH.md:45 - Control Plane...`
+- [ ] Results sorted by relevance by default
+- [ ] `--sort recent` flag to sort by file modification time instead
+
+**Effort:** Low-Medium
+
+**Sources:** C1, FINAL-SYNTHESIS
+
+---
+
+### 15. Troubleshooting Guide
+
+**Problem:** Agents encountered issues (0 results, truncation, embeddings) without clear resolution paths.
+
+**Impact:** Wasted turns debugging tool behavior.
+
+**Solution:** Add troubleshooting documentation.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext troubleshoot` command shows common issues and fixes
+- [ ] Covers: 0 results, embeddings setup, truncation, index staleness
+- [ ] OR: Add troubleshooting section to README
+
+**Effort:** Low
+
+**Sources:** FINAL-SYNTHESIS
+
+---
+
+## Summary
+
+| Priority | Count | Theme |
+|----------|-------|-------|
+| P0 Critical | 3 | Boolean search, embeddings UX, section extraction |
+| P1 High | 4 | Context lines, pagination, truncation, phrases |
+| P2 Medium | 4 | Cross-file analysis, neighborhood view, mode indicator, help |
+| P3 Low | 4 | Glob context, aliases, relevance scores, docs |
+
+**Total: 15 actionable tasks**
+
+---
+
+## Validation Sources
+
+All tasks derived from agent feedback across three strategies:
+
+- **Strategy A:** A1, A2, A3, A-Synth (4 agents, by-folder approach)
+- **Strategy B:** B1, B2, B3, B-Synth (4 agents, by-question approach)
+- **Strategy C:** C1, C2-C6, C-Synth (7 agents, two-phase approach)
+- **Final Synthesis:** Cross-strategy analysis
+
+Full reports: `/Users/alphab/Dev/LLM/DEV/TMP/memory/reports/`

package/README.md

@@ -1,19 +1,239 @@
 # mdcontext
 
-Intelligent markdown context extraction for LLMs.
+**Give LLMs exactly the markdown they need. Nothing more.**
 
-**Coming soon.** This package is a placeholder for the upcoming release.
+```bash
+QUICK REFERENCE
+  mdcontext index [path]           Index markdown files (add --embed for semantic search)
+  mdcontext search <query> [path]  Search by meaning or structure
+  mdcontext context <files...>     Get LLM-ready summary
+  mdcontext tree [path|file]       Show files or document outline
+  mdcontext links <file>           Outgoing links
+  mdcontext backlinks <file>       Incoming links
+  mdcontext stats [path]           Index statistics
+```
 
-## What is mdcontext?
+---
 
-mdcontext extracts structured, token-optimized context from markdown documentation for use with LLMs. Features include:
+## Why?
 
-- Semantic search with embeddings
-- Smart section extraction
-- Token counting and optimization
-- Context-aware chunking
+Your documentation is 50K tokens of markdown. LLM context windows are limited. Raw markdown dumps waste tokens on structure, headers, and noise.
 
-## Links
+mdcontext extracts *structure* instead of dumping *text*. The result: **80%+ fewer tokens** while preserving everything needed to understand your docs.
 
-- Website: https://mdcontext.com
-- GitHub: https://github.com/mdcontext/mdcontext
+```bash
+npm install -g mdcontext
+mdcontext index .                     # Index your docs
+mdcontext search "authentication"     # Find by meaning
+mdcontext context README.md           # Get LLM-ready summary
+```
+
+---
+
+## Installation
+
+```bash
+npm install -g mdcontext
+```
+
+Requires Node.js 18+. Semantic search requires `OPENAI_API_KEY`.
+
+---
+
+## Commands
+
+### index
+
+Index markdown files. Run this first.
+
+```bash
+mdcontext index                       # Index current directory (prompts for semantic)
+mdcontext index ./docs                # Index specific path
+mdcontext index --embed               # Also build embeddings for semantic search
+mdcontext index --no-embed            # Skip the semantic search prompt
+mdcontext index --watch               # Watch for changes
+mdcontext index --force               # Force full rebuild
+```
+
+### search
+
+Search by meaning (semantic) or keyword (text match).
+
+```bash
+mdcontext search "how to authenticate"        # Semantic search (if embeddings exist)
+mdcontext search -k "auth.*flow"              # Keyword search (text match)
+mdcontext search -n 5 "setup"                 # Limit to 5 results
+mdcontext search --threshold 0.8 "deploy"     # Higher similarity threshold
+```
+
+#### Context Lines
+
+Show surrounding lines around matches (like grep):
+
+```bash
+mdcontext search "checkpoint" -C 3            # 3 lines before AND after each match
+mdcontext search "error" -B 2 -A 5            # 2 lines before, 5 lines after
+```
+
+Auto-detection: Uses semantic search if embeddings exist and query looks like natural language. Use `-k` to force keyword search.
+
+### context
+
+Get LLM-ready summaries from one or more files.
+
+```bash
+mdcontext context README.md                   # Single file
+mdcontext context README.md docs/api.md       # Multiple files
+mdcontext context docs/*.md                   # Glob patterns work
+mdcontext context -t 500 README.md            # Token budget
+mdcontext context --brief README.md           # Minimal output
+mdcontext context --full README.md            # Include full content
+```
+
+#### Section Filtering
+
+Extract specific sections instead of entire files:
+
+```bash
+mdcontext context doc.md --sections           # List available sections
+mdcontext context doc.md --section "Setup"    # Extract by section name
+mdcontext context doc.md --section "2.1"      # Extract by section number
+mdcontext context doc.md --section "API*"     # Glob pattern matching
+mdcontext context doc.md --section "Config" --shallow  # Top-level only (no nested subsections)
+```
+
+The `--sections` flag shows all sections with their numbers and token counts, helping you target exactly what you need.
+
+### tree
+
+Show file structure or document outline.
+
+```bash
+mdcontext tree                        # List markdown files in current directory
+mdcontext tree ./docs                 # List files in specific directory
+mdcontext tree README.md              # Show document outline (heading hierarchy)
+```
+
+Auto-detection: Directory shows file list, file shows document outline.
+
+### links / backlinks
+
+Analyze link relationships.
+
+```bash
+mdcontext links README.md             # What does this file link to?
+mdcontext backlinks docs/api.md       # What files link to this?
+```
+
+### stats
+
+Show index statistics.
+
+```bash
+mdcontext stats                       # Current directory
+mdcontext stats ./docs                # Specific path
+```
+
+---
+
+## Workflows
+
+### Before Adding Context to LLM
+
+```bash
+mdcontext tree docs/                          # See what's available
+mdcontext tree docs/api.md                    # Check document structure
+mdcontext context -t 500 docs/api.md          # Get summary within token budget
+```
+
+### Finding Documentation
+
+```bash
+mdcontext search "authentication"             # By meaning
+mdcontext search -k "Setup|Install"           # By keyword pattern
+```
+
+### Setting Up Semantic Search
+
+```bash
+export OPENAI_API_KEY=sk-...
+mdcontext index --embed                       # Build embeddings
+mdcontext search "how to deploy"              # Now works semantically
+```
+
+---
+
+## MCP Integration
+
+For Claude Desktop, add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "mdcontext": {
+      "command": "mdcontext-mcp",
+      "args": []
+    }
+  }
+}
+```
+
+For Claude Code, add to `.claude/settings.json`:
+
+```json
+{
+  "mcpServers": {
+    "mdcontext": {
+      "command": "mdcontext-mcp",
+      "args": []
+    }
+  }
+}
+```
+
+### MCP Tools
+
+| Tool | Description |
+|------|-------------|
+| `md_search` | Semantic search across indexed docs |
+| `md_context` | Get LLM-ready summary for a file |
+| `md_structure` | Get document outline |
+
+---
+
+## Configuration
+
+### Index Location
+
+Indexes are stored in `.mdcontext/` in your project root:
+
+```
+.mdcontext/
+  indexes/
+    documents.json    # Document metadata
+    sections.json     # Section index
+    links.json        # Link graph
+    vectors.bin       # Embeddings (if enabled)
+```
+
+### Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `OPENAI_API_KEY` | Required for semantic search |
+
+---
+
+## Performance
+
+| Metric | Raw Markdown | mdcontext | Savings |
+|--------|--------------|---------|---------|
+| Context for single doc | 2,500 tokens | 400 tokens | **84%** |
+| Context for 10 docs | 25,000 tokens | 4,000 tokens | **84%** |
+| Search latency | N/A | <100ms | - |
+
+---
+
+## License
+
+MIT

package/biome.json

@@ -0,0 +1,36 @@
+{
+  "$schema": "https://biomejs.dev/schemas/2.3.11/schema.json",
+  "assist": {
+    "actions": {
+      "source": {
+        "organizeImports": "on"
+      }
+    }
+  },
+  "linter": {
+    "enabled": true,
+    "rules": {
+      "recommended": true,
+      "suspicious": {
+        "noExplicitAny": "off"
+      },
+      "style": {
+        "noNonNullAssertion": "off"
+      }
+    }
+  },
+  "formatter": {
+    "enabled": true,
+    "indentStyle": "space",
+    "indentWidth": 2
+  },
+  "javascript": {
+    "formatter": {
+      "semicolons": "asNeeded",
+      "quoteStyle": "single"
+    }
+  },
+  "files": {
+    "includes": ["src/**", "tests/**", "*.json", "*.ts"]
+  }
+}

package/cspell.config.yaml

@@ -0,0 +1,14 @@
+version: "0.2"
+ignorePaths: []
+dictionaryDefinitions: []
+dictionaries: []
+words:
+  - attw
+  - backlinks
+  - Hnsw
+  - hnswlib
+  - mdcontext
+  - modelcontextprotocol
+  - tiktoken
+ignoreWords: []
+import: []