mdcontext 0.0.1 → 0.2.0

package/BACKLOG.md

@@ -0,0 +1,338 @@
+# mdcontext Improvement Backlog
+
+> Generated from validation experiment with 11 AI agents across 3 strategies.
+> See `/reports/FINAL-SYNTHESIS.md` for full analysis.
+
+---
+
+## P0 - Critical (Blocking Agent Workflows)
+
+### 1. Boolean Query Operators
+
+**Problem:** Agents couldn't search for multi-term concepts like "architecture AND criticism" or "checkpoint NOT example". Phrases returned 0 results even when concepts were present separately.
+
+**Impact:** 3-5x more commands needed for workaround (multiple single-term searches + manual correlation). Strategy B agents rated tool lower (4.2/5) partly due to this.
+
+**Solution:** Add boolean operators to search command.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "auth AND criticism"` returns results containing both terms
+- [ ] `mdcontext search "checkpoint OR gate"` returns results containing either term
+- [ ] `mdcontext search "implementation NOT example"` excludes results with "example"
+- [ ] `mdcontext search --help` documents boolean syntax
+
+**Effort:** Medium
+
+**Sources:** B1, B2, B3, C3, C4, FINAL-SYNTHESIS
+
+---
+
+### 2. Graceful Embeddings Fallback
+
+**Problem:** Semantic searches returned 0 results without clear indication that embeddings weren't built. Agent A2 was confused by silent failures.
+
+**Impact:** Inconsistent behavior, wasted agent turns, reduced confidence. A2 rating would be 5/5 with better UX.
+
+**Solution:** Auto-detect embeddings state and provide clear feedback.
+
+**Acceptance Criteria:**
+- [ ] When embeddings don't exist, search shows: "Semantic search unavailable. Using structural search. Run `mdcontext index --embed` for semantic search."
+- [ ] OR: Auto-prompt on first search: "Enable semantic search? (requires ~30s indexing)"
+- [ ] Search output shows mode indicator: `[semantic]` or `[structural]`
+- [ ] `mdcontext stats` shows embeddings status: "Embeddings: Yes/No (run index --embed to enable)"
+
+**Effort:** Low-Medium
+
+**Sources:** A2, FINAL-SYNTHESIS
+
+---
+
+### 3. Section-Level Context Extraction
+
+**Problem:** Agents couldn't request context for a specific section without retrieving the entire file. When investigating a specific subsection, full-file context wastes tokens.
+
+**Impact:** Over-retrieval or multiple refined searches needed. Forces choosing between full context or aggressive summarization.
+
+**Solution:** Enable section-targeted context extraction.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext context file.md --section "Memory Model"` returns only that section
+- [ ] `mdcontext context file.md:5.3` returns section 5.3 (by number)
+- [ ] `mdcontext context file.md --section "Memory*"` supports glob patterns
+- [ ] Nested sections included by default, `--shallow` flag for top-level only
+
+**Effort:** Medium-High
+
+**Sources:** A1, A2, C2, C4, FINAL-SYNTHESIS
+
+---
+
+## P1 - High Priority (Significant UX Improvement)
+
+### 4. Search Result Context Lines
+
+**Problem:** Search results show line numbers but minimal surrounding context. Hard to evaluate relevance without reading full sections.
+
+**Impact:** Extra commands needed to fetch context around matches. Agents requested grep-like `-C` behavior.
+
+**Solution:** Add context lines around search matches.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "checkpoint" -C 3` shows 3 lines before/after each match
+- [ ] `mdcontext search "checkpoint" -B 2 -A 5` shows 2 before, 5 after
+- [ ] Context lines clearly delineated from match lines
+- [ ] Works with both structural and semantic search
+
+**Effort:** Low-Medium
+
+**Sources:** B2, C3, C5, FINAL-SYNTHESIS
+
+---
+
+### 5. Remove 10-Result Limit / Add Pagination
+
+**Problem:** Default 10 results per query makes it hard to see ALL occurrences of a theme. Agents needed multiple queries to ensure comprehensive coverage.
+
+**Impact:** Incomplete results for common terms, extra commands for pagination workarounds.
+
+**Solution:** Add flags for result limit control.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "workflow" --all` shows all matches (no limit)
+- [ ] `mdcontext search "workflow" -n 50` shows up to 50 results
+- [ ] `mdcontext search "workflow" --offset 10 -n 10` for pagination
+- [ ] Default remains 10 for quick searches
+
+**Effort:** Low
+
+**Sources:** B1, B3, C1, FINAL-SYNTHESIS
+
+---
+
+### 6. Truncation UX Improvement
+
+**Problem:** When output truncated, agents couldn't selectively access missing sections. Truncation note appeared at end, not clearly signaled upfront.
+
+**Impact:** Forces over-retrieval or multiple refined searches. 50-96% reduction sometimes excessive.
+
+**Solution:** Better truncation signaling and navigation.
+
+**Acceptance Criteria:**
+- [ ] Truncation warning appears at TOP of output: "Output truncated (showing 2000/8500 tokens). Use --full or --section to retrieve more."
+- [ ] `mdcontext context file.md --full` shows complete content (no truncation)
+- [ ] Truncated output shows which sections were included/excluded
+- [ ] `mdcontext context file.md --sections` lists available sections for targeted retrieval
+
+**Effort:** Medium
+
+**Sources:** A2, B2, C2, FINAL-SYNTHESIS
+
+---
+
+### 7. Phrase Search with Quotes
+
+**Problem:** No way to search for exact phrases. "context resumption" as two words found irrelevant matches.
+
+**Impact:** Reduced precision for multi-word concepts.
+
+**Solution:** Support quoted phrase search.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search '"context resumption"'` matches exact phrase only
+- [ ] `mdcontext search '"drift-free"'` matches hyphenated terms
+- [ ] Can combine with boolean: `mdcontext search '"context resumption" AND drift`
+- [ ] `mdcontext search --help` documents phrase syntax
+
+**Effort:** Medium
+
+**Sources:** B1, B2, FINAL-SYNTHESIS
+
+---
+
+## P2 - Medium Priority (Nice to Have)
+
+### 8. Cross-File Link Analysis
+
+**Problem:** No command to find which files reference which concepts. Backlinks/links commands showed 0 for most queries.
+
+**Impact:** Manual grep needed to understand document relationships. Particularly wanted by Strategy C agents for architectural investigation.
+
+**Solution:** Implement concept-based cross-file analysis.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext refs "Execution Context"` shows all files mentioning this concept
+- [ ] `mdcontext refs --graph` outputs dependency graph (mermaid/dot format)
+- [ ] `mdcontext backlinks file.md` shows files that link TO this file
+- [ ] Works with markdown links `[text](file.md)` and concept mentions
+
+**Effort:** High
+
+**Sources:** C1, C6, FINAL-SYNTHESIS
+
+---
+
+### 9. Neighborhood View Around Search Results
+
+**Problem:** Search results show isolated matches. Agents wanted to see adjacent sections for context without fetching entire file.
+
+**Impact:** Extra context commands needed to understand match surroundings.
+
+**Solution:** Add section-level context around search results.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search "checkpoint" --context-sections 1` shows 1 section before/after each match
+- [ ] Section context clearly labeled with headers
+- [ ] Works with structural and semantic search
+
+**Effort:** Medium
+
+**Sources:** C2, C3, FINAL-SYNTHESIS
+
+---
+
+### 10. Search Mode Indicator
+
+**Problem:** Agents couldn't tell if search was using semantic or structural mode. Behavior varied based on embeddings presence.
+
+**Impact:** Confusion about why some searches worked differently than others.
+
+**Solution:** Always show search mode in output.
+
+**Acceptance Criteria:**
+- [ ] Search results header shows: `[semantic search]` or `[structural search]`
+- [ ] If semantic attempted but embeddings missing, show: `[structural search - embeddings not found]`
+- [ ] `mdcontext search --mode` flag to force mode: `--mode semantic` or `--mode structural`
+
+**Effort:** Low
+
+**Sources:** A2, FINAL-SYNTHESIS
+
+---
+
+### 11. Query Syntax Help
+
+**Problem:** Agents had to discover search syntax through trial and error. No examples in help.
+
+**Impact:** Wasted turns on failed queries, inconsistent usage patterns.
+
+**Solution:** Improve search help with examples.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext search --help` includes example section:
+  ```
+  EXAMPLES:
+    mdcontext search "auth"                    # Single term (structural)
+    mdcontext search "how to deploy"           # Semantic (if embeddings exist)
+    mdcontext search "auth AND deploy"         # Boolean AND
+    mdcontext search '"context resumption"'    # Exact phrase
+    mdcontext search "impl NOT test" -C 3      # Exclude term, show context
+  ```
+- [ ] `mdcontext search --examples` shows extended examples with explanations
+
+**Effort:** Low
+
+**Sources:** B2, B3, FINAL-SYNTHESIS
+
+---
+
+## P3 - Low Priority (Future Enhancements)
+
+### 12. Multi-File Glob Context
+
+**Problem:** Agents had to run context command on each file separately for batch operations.
+
+**Impact:** More commands needed for comprehensive extraction.
+
+**Solution:** Support glob patterns in context command.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext context "docs/**/*.md"` extracts context from all matching files
+- [ ] `mdcontext context "docs/*.md" -t 5000` applies token budget across all files
+- [ ] Output clearly shows which content came from which file
+
+**Effort:** Medium
+
+**Sources:** FINAL-SYNTHESIS
+
+---
+
+### 13. Saved Queries / Aliases
+
+**Problem:** Agents repeatedly ran similar complex queries.
+
+**Impact:** Command duplication, typo risk.
+
+**Solution:** Allow saving common query patterns.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext alias add arch-issues "search 'architecture AND (problem OR issue OR concern)'"`
+- [ ] `mdcontext arch-issues` runs saved query
+- [ ] Aliases stored in `.mdcontext/aliases.json`
+
+**Effort:** Medium
+
+**Sources:** FINAL-SYNTHESIS
+
+---
+
+### 14. Relevance Score Display
+
+**Problem:** Search results showed matches but not how relevant each match was.
+
+**Impact:** Harder to prioritize which results to investigate first.
+
+**Solution:** Show relevance/similarity scores.
+
+**Acceptance Criteria:**
+- [ ] Semantic search shows similarity score: `[0.87] docs/ARCH.md:45 - Control Plane...`
+- [ ] Results sorted by relevance by default
+- [ ] `--sort recent` flag to sort by file modification time instead
+
+**Effort:** Low-Medium
+
+**Sources:** C1, FINAL-SYNTHESIS
+
+---
+
+### 15. Troubleshooting Guide
+
+**Problem:** Agents encountered issues (0 results, truncation, embeddings) without clear resolution paths.
+
+**Impact:** Wasted turns debugging tool behavior.
+
+**Solution:** Add troubleshooting documentation.
+
+**Acceptance Criteria:**
+- [ ] `mdcontext troubleshoot` command shows common issues and fixes
+- [ ] Covers: 0 results, embeddings setup, truncation, index staleness
+- [ ] OR: Add troubleshooting section to README
+
+**Effort:** Low
+
+**Sources:** FINAL-SYNTHESIS
+
+---
+
+## Summary
+
+| Priority | Count | Theme |
+|----------|-------|-------|
+| P0 Critical | 3 | Boolean search, embeddings UX, section extraction |
+| P1 High | 4 | Context lines, pagination, truncation, phrases |
+| P2 Medium | 4 | Cross-file analysis, neighborhood view, mode indicator, help |
+| P3 Low | 4 | Glob context, aliases, relevance scores, docs |
+
+**Total: 15 actionable tasks**
+
+---
+
+## Validation Sources
+
+All tasks derived from agent feedback across three strategies:
+
+- **Strategy A:** A1, A2, A3, A-Synth (4 agents, by-folder approach)
+- **Strategy B:** B1, B2, B3, B-Synth (4 agents, by-question approach)
+- **Strategy C:** C1, C2-C6, C-Synth (7 agents, two-phase approach)
+- **Final Synthesis:** Cross-strategy analysis
+
+Full reports: `/Users/alphab/Dev/LLM/DEV/TMP/memory/reports/`

package/CONTRIBUTING.md

@@ -0,0 +1,186 @@
+# Contributing to mdcontext
+
+Thank you for your interest in contributing to mdcontext! This guide will help you get started.
+
+## Development Setup
+
+### Prerequisites
+
+- Node.js >= 18.0.0
+- pnpm (recommended, version 10+)
+- Python 3.12+ (for native module compilation)
+
+### Getting Started
+
+```bash
+# Clone the repository
+git clone https://github.com/mdcontext/mdcontext.git
+cd mdcontext
+
+# Install dependencies
+pnpm install
+
+# Build the project
+pnpm build
+
+# Run tests
+pnpm test
+```
+
+### Available Commands
+
+| Command | Description |
+|---------|-------------|
+| `pnpm build` | Build the project |
+| `pnpm test` | Run the test suite |
+| `pnpm test:all` | Run tests including semantic search tests (requires `OPENAI_API_KEY`) |
+| `pnpm typecheck` | Run TypeScript type checking |
+| `pnpm lint` | Run Biome linter |
+| `pnpm format` | Format code with Biome |
+| `pnpm check` | Run format, lint, and typecheck |
+
+## Making Changes
+
+1. **Create a branch** for your changes:
+   ```bash
+   git checkout -b feature/my-awesome-feature
+   ```
+
+2. **Make your changes** and ensure:
+   - All tests pass: `pnpm test`
+   - Type checking passes: `pnpm typecheck`
+   - Code is formatted: `pnpm format`
+
+3. **Create a changeset** (see below)
+
+4. **Submit a pull request** to the `main` branch
+
+## Changeset Workflow
+
+We use [Changesets](https://github.com/changesets/changesets) to manage versions and releases. **Every PR that affects the published package should include a changeset.**
+
+### Creating a Changeset
+
+After making your changes, run:
+
+```bash
+pnpm changeset
+```
+
+This will prompt you to:
+1. Select the package (mdcontext)
+2. Choose a bump type (patch/minor/major)
+3. Write a summary of your changes
+
+A markdown file will be created in the `.changeset/` directory. **Commit this file with your PR.**
+
+### Choosing a Bump Type
+
+| Type | When to use | Example |
+|------|-------------|---------|
+| **patch** | Bug fixes, documentation updates, internal refactors | Fixing a search result formatting bug |
+| **minor** | New features that are backwards compatible | Adding a new CLI flag or command |
+| **major** | Breaking changes to public API or CLI | Changing command syntax, removing flags |
+
+#### Examples
+
+**Patch** (bug fix):
+```markdown
+---
+"mdcontext": patch
+---
+
+Fixed section extraction to correctly handle nested headings
+```
+
+**Minor** (new feature):
+```markdown
+---
+"mdcontext": minor
+---
+
+Added --json flag to context command for structured output
+```
+
+**Major** (breaking change):
+```markdown
+---
+"mdcontext": major
+---
+
+Changed search command syntax: path argument now comes before query
+
+Migration: `mdcontext search "query" path/` becomes `mdcontext search path/ "query"`
+```
+
+### When You Don't Need a Changeset
+
+- Documentation-only changes (README, comments)
+- Test-only changes
+- CI/tooling changes that don't affect the package
+- Internal refactors with no user-facing changes
+
+If the changesets bot comments on your PR asking for a changeset and you believe one isn't needed, you can add an empty changeset:
+
+```bash
+pnpm changeset add --empty
+```
+
+## Release Process
+
+Here's what happens after your PR is merged:
+
+1. **Changesets are collected**: The release workflow detects changeset files in `main`
+
+2. **Version Packages PR is created**: GitHub Actions creates a PR titled "chore: release packages" that:
+   - Bumps the version in `package.json`
+   - Updates `CHANGELOG.md` with your changeset descriptions
+   - Deletes the changeset files
+
+3. **Publish on merge**: When a maintainer merges the "Version Packages" PR:
+   - The package is automatically published to npm
+   - A GitHub release is created
+   - Package provenance is attached via npm's OIDC support
+
+### For Maintainers
+
+To trigger a release:
+1. Review the auto-generated "Version Packages" PR
+2. Verify the version bump and changelog look correct
+3. Merge the PR
+4. The release happens automatically
+
+## Code Style
+
+- We use [Biome](https://biomejs.dev/) for formatting and linting
+- Run `pnpm format` before committing
+- TypeScript strict mode is enabled
+- Prefer functional patterns (Effect-TS is used throughout)
+
+## Testing
+
+- Write tests for new features and bug fixes
+- Tests are located in `*.test.ts` files alongside source files
+- Use [Vitest](https://vitest.dev/) for testing
+- Semantic search tests require `OPENAI_API_KEY` and are skipped by default
+
+### Running Specific Tests
+
+```bash
+# Run a specific test file
+pnpm vitest run src/search/searcher.test.ts
+
+# Run tests matching a pattern
+pnpm vitest run -t "keyword search"
+
+# Watch mode during development
+pnpm test:watch
+```
+
+## Questions?
+
+- Open an issue for bugs or feature requests
+- Check existing issues before creating new ones
+- For questions about using mdcontext, see the README and docs/
+
+Thank you for contributing!

package/NOTES/NOTES

@@ -0,0 +1,44 @@
+  ralph orchestrate search-enhancements
+  ralph orchestrate context-precision
+  ralph orchestrate embeddings-ux
+
+  - Semantic search with embeddings
+  - Boolean operators (AND, OR, NOT)
+  - Phrase search (exact match)
+  - Section-level context extraction
+  - Search context lines
+
+  Ideas for Part 3:
+  Option: A
+  Strategy: Feature Battle
+  Focus: Parallel agents each test ONE new feature (semantic,
+    boolean, phrase) - compare effectiveness
+  ────────────────────────────────────────
+  Option: B
+  Strategy: Meta-Dogfood
+  Focus: Use mdtldr to explore md-tldr's own codebase - eat our
+  own
+     dogfood
+  ────────────────────────────────────────
+  Option: C
+  Strategy: Query Refinement
+  Focus: Single agent that does iterative search, refining queries
+
+    based on results
+  ────────────────────────────────────────
+  Option: D
+  Strategy: Scale Test
+  Focus: Same docs corpus but with much larger token budgets, more
+
+-----
+
+
+Let have a subagent review
+  /Users/alphab/Dev/LLM/DEV/md-tldr/docs/002-research-embedding-
+  models.md
+  /Users/alphab/Dev/LLM/DEV/md-tldr/docs/003-research-rag-altern
+  atives.md
+  /Users/alphab/Dev/LLM/DEV/md-tldr/docs/004-research-vector-sea
+  rch.md
+
+  and document task candidates