mdcontext 0.0.1 → 0.2.0

package/docs/ROADMAP.md

@@ -0,0 +1,407 @@
+# Roadmap: @hw/mdcontext
+
+## Overview
+
+Build a token-efficient markdown analysis tool for LLM consumption. Each phase delivers testable functionality, building toward a complete system with parsing, semantic search, summarization, and analytics.
+
+## Phases
+
+- [ ] **Phase 1: Core Parsing** — Markdown AST extraction and structure
+- [ ] **Phase 2: Index & Storage** — Persistent indexes, file watching, caching
+- [ ] **Phase 3: Semantic Layer** — Embeddings, vector search
+- [ ] **Phase 4: Summarization** — Hierarchical compression, token optimization
+- [ ] **Phase 5: Analytics** — Performance metrics, query tracking
+- [ ] **Phase 6: Integration** — CLI, MCP server, HumanWork skills
+
+---
+
+## Phase 1: Core Parsing
+
+**Goal:** Extract structured data from markdown files.
+
+### 1.1: Project Setup
+
+- Initialize `packages/hw_mdcontext` in monorepo
+- TypeScript + Effect setup
+- Test infrastructure (vitest)
+- Basic CI integration
+
+**Deliverables:**
+
+- Package scaffolding
+- Build working
+- First test passing
+
+### 1.2: Markdown Parser
+
+- Integrate remark/unified
+- Parse to mdast (Markdown AST)
+- Handle frontmatter (YAML)
+- Handle GFM extensions (tables, task lists)
+
+**Deliverables:**
+
+- `parse(content: string): MdastRoot`
+- Frontmatter extraction
+- Unit tests for various markdown features
+
+### 1.3: Structure Extraction
+
+- Extract heading hierarchy
+- Identify sections (content between headings)
+- Extract code blocks with language tags
+- Extract links (internal, external, images)
+- Extract lists and tables
+
+**Deliverables:**
+
+- `extractStructure(ast): DocumentStructure`
+- Section tree with content
+- Link graph per document
+- Code block inventory
+
+### 1.4: Document Model
+
+- Define document schema (Effect Schema)
+- Section schema with metadata
+- Serialize/deserialize to JSON
+
+**Deliverables:**
+
+- `Document` type with full structure
+- `Section` type with bounds, content, metadata
+- JSON round-trip tests
+
+---
+
+## Phase 2: Index & Storage
+
+**Goal:** Persist parsed data, enable fast lookups, handle updates.
+
+### 2.1: Storage Interface
+
+- Define `MdStore` interface
+- In-memory implementation for testing
+- File-based implementation for persistence
+
+**Deliverables:**
+
+- `MdStore` interface (save, load, query)
+- `MemoryMdStore`
+- `FileMdStore` (JSON files in `.mdcontext/`)
+
+### 2.2: Document Indexing
+
+- Index documents by path
+- Index sections by heading
+- Index links (forward and back)
+- Incremental updates (changed files only)
+
+**Deliverables:**
+
+- Path → Document lookup
+- Heading → Section lookup
+- Backlink index
+- Change detection (mtime, hash)
+
+### 2.3: File Watching
+
+- Watch directory for changes
+- Debounce rapid changes
+- Incremental re-index
+- Configurable ignore patterns
+
+**Deliverables:**
+
+- `watch(dir, options): Effect<void>`
+- `.mdcontextignore` support
+- Debounce logic (default 500ms)
+
+### 2.4: Cache Management
+
+- Cache parsed documents
+- Cache structure indexes
+- Invalidation on file change
+- Size limits and eviction
+
+**Deliverables:**
+
+- LRU cache for documents
+- Index persistence to disk
+- Cache stats (hits, misses, size)
+
+---
+
+## Phase 3: Semantic Layer
+
+**Goal:** Enable meaning-based search via embeddings.
+
+### 3.1: Embedding Interface
+
+- Define `Embedder` interface
+- Pluggable backends (API, local)
+- Batch embedding support
+
+**Deliverables:**
+
+- `Embedder` interface
+- `embed(texts: string[]): Effect<Vector[]>`
+- Configuration for model selection
+
+### 3.2: OpenAI Embeddings
+
+- Implement OpenAI text-embedding-3-small
+- Rate limiting and retry logic
+- Cost tracking
+
+**Deliverables:**
+
+- `OpenAIEmbedder`
+- Automatic batching (max 8k tokens)
+- Cost per query metric
+
+### 3.3: Local Embeddings (Optional)
+
+- Python subprocess for sentence-transformers
+- Or ONNX runtime in Node
+- Fallback when API unavailable
+
+**Deliverables:**
+
+- `LocalEmbedder` (stretch goal)
+- Model download management
+
+### 3.4: Vector Index
+
+- Store embeddings with document/section IDs
+- Similarity search (cosine)
+- FAISS or hnswlib integration
+
+**Deliverables:**
+
+- `VectorIndex` interface
+- `search(query: Vector, k: number): Result[]`
+- Persistence to disk
+
+### 3.5: Semantic Search API
+
+- Text query → embed → search
+- Combine with structural filters
+- Rank and return results
+
+**Deliverables:**
+
+- `semanticSearch(query: string, options): SearchResult[]`
+- Filter by path pattern, heading level
+- Result with score, snippet, location
+
+---
+
+## Phase 4: Summarization
+
+**Goal:** Generate token-efficient summaries at multiple granularities.
+
+### 4.1: Token Counting
+
+- Accurate token counting (tiktoken or similar)
+- Budget management
+- Truncation strategies
+
+**Deliverables:**
+
+- `countTokens(text: string): number`
+- `truncateToTokens(text, limit): string`
+- Model-specific tokenizers (GPT-4, Claude)
+
+### 4.2: Section Summarization
+
+- Extract key points from section
+- Preserve structure indicators
+- Configurable compression ratio
+
+**Deliverables:**
+
+- `summarizeSection(section, options): Summary`
+- Key sentence extraction
+- Heading preservation
+
+### 4.3: Document Summarization
+
+- Hierarchical: summarize sections, then combine
+- TOC generation
+- Key topics extraction
+
+**Deliverables:**
+
+- `summarizeDocument(doc, options): DocSummary`
+- Multi-level output (100, 500, 2000 tokens)
+- Topic list
+
+### 4.4: Context Assembly
+
+- Build LLM-ready context from multiple sources
+- Priority-based inclusion
+- Token budget management
+
+**Deliverables:**
+
+- `assembleContext(sources, budget): string`
+- Source attribution
+- Overflow handling (truncate vs omit)
+
+---
+
+## Phase 5: Analytics
+
+**Goal:** Built-in observability for performance and usage.
+
+### 5.1: Metrics Foundation
+
+- Effect Metrics integration
+- Counter, Gauge, Histogram types
+- Metric naming conventions
+
+**Deliverables:**
+
+- Metrics layer setup
+- Standard metric types
+- Tagging (operation, status)
+
+### 5.2: Performance Metrics
+
+- Query latency (p50, p95, p99)
+- Index build time
+- Cache hit/miss rates
+- Embedding API latency
+
+**Deliverables:**
+
+- `mdcontext_query_duration_ms` histogram
+- `mdcontext_cache_hits_total` counter
+- `mdcontext_index_build_duration_ms` gauge
+
+### 5.3: Usage Metrics
+
+- Queries per time period
+- Token usage (input/output)
+- Most queried documents/sections
+- Search result click-through (if applicable)
+
+**Deliverables:**
+
+- `mdcontext_queries_total` counter
+- `mdcontext_tokens_used` counter
+- Query log with timestamps
+
+### 5.4: Reporting
+
+- Metrics export (Prometheus format)
+- Simple CLI report command
+- Alerting thresholds (optional)
+
+**Deliverables:**
+
+- `mdcontext metrics` CLI command
+- JSON and text output formats
+- Configurable retention
+
+---
+
+## Phase 6: Integration
+
+**Goal:** Make mdcontext usable from CLI, MCP, and HumanWork.
+
+### 6.1: CLI Tool
+
+- `mdcontext index <dir>` — build index
+- `mdcontext search <query>` — semantic search
+- `mdcontext context <path>` — LLM-ready summary
+- `mdcontext structure <path>` — show document structure
+
+**Deliverables:**
+
+- CLI with subcommands
+- Output formats (text, JSON)
+- Config file support
+
+### 6.2: Daemon Mode
+
+- `mdcontext daemon` — run as background service
+- HTTP/IPC API for queries
+- Auto-rebuild on changes
+
+**Deliverables:**
+
+- Daemon process management
+- Query API (REST or IPC)
+- Health check endpoint
+
+### 6.3: MCP Server
+
+- Expose tools for Claude integration
+- `md_search` — semantic search
+- `md_context` — get context for file/section
+- `md_structure` — document outline
+
+**Deliverables:**
+
+- MCP server implementation
+- Tool definitions
+- Claude Desktop/Code integration docs
+
+### 6.4: HumanWork Skills
+
+- `hw-md-search` — search markdown in .humanwork/
+- `hw-md-context` — get context for task/session
+- Integration with session-history
+
+**Deliverables:**
+
+- Skill definitions
+- Integration with existing HumanWork skills
+- Documentation
+
+---
+
+## Progress
+
+| Phase              | Status      | Plans | Completed |
+| ------------------ | ----------- | ----- | --------- |
+| 1. Core Parsing    | Not started | 4     | -         |
+| 2. Index & Storage | Not started | 4     | -         |
+| 3. Semantic Layer  | Not started | 5     | -         |
+| 4. Summarization   | Not started | 4     | -         |
+| 5. Analytics       | Not started | 4     | -         |
+| 6. Integration     | Not started | 4     | -         |
+
+**Total: 25 tasks across 6 phases**
+
+---
+
+## Dependencies
+
+```
+Phase 1 ─────────────────────────────────────────┐
+    │                                            │
+    ▼                                            │
+Phase 2 ──────────────┐                          │
+    │                 │                          │
+    ▼                 ▼                          │
+Phase 3          Phase 4                         │
+    │                 │                          │
+    └────────┬────────┘                          │
+             ▼                                   │
+         Phase 5 ◄───────────────────────────────┘
+             │
+             ▼
+         Phase 6
+```
+
+- Phase 2 depends on Phase 1 (need parser for indexing)
+- Phase 3 & 4 can parallel after Phase 2
+- Phase 5 spans all (analytics hooks added throughout)
+- Phase 6 integrates everything
+
+---
+
+_Created: 2025-01-18_

package/docs/summarization.md

@@ -0,0 +1,320 @@
+# AI Summarization Architecture
+
+This document covers the architecture and implementation details of mdcontext's AI-powered search result summarization feature.
+
+## Overview
+
+mdcontext can generate AI-powered summaries of search results using either:
+
+1. **CLI tools** (Claude Code, Copilot CLI, OpenCode) - Free with your subscription
+2. **API providers** (DeepSeek, Anthropic, OpenAI, Gemini) - Pay per query
+
+The design prioritizes CLI providers as the primary option since they leverage existing subscriptions that developers already have.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                        CLI (search.ts)                          │
+│  --summarize flag triggers summarization pipeline               │
+└─────────────────────────┬───────────────────────────────────────┘
+                          │
+                          ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                   Provider Factory                               │
+│  getBestAvailableSummarizer() / createSummarizer()              │
+│  - Detects installed CLI tools                                   │
+│  - Creates appropriate provider instance                         │
+└─────────────────────────┬───────────────────────────────────────┘
+                          │
+          ┌───────────────┴───────────────┐
+          ▼                               ▼
+┌─────────────────────┐       ┌─────────────────────┐
+│   CLI Providers     │       │   API Providers     │
+│   (Free)            │       │   (Pay-per-use)     │
+│                     │       │                     │
+│ - ClaudeCLI         │       │ - DeepSeek          │
+│ - OpenCode          │       │ - Anthropic         │
+│ - Copilot           │       │ - OpenAI            │
+│ - Aider             │       │ - Gemini            │
+│ - Cline             │       │ - Qwen              │
+└─────────────────────┘       └─────────────────────┘
+          │                               │
+          └───────────────┬───────────────┘
+                          ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                    Summarizer Interface                          │
+│  summarize(input, prompt) → SummaryResult                       │
+│  summarizeStream(input, prompt, options) → void                 │
+│  estimateCost(inputTokens) → number                             │
+│  isAvailable() → boolean                                         │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Components
+
+### Provider Detection (`cli-providers/detection.ts`)
+
+Automatically discovers installed CLI tools:
+
+```typescript
+import { detectInstalledCLIs } from './summarization/index.js'
+
+const installed = await detectInstalledCLIs()
+// [{ name: 'claude', command: 'claude', displayName: 'Claude Code', ... }]
+```
+
+Detection uses `which` (Unix) or `where` (Windows) via `spawn()` - never shell interpolation.
+
+### Provider Factory (`provider-factory.ts`)
+
+Creates summarizer instances based on configuration:
+
+```typescript
+import { createSummarizer, getBestAvailableSummarizer } from './summarization/index.js'
+
+// Auto-detect best available provider
+const result = await getBestAvailableSummarizer()
+if (result) {
+  const { summarizer, config } = result
+  // Use summarizer...
+}
+
+// Or create from explicit config
+const summarizer = await createSummarizer({
+  mode: 'cli',
+  provider: 'claude',
+})
+```
+
+### Cost Estimation (`cost.ts`)
+
+Estimates costs before execution:
+
+```typescript
+import { estimateSummaryCost, formatCostDisplay } from './summarization/index.js'
+
+const estimate = estimateSummaryCost(inputText, 'api', 'deepseek')
+// {
+//   inputTokens: 2500,
+//   outputTokens: 500,
+//   estimatedCost: 0.0007,
+//   provider: 'deepseek',
+//   isPaid: true,
+//   formattedCost: '$0.0007'
+// }
+
+console.log(formatCostDisplay(estimate))
+// "Estimated cost: $0.0007"
+```
+
+CLI providers always return `isPaid: false` with `formattedCost: 'FREE (subscription)'`.
+
+### Prompt Templates (`prompts.ts`)
+
+Pre-built prompts for different summarization styles:
+
+| Template | Description |
+|----------|-------------|
+| `default` | Balanced summary with key findings |
+| `concise` | 2-3 sentence quick summary |
+| `detailed` | Comprehensive analysis |
+| `actionable` | Focus on next steps |
+| `technical` | Code patterns and API details |
+
+```typescript
+import { buildPrompt } from './summarization/index.js'
+
+const prompt = buildPrompt({
+  query: 'authentication',
+  resultCount: 10,
+  searchMode: 'hybrid',
+}, 'actionable')
+```
+
+### Error Handling (`error-handler.ts`)
+
+Graceful degradation on failures:
+
+```typescript
+import { displaySummarizationError, isRecoverableError } from './summarization/index.js'
+
+try {
+  await summarizer.summarize(input, prompt)
+} catch (error) {
+  if (isRecoverableError(error)) {
+    // Retry logic
+  } else {
+    displaySummarizationError(error)
+    // Shows user-friendly message, search results still displayed
+  }
+}
+```
+
+## Security Considerations
+
+### Shell Injection Prevention
+
+All CLI invocations use `spawn()` with argument arrays - **NEVER** `exec()` with string interpolation:
+
+```typescript
+// CORRECT - Safe from shell injection
+spawn('claude', ['-p', userInput, '--output-format', 'text'])
+
+// WRONG - Vulnerable to shell injection
+exec(`claude -p "${userInput}"`)  // NEVER DO THIS
+```
+
+This is enforced throughout the codebase. User input is passed as array elements, never interpolated into shell commands.
+
+### API Key Handling
+
+- API keys are sourced from environment variables only
+- Never stored in config files
+- Environment variable names follow provider conventions:
+  - `DEEPSEEK_API_KEY`
+  - `ANTHROPIC_API_KEY`
+  - `OPENAI_API_KEY`
+  - `GOOGLE_API_KEY` (for Gemini)
+  - `QWEN_API_KEY`
+
+### Timeout Protection
+
+CLI processes have a default 60-second timeout to prevent hung processes.
+
+## Adding New Providers
+
+### CLI Provider
+
+1. Add to `KNOWN_CLIS` in `cli-providers/detection.ts`:
+
+```typescript
+{
+  name: 'newcli',
+  command: 'newcli',
+  displayName: 'New CLI Tool',
+  args: ['--prompt'],
+  useStdin: false,
+}
+```
+
+2. Create implementation in `cli-providers/newcli.ts`:
+
+```typescript
+import { spawn } from 'node:child_process'
+import type { Summarizer, SummaryResult } from '../types.js'
+
+export class NewCLISummarizer implements Summarizer {
+  async summarize(input: string, prompt: string): Promise<SummaryResult> {
+    // SECURITY: Always use spawn() with argument arrays
+    const proc = spawn('newcli', ['--prompt', prompt, input])
+    // ... implementation
+  }
+
+  async isAvailable(): Promise<boolean> {
+    // Check if CLI is installed
+  }
+}
+```
+
+3. Add to factory in `provider-factory.ts`
+
+### API Provider
+
+1. Add pricing to `cost.ts`:
+
+```typescript
+export const API_PRICING = {
+  // ... existing providers
+  newapi: { input: 0.50, output: 1.00, displayName: 'New API' },
+}
+```
+
+2. Create implementation using Vercel AI SDK (when implemented):
+
+```typescript
+import { createOpenAI } from '@ai-sdk/openai'
+
+export class NewAPISummarizer implements Summarizer {
+  // Use Vercel AI SDK for OpenAI-compatible APIs
+}
+```
+
+## Performance
+
+| Provider Type | Latency | Cost |
+|--------------|---------|------|
+| CLI (Claude) | 2-5s | Free |
+| CLI (OpenCode) | 2-5s | Free |
+| API (DeepSeek) | 1-3s | ~$0.0007/query |
+| API (OpenAI) | 1-2s | ~$0.005/query |
+
+### Token Limits
+
+- Input is automatically truncated at 100K characters (~25K tokens)
+- Result content is truncated to 500 chars per result
+- Output tokens capped at 500 for cost estimates
+
+## Configuration Reference
+
+### Config File
+
+```javascript
+// mdcontext.config.js
+/** @type {import('mdcontext').PartialMdContextConfig} */
+export default {
+  aiSummarization: {
+    mode: 'cli',           // 'cli' or 'api'
+    provider: 'claude',    // Provider name
+    model: 'deepseek-chat', // Model for API providers
+    stream: false,         // Enable streaming
+  },
+}
+```
+
+### Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `MDCONTEXT_AISUMMARIZATION_MODE` | 'cli' or 'api' |
+| `MDCONTEXT_AISUMMARIZATION_PROVIDER` | Provider name |
+| `MDCONTEXT_AISUMMARIZATION_MODEL` | Model name (API only) |
+| `MDCONTEXT_AISUMMARIZATION_STREAM` | 'true' or 'false' |
+
+## Troubleshooting
+
+### "CLI tool 'claude' not found"
+
+**Solution:** Install Claude Code from https://claude.ai/download
+
+### "CLI tool 'opencode' not found"
+
+**Solution:** Install OpenCode from https://github.com/opencode-ai/opencode
+
+### "Authentication failed for anthropic"
+
+**Solution:** Set API key: `export ANTHROPIC_API_KEY=sk-...`
+
+### "Rate limit exceeded"
+
+**Solution:** Wait and retry. Consider switching to CLI provider (free).
+
+### "Summarization failed: timeout"
+
+**Solution:** Reduce result set with `--limit` or increase timeout in config.
+
+### "No summarization providers available"
+
+**Solution:** Either:
+1. Install a CLI tool (Claude Code, OpenCode)
+2. Configure an API provider with valid API key
+
+### OpenCode JSON format errors
+
+**Solution:** OpenCode JSON format is undocumented. Try updating OpenCode or switch to Claude CLI.
+
+## Related Documentation
+
+- [README.md](../README.md#ai-summarization) - Quick start guide
+- [CONFIG.md](./CONFIG.md) - Full configuration reference
+- [ERRORS.md](./ERRORS.md) - Error handling patterns