mdcontext 0.1.0 → 0.2.0

package/docs/ERRORS.md

@@ -0,0 +1,383 @@
+# Error Handling Patterns
+
+This document describes the error handling conventions used in mdcontext, following Effect's "errors as values" philosophy.
+
+## Error Type Taxonomy
+
+All domain errors are defined in `src/errors/index.ts` using Effect's `Data.TaggedError`:
+
+```typescript
+export class FileReadError extends Data.TaggedError('FileReadError')<{
+  readonly path: string
+  readonly message: string
+  readonly cause?: unknown
+}> {
+  get code() { return ErrorCode.FILE_READ }
+}
+```
+
+### Error Categories
+
+- **File System**: `FileReadError`, `FileWriteError`, `DirectoryCreateError`, `DirectoryWalkError`
+- **Parsing**: `ParseError` (for markdown parsing failures)
+- **API**: `ApiKeyMissingError`, `ApiKeyInvalidError`
+- **Embeddings**: `EmbeddingError` (rate limits, quota, network failures)
+- **Index**: `IndexNotFoundError`, `IndexCorruptedError`, `IndexBuildError`
+- **Search**: `DocumentNotFoundError`, `EmbeddingsNotFoundError`
+- **Config**: `ConfigError`
+- **Vector Store**: `VectorStoreError`
+- **Watch**: `WatchError`
+- **CLI**: `CliValidationError`
+
+## Error Codes
+
+Each error type has a unique error code for programmatic handling. Error codes are stable identifiers that don't change when messages are updated.
+
+### Code Format
+
+Codes follow the pattern `E{category}{number}`:
+
+| Category | Code Range | Description |
+|----------|------------|-------------|
+| File System | E1xx | File and directory operations |
+| Parse | E2xx | Markdown parsing errors |
+| API | E3xx | API authentication and embedding errors |
+| Index | E4xx | Index operations |
+| Search | E5xx | Search operations |
+| Vector Store | E6xx | Vector store operations |
+| Config | E7xx | Configuration errors |
+| Watch | E8xx | File watcher errors |
+| CLI | E9xx | CLI validation errors |
+
+### Error Code Reference
+
+| Code | Error Type | Description |
+|------|------------|-------------|
+| E100 | FileReadError | Cannot read file |
+| E101 | FileWriteError | Cannot write file |
+| E102 | DirectoryCreateError | Cannot create directory |
+| E103 | DirectoryWalkError | Cannot traverse directory |
+| E200 | ParseError | Markdown parsing failed |
+| E300 | ApiKeyMissingError | API key not set in environment |
+| E301 | ApiKeyInvalidError | API key rejected by provider |
+| E310 | EmbeddingError (RateLimit) | Rate limit exceeded |
+| E311 | EmbeddingError (QuotaExceeded) | API quota exceeded |
+| E312 | EmbeddingError (Network) | Network error during embedding |
+| E313 | EmbeddingError (ModelError) | Model error |
+| E319 | EmbeddingError (Unknown) | Unknown embedding error |
+| E400 | IndexNotFoundError | Index does not exist |
+| E401 | IndexCorruptedError | Index is corrupted |
+| E402 | IndexBuildError | Failed to build index |
+| E500 | DocumentNotFoundError | Document not in index |
+| E501 | EmbeddingsNotFoundError | Embeddings not found |
+| E600 | VectorStoreError | Vector store operation failed |
+| E700 | ConfigError | Configuration error |
+| E800 | WatchError | File watcher error |
+| E900 | CliValidationError | Invalid CLI arguments |
+
+### Exit Codes
+
+CLI exit codes map to error categories:
+
+| Exit Code | Category | Description |
+|-----------|----------|-------------|
+| 0 | Success | Operation completed successfully |
+| 1 | User Error | Invalid arguments, missing config, etc. |
+| 2 | System Error | File system, network, etc. |
+| 3 | API Error | Authentication, rate limits, etc. |
+
+### Usage in Scripts
+
+Error codes enable reliable scripting and CI/CD integration:
+
+```bash
+# Check for specific error codes in output
+mdcontext search "query" 2>&1 | grep -q "\[E400\]" && echo "Index not found"
+
+# Use exit codes for control flow
+mdcontext index || {
+  case $? in
+    1) echo "User error - check arguments" ;;
+    2) echo "System error - check permissions" ;;
+    3) echo "API error - check credentials" ;;
+  esac
+}
+```
+
+### Programmatic Access
+
+```typescript
+import { FileReadError, ErrorCode } from './errors/index.js'
+
+const error = new FileReadError({ path: '/file.md', message: 'ENOENT' })
+console.log(error.code) // 'E100'
+console.log(error._tag) // 'FileReadError'
+```
+
+## Transformation Patterns
+
+### 1. `mapError` - Transform Error Types
+
+Use `mapError` to convert low-level errors to domain errors. This preserves error specificity while adapting the error type.
+
+```typescript
+// GOOD - Maps to domain error with context
+parse(content, options).pipe(
+  Effect.mapError((e) =>
+    new ParseError({
+      message: e.message,
+      path: filePath,
+      cause: e,
+    })
+  )
+)
+
+// BAD - Loses type information
+Effect.mapError((e) => new Error(`${e._tag}: ${e.message}`))
+```
+
+**When to use:**
+- Converting library errors to domain errors
+- Adding context (path, operation) to errors
+- Translating between error domains
+
+### 2. `catchTag` / `catchTags` - Handle Specific Errors
+
+Use `catchTag` when you need to handle a specific known error type. This enables exhaustive error handling and type-safe recovery.
+
+```typescript
+// Handle specific error with recovery
+estimateEmbeddingCost(dir).pipe(
+  Effect.catchTag('IndexNotFoundError', () =>
+    Effect.succeed(null)  // Index doesn't exist, return null estimate
+  )
+)
+
+// Handle multiple specific errors
+buildEmbeddings(dir).pipe(
+  Effect.catchTags({
+    ApiKeyMissingError: (e) => {
+      console.error(e.message)
+      return Effect.succeed(null)
+    },
+    ApiKeyInvalidError: (e) => {
+      console.error(e.message)
+      return Effect.succeed(null)
+    },
+  })
+)
+```
+
+**When to use:**
+- Recovering from expected error conditions
+- Providing fallback values for specific failures
+- Implementing retry logic for transient errors
+- Filtering/handling known error types mid-pipeline
+
+### 3. `catchAll` - Boundary Error Handling
+
+Use `catchAll` **only at system boundaries** where all errors must be converted to a final format (user message, JSON response, etc.).
+
+```typescript
+// GOOD - At CLI boundary (main.ts)
+program.pipe(
+  Effect.catchAll((error) => {
+    console.error(formatError(error))
+    return Effect.succeed(ExitCode.failure)
+  })
+)
+
+// GOOD - At MCP boundary (server.ts)
+// MCP protocol requires JSON responses for all operations
+handler.pipe(
+  Effect.catchAll((e) =>
+    Effect.succeed({
+      isError: true,
+      content: [{ type: 'text', text: `Error: ${e.message}` }]
+    })
+  )
+)
+
+// BAD - In middle of pipeline (loses type information)
+readFile(path).pipe(
+  Effect.catchAll(() => Effect.succeed(null))  // Silent failure!
+)
+```
+
+**When to use:**
+- CLI entry points (converting to exit codes)
+- MCP/API handlers (converting to protocol responses)
+- Top-level program error handling
+
+**When NOT to use:**
+- Middle of pipelines (use `catchTag` instead)
+- When error type information is needed downstream
+- For silent failures without logging
+
+### 4. `Effect.tryPromise` / `Effect.try` - Lift External Operations
+
+Use these to wrap promise-based or synchronous operations, converting thrown errors to Effect failures.
+
+```typescript
+// For promises
+Effect.tryPromise({
+  try: () => fs.readFile(path, 'utf-8'),
+  catch: (e) => new FileReadError({
+    path,
+    message: e instanceof Error ? e.message : String(e),
+    cause: e,
+  })
+})
+
+// For synchronous code that may throw
+Effect.try({
+  try: () => JSON.parse(content),
+  catch: (e) => new IndexCorruptedError({
+    path,
+    reason: 'InvalidJson',
+    details: e instanceof Error ? e.message : undefined,
+  })
+})
+```
+
+## Best Practices
+
+### Do's
+
+- **Always use domain errors** - Never map to generic `Error`
+- **Preserve cause chains** - Include `cause` field for debugging
+- **Add context** - Include path, operation, and relevant metadata
+- **Document error types** - Use JSDoc to specify thrown errors
+- **Log at boundaries** - When swallowing errors, log for debugging
+
+### Don'ts
+
+- **Don't swallow errors silently** - Always log or handle explicitly
+- **Don't use `catchAll` mid-pipeline** - Use `catchTag` instead
+- **Don't mix paradigms** - Avoid try/catch inside Effect.gen
+- **Don't map to generic Error** - Always use typed domain errors
+
+### Batch Processing Pattern
+
+When processing multiple items where individual failures shouldn't stop the batch:
+
+```typescript
+const processFile = Effect.gen(function* () {
+  // ... processing logic
+}).pipe(
+  // Note: catchAll intentional for batch processing
+  // Individual file failures collected in errors array
+  // rather than stopping the entire operation
+  Effect.catchAll((error) => {
+    errors.push({ path, message: error.message })
+    return Effect.void
+  })
+)
+```
+
+Always add a comment explaining why `catchAll` is appropriate.
+
+### Graceful Degradation Pattern
+
+When a feature is optional and failure shouldn't block the main operation:
+
+```typescript
+// Optional embedding cost estimate for user prompt
+const estimate = yield* estimateEmbeddingCost(dir).pipe(
+  Effect.catchTag('IndexNotFoundError', () => Effect.succeed(null)),
+  // Note: catchAll for graceful degradation
+  // This is optional information - failure shouldn't block indexing
+  Effect.catchAll((e) => {
+    Effect.runSync(Effect.logWarning(`Could not estimate: ${e.message}`))
+    return Effect.succeed(null)
+  })
+)
+```
+
+## Summarization Error Handling
+
+AI summarization uses a separate error system with graceful degradation - errors never prevent search results from being displayed.
+
+### Summarization Error Codes
+
+| Code | Error Type | Description |
+|------|------------|-------------|
+| PROVIDER_NOT_FOUND | Provider name unknown | Check provider spelling |
+| PROVIDER_NOT_AVAILABLE | CLI tool not installed | Install the CLI tool |
+| CLI_EXECUTION_FAILED | CLI process error | Check CLI authentication |
+| API_REQUEST_FAILED | API call failed | Check API key and network |
+| RATE_LIMITED | Too many requests | Wait and retry |
+| INVALID_RESPONSE | Bad provider response | Report as bug |
+| TIMEOUT | Request timed out | Reduce result set |
+| NO_API_KEY | Missing API key | Set environment variable |
+
+### Troubleshooting Summarization
+
+**"CLI tool 'claude' not found"**
+```bash
+# Install Claude Code
+# Visit: https://claude.ai/download
+```
+
+**"CLI tool 'opencode' not found"**
+```bash
+# Install OpenCode
+npm install -g @opencode/cli
+# Or: https://github.com/opencode-ai/opencode
+```
+
+**"Authentication failed for anthropic"**
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+**"Rate limit exceeded"**
+- Wait 60 seconds and retry
+- Consider switching to CLI provider (free with subscription)
+
+**"Summarization failed: timeout"**
+- Reduce results: `mdcontext search "query" --limit 5 --summarize`
+- The default timeout is 60 seconds
+
+**"No summarization providers available"**
+Either:
+1. Install a CLI tool: `claude`, `opencode`, or `gh copilot`
+2. Configure an API provider with a valid API key
+
+### Graceful Degradation
+
+Summarization errors never crash the CLI. When summarization fails:
+
+1. Error message is displayed
+2. Search results are shown normally
+3. Exit code remains 0 (success)
+
+```typescript
+// Implementation pattern in search.ts
+const runSummarization = (options: SummarizationOptions): Effect.Effect<void, never> =>
+  runSummarizationUnsafe(options).pipe(
+    Effect.catchAll((error) =>
+      Effect.sync(() => {
+        displaySummarizationError(error)
+        // Search results still displayed
+      }),
+    ),
+  )
+```
+
+## Error Formatting
+
+Error formatting (user-friendly messages) should only happen at the CLI boundary in `src/cli/error-handler.ts`. Internal errors carry structured data; presentation is separate from logic.
+
+```typescript
+// src/cli/error-handler.ts
+const formatError = (error: MdContextError): string => {
+  switch (error._tag) {
+    case 'FileReadError':
+      return `Cannot read file: ${error.path}\n${error.message}`
+    case 'ApiKeyMissingError':
+      return `API key not configured.\n\nSet ${error.envVar} environment variable.`
+    // ... other error types
+  }
+}
+```

package/docs/summarization.md

@@ -0,0 +1,320 @@
+# AI Summarization Architecture
+
+This document covers the architecture and implementation details of mdcontext's AI-powered search result summarization feature.
+
+## Overview
+
+mdcontext can generate AI-powered summaries of search results using either:
+
+1. **CLI tools** (Claude Code, Copilot CLI, OpenCode) - Free with your subscription
+2. **API providers** (DeepSeek, Anthropic, OpenAI, Gemini) - Pay per query
+
+The design prioritizes CLI providers as the primary option since they leverage existing subscriptions that developers already have.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                        CLI (search.ts)                          │
+│  --summarize flag triggers summarization pipeline               │
+└─────────────────────────┬───────────────────────────────────────┘
+                          │
+                          ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                   Provider Factory                               │
+│  getBestAvailableSummarizer() / createSummarizer()              │
+│  - Detects installed CLI tools                                   │
+│  - Creates appropriate provider instance                         │
+└─────────────────────────┬───────────────────────────────────────┘
+                          │
+          ┌───────────────┴───────────────┐
+          ▼                               ▼
+┌─────────────────────┐       ┌─────────────────────┐
+│   CLI Providers     │       │   API Providers     │
+│   (Free)            │       │   (Pay-per-use)     │
+│                     │       │                     │
+│ - ClaudeCLI         │       │ - DeepSeek          │
+│ - OpenCode          │       │ - Anthropic         │
+│ - Copilot           │       │ - OpenAI            │
+│ - Aider             │       │ - Gemini            │
+│ - Cline             │       │ - Qwen              │
+└─────────────────────┘       └─────────────────────┘
+          │                               │
+          └───────────────┬───────────────┘
+                          ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                    Summarizer Interface                          │
+│  summarize(input, prompt) → SummaryResult                       │
+│  summarizeStream(input, prompt, options) → void                 │
+│  estimateCost(inputTokens) → number                             │
+│  isAvailable() → boolean                                         │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Components
+
+### Provider Detection (`cli-providers/detection.ts`)
+
+Automatically discovers installed CLI tools:
+
+```typescript
+import { detectInstalledCLIs } from './summarization/index.js'
+
+const installed = await detectInstalledCLIs()
+// [{ name: 'claude', command: 'claude', displayName: 'Claude Code', ... }]
+```
+
+Detection uses `which` (Unix) or `where` (Windows) via `spawn()` - never shell interpolation.
+
+### Provider Factory (`provider-factory.ts`)
+
+Creates summarizer instances based on configuration:
+
+```typescript
+import { createSummarizer, getBestAvailableSummarizer } from './summarization/index.js'
+
+// Auto-detect best available provider
+const result = await getBestAvailableSummarizer()
+if (result) {
+  const { summarizer, config } = result
+  // Use summarizer...
+}
+
+// Or create from explicit config
+const summarizer = await createSummarizer({
+  mode: 'cli',
+  provider: 'claude',
+})
+```
+
+### Cost Estimation (`cost.ts`)
+
+Estimates costs before execution:
+
+```typescript
+import { estimateSummaryCost, formatCostDisplay } from './summarization/index.js'
+
+const estimate = estimateSummaryCost(inputText, 'api', 'deepseek')
+// {
+//   inputTokens: 2500,
+//   outputTokens: 500,
+//   estimatedCost: 0.0007,
+//   provider: 'deepseek',
+//   isPaid: true,
+//   formattedCost: '$0.0007'
+// }
+
+console.log(formatCostDisplay(estimate))
+// "Estimated cost: $0.0007"
+```
+
+CLI providers always return `isPaid: false` with `formattedCost: 'FREE (subscription)'`.
+
+### Prompt Templates (`prompts.ts`)
+
+Pre-built prompts for different summarization styles:
+
+| Template | Description |
+|----------|-------------|
+| `default` | Balanced summary with key findings |
+| `concise` | 2-3 sentence quick summary |
+| `detailed` | Comprehensive analysis |
+| `actionable` | Focus on next steps |
+| `technical` | Code patterns and API details |
+
+```typescript
+import { buildPrompt } from './summarization/index.js'
+
+const prompt = buildPrompt({
+  query: 'authentication',
+  resultCount: 10,
+  searchMode: 'hybrid',
+}, 'actionable')
+```
+
+### Error Handling (`error-handler.ts`)
+
+Graceful degradation on failures:
+
+```typescript
+import { displaySummarizationError, isRecoverableError } from './summarization/index.js'
+
+try {
+  await summarizer.summarize(input, prompt)
+} catch (error) {
+  if (isRecoverableError(error)) {
+    // Retry logic
+  } else {
+    displaySummarizationError(error)
+    // Shows user-friendly message, search results still displayed
+  }
+}
+```
+
+## Security Considerations
+
+### Shell Injection Prevention
+
+All CLI invocations use `spawn()` with argument arrays - **NEVER** `exec()` with string interpolation:
+
+```typescript
+// CORRECT - Safe from shell injection
+spawn('claude', ['-p', userInput, '--output-format', 'text'])
+
+// WRONG - Vulnerable to shell injection
+exec(`claude -p "${userInput}"`)  // NEVER DO THIS
+```
+
+This is enforced throughout the codebase. User input is passed as array elements, never interpolated into shell commands.
+
+### API Key Handling
+
+- API keys are sourced from environment variables only
+- Never stored in config files
+- Environment variable names follow provider conventions:
+  - `DEEPSEEK_API_KEY`
+  - `ANTHROPIC_API_KEY`
+  - `OPENAI_API_KEY`
+  - `GOOGLE_API_KEY` (for Gemini)
+  - `QWEN_API_KEY`
+
+### Timeout Protection
+
+CLI processes have a default 60-second timeout to prevent hung processes.
+
+## Adding New Providers
+
+### CLI Provider
+
+1. Add to `KNOWN_CLIS` in `cli-providers/detection.ts`:
+
+```typescript
+{
+  name: 'newcli',
+  command: 'newcli',
+  displayName: 'New CLI Tool',
+  args: ['--prompt'],
+  useStdin: false,
+}
+```
+
+2. Create implementation in `cli-providers/newcli.ts`:
+
+```typescript
+import { spawn } from 'node:child_process'
+import type { Summarizer, SummaryResult } from '../types.js'
+
+export class NewCLISummarizer implements Summarizer {
+  async summarize(input: string, prompt: string): Promise<SummaryResult> {
+    // SECURITY: Always use spawn() with argument arrays
+    const proc = spawn('newcli', ['--prompt', prompt, input])
+    // ... implementation
+  }
+
+  async isAvailable(): Promise<boolean> {
+    // Check if CLI is installed
+  }
+}
+```
+
+3. Add to factory in `provider-factory.ts`
+
+### API Provider
+
+1. Add pricing to `cost.ts`:
+
+```typescript
+export const API_PRICING = {
+  // ... existing providers
+  newapi: { input: 0.50, output: 1.00, displayName: 'New API' },
+}
+```
+
+2. Create implementation using Vercel AI SDK (when implemented):
+
+```typescript
+import { createOpenAI } from '@ai-sdk/openai'
+
+export class NewAPISummarizer implements Summarizer {
+  // Use Vercel AI SDK for OpenAI-compatible APIs
+}
+```
+
+## Performance
+
+| Provider Type | Latency | Cost |
+|--------------|---------|------|
+| CLI (Claude) | 2-5s | Free |
+| CLI (OpenCode) | 2-5s | Free |
+| API (DeepSeek) | 1-3s | ~$0.0007/query |
+| API (OpenAI) | 1-2s | ~$0.005/query |
+
+### Token Limits
+
+- Input is automatically truncated at 100K characters (~25K tokens)
+- Result content is truncated to 500 chars per result
+- Output tokens capped at 500 for cost estimates
+
+## Configuration Reference
+
+### Config File
+
+```javascript
+// mdcontext.config.js
+/** @type {import('mdcontext').PartialMdContextConfig} */
+export default {
+  aiSummarization: {
+    mode: 'cli',           // 'cli' or 'api'
+    provider: 'claude',    // Provider name
+    model: 'deepseek-chat', // Model for API providers
+    stream: false,         // Enable streaming
+  },
+}
+```
+
+### Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `MDCONTEXT_AISUMMARIZATION_MODE` | 'cli' or 'api' |
+| `MDCONTEXT_AISUMMARIZATION_PROVIDER` | Provider name |
+| `MDCONTEXT_AISUMMARIZATION_MODEL` | Model name (API only) |
+| `MDCONTEXT_AISUMMARIZATION_STREAM` | 'true' or 'false' |
+
+## Troubleshooting
+
+### "CLI tool 'claude' not found"
+
+**Solution:** Install Claude Code from https://claude.ai/download
+
+### "CLI tool 'opencode' not found"
+
+**Solution:** Install OpenCode from https://github.com/opencode-ai/opencode
+
+### "Authentication failed for anthropic"
+
+**Solution:** Set API key: `export ANTHROPIC_API_KEY=sk-...`
+
+### "Rate limit exceeded"
+
+**Solution:** Wait and retry. Consider switching to CLI provider (free).
+
+### "Summarization failed: timeout"
+
+**Solution:** Reduce result set with `--limit` or increase timeout in config.
+
+### "No summarization providers available"
+
+**Solution:** Either:
+1. Install a CLI tool (Claude Code, OpenCode)
+2. Configure an API provider with valid API key
+
+### OpenCode JSON format errors
+
+**Solution:** OpenCode JSON format is undocumented. Try updating OpenCode or switch to Claude CLI.
+
+## Related Documentation
+
+- [README.md](../README.md#ai-summarization) - Quick start guide
+- [CONFIG.md](./CONFIG.md) - Full configuration reference
+- [ERRORS.md](./ERRORS.md) - Error handling patterns