purecontext-mcp 1.1.0 → 1.1.2

package/docs/25-architecture-overview.md

@@ -0,0 +1,240 @@
+# Architecture Overview
+
+
+This section explains PureContext's internal design. Useful for contributors, developers building on top of PureContext, or anyone who wants to understand why it works the way it does.
+
+---
+
+## Three-layer architecture
+
+```
+┌──────────────────────────────────────────┐
+│  Framework Adapters  (src/adapters/)     │  Vue, React, Django, Spring, ...
+└───────────────────┬──────────────────────┘
+                    │ uses
+┌───────────────────▼──────────────────────┐
+│  Language Handlers  (src/handlers/)      │  TypeScript, Python, Go, ...
+└───────────────────┬──────────────────────┘
+                    │ uses
+┌───────────────────▼──────────────────────┐
+│  Core              (src/core/)           │  Parse, Store, Search, Watch, MCP
+└──────────────────────────────────────────┘
+```
+
+**The dependency direction is strictly downward: Adapters → Handlers → Core. Never the reverse.**
+
+This constraint means:
+- Core knows nothing about specific languages or frameworks
+- Handlers know nothing about frameworks
+- Adding a new language never touches Core
+- Adding a new framework never touches Core or existing Handlers
+
+---
+
+## Core layer (`src/core/`)
+
+The core is language and framework agnostic. Responsibilities:
+
+| Module | Responsibility |
+|--------|----------------|
+| `index-manager.ts` | Orchestrates the full indexing pipeline |
+| `file-processor.ts` | Reads files, checks hash cache, dispatches to handlers |
+| `parse-dispatcher.ts` | Routes files to the correct LanguageHandler by extension |
+| `types.ts` | Core type definitions (SymbolRecord, ImportRecord, etc.) |
+| `errors.ts` | Typed error classes extending `PureContextError` |
+| `db/schema.ts` | SQLite table definitions and migrations |
+| `db/symbol-store.ts` | Symbol CRUD — insert, search, retrieve |
+| `db/file-store.ts` | Raw file content cache for `get_file_content` |
+| `db/embedding-store.ts` | HNSW vector index management |
+| `watcher/file-watcher.ts` | Chokidar wrapper with debounce and fast incremental path |
+
+The MCP server (`src/server/`) and semantic search (`src/semantic/`) also build on Core.
+
+---
+
+## Language handler layer (`src/handlers/`)
+
+Each handler implements `LanguageHandler`:
+
+```typescript
+interface LanguageHandler {
+  extensions(): string[];           // ['.ts', '.tsx']
+  grammarPath(): string;            // path to .wasm grammar file
+  extractSymbols(tree: Tree, source: Buffer): SymbolRecord[];
+  extractImports(tree: Tree, source: Buffer): ImportRecord[];
+  extractDocstring(node: SyntaxNode): string | null;
+}
+```
+
+Handlers know how to:
+- Load their tree-sitter WASM grammar
+- Walk the AST to find symbol-bearing nodes
+- Extract names, byte offsets, signatures, and docstrings
+- Extract import statements and resolve specifiers
+
+Handlers know **nothing** about frameworks, adapters, or the database.
+
+---
+
+## Framework adapter layer (`src/adapters/`)
+
+Each adapter implements `FrameworkAdapter`:
+
+```typescript
+interface FrameworkAdapter {
+  name: string;
+  detect(projectRoot: string): Promise<boolean>;
+  fileFilter(filePath: string): boolean;
+  preProcess?(source: Buffer, filePath: string): ProcessedBlock[];
+  extractFrameworkSymbols(tree: Tree, source: Buffer, filePath: string): SymbolRecord[];
+  enrichMetadata?(symbol: SymbolRecord): SymbolRecord;
+}
+```
+
+Adapters compose on top of language handlers. Multiple adapters can be active at once. Auto-detection runs at index time by calling `detect()` on every registered adapter.
+
+---
+
+## Data flow: indexing pipeline
+
+```
+index_folder(path)
+    ↓
+FileDiscovery
+  - scan directory recursively
+  - apply exclude patterns (node_modules, .git, *.pem, etc.)
+  - prioritize by type (source files before config files)
+    ↓
+FileProcessor (per file, parallel via worker threads)
+  - read file content
+  - compute SHA-256 hash
+  - compare with hash cache → skip if unchanged
+    ↓
+ParseDispatcher
+  - route file to LanguageHandler by extension
+    ↓
+LanguageHandler.parse(buffer) → tree-sitter AST
+    ↓
+LanguageHandler.extractSymbols(AST) → SymbolRecord[]
+LanguageHandler.extractImports(AST) → ImportRecord[]
+    ↓
+FrameworkAdapter.extractFrameworkSymbols(AST) → SymbolRecord[] (if active)
+FrameworkAdapter.enrichMetadata(symbol) → SymbolRecord (if active)
+    ↓
+AISummarizer (optional)
+  - fill in missing summaries via AI API
+    ↓
+SymbolStore.insertBatch(symbols) → SQLite (symbols table)
+PathResolver.resolve(imports) → DepEdge[]
+DepStore.insertBatch(edges) → SQLite (dep_edges table)
+SemanticIndexer.index(symbols) → HNSW index (if enabled)
+```
+
+---
+
+## Data flow: query pipeline
+
+```
+MCP tool call (e.g., search_symbols)
+    ↓
+Tool handler (src/server/tools/search-symbols.ts)
+  - parse and validate input
+    ↓
+SymbolStore.search(query)
+  - FTS5 query → BM25 ranked results
+  - RelevanceRanker re-scores and sorts
+    ↓
+Tool handler
+  - compute _tokenEstimate
+  - record token savings in savings tracker
+    ↓
+JSON response → MCP client
+```
+
+---
+
+## SQLite schema
+
+Four tables:
+
+```sql
+CREATE TABLE repos (
+  id TEXT PRIMARY KEY,          -- SHA-256(absolutePath).slice(0,16)
+  root_path TEXT NOT NULL,
+  indexed_at INTEGER,
+  file_count INTEGER,
+  symbol_count INTEGER
+);
+
+CREATE TABLE files (
+  id TEXT PRIMARY KEY,
+  repo_id TEXT NOT NULL,
+  file_path TEXT NOT NULL,      -- relative to repo root
+  content_hash TEXT,
+  content BLOB,                 -- cached raw content
+  last_modified INTEGER
+);
+
+CREATE TABLE symbols (
+  id TEXT PRIMARY KEY,          -- SHA-256(filePath:name:kind).slice(0,16)
+  repo_id TEXT NOT NULL,
+  name TEXT NOT NULL,
+  kind TEXT NOT NULL,
+  file_path TEXT NOT NULL,
+  start_byte INTEGER,
+  end_byte INTEGER,
+  signature TEXT,
+  summary TEXT,
+  framework_meta TEXT           -- JSON blob
+);
+
+CREATE TABLE dep_edges (
+  id TEXT PRIMARY KEY,
+  repo_id TEXT NOT NULL,
+  source_file TEXT NOT NULL,
+  target_file TEXT NOT NULL,
+  specifier TEXT,               -- raw import path
+  imported_names TEXT           -- JSON array
+);
+```
+
+FTS5 virtual table on `symbols(name, summary, signature)` provides full-text search.
+
+---
+
+## Deterministic IDs
+
+IDs are deterministic and stable across re-indexes:
+
+- `repoId = SHA-256(absolutePath).slice(0, 16)`
+- `symbolId = SHA-256(filePath:name:kind).slice(0, 16)`
+
+This means the same symbol in the same file always has the same ID. Agents can store symbol IDs in long-term memory and retrieve them reliably on the next session.
+
+---
+
+## File watcher
+
+`chokidar` watches the indexed directory for file changes. A 2-second debounce (configurable) prevents thrashing on bulk saves.
+
+**Fast path for single file changes:**
+1. Re-parse only the changed file
+2. Delete old symbols for that file from SQLite
+3. Insert new symbols
+4. Update dep_edges for that file
+5. Skip full re-scan
+
+This makes editor save → symbol update latency typically < 200ms.
+
+---
+
+## Storage locations
+
+| Path | Contents |
+|------|----------|
+| `~/.purecontext/indexes/` | SQLite databases (one per project) |
+| `~/.purecontext/indexes/{repoId}/hnsw.idx` | HNSW vector index |
+| `~/.purecontext/clones/` | Remote repo clones (via `index_repo`) |
+| `~/.purecontext/config.json` | Configuration file |
+| `~/.purecontext/_savings.json` | Cumulative token savings |
+| `~/.purecontext/telemetry.jsonl` | Local telemetry audit log (if enabled) |

package/docs/26-troubleshooting.md

@@ -0,0 +1,234 @@
+# Troubleshooting
+
+
+---
+
+## Health check
+
+Before investigating any problem, run the health check:
+
+```bash
+purecontext-mcp --health
+```
+
+```json
+{
+  "status": "ok",
+  "version": "1.x.x",
+  "uptime": 0,
+  "nodeVersion": "20.11.0",
+  "platform": "linux",
+  "indexDir": "/home/user/.purecontext/indexes",
+  "repoCount": 3,
+  "grammars": {
+    "tree-sitter-typescript.wasm": true,
+    "tree-sitter-javascript.wasm": true
+  }
+}
+```
+
+In HTTP mode:
+
+```bash
+curl http://localhost:3000/health
+```
+
+A non-zero exit code (CLI) or non-200 response (HTTP) indicates something is wrong.
+
+---
+
+## Debug logging
+
+```bash
+purecontext-mcp --log-level debug
+# or
+LOG_LEVEL=debug purecontext-mcp
+```
+
+Debug logs show: file-by-file parsing progress, hash cache hits/misses, worker thread activity, query plan details, rate limit decisions.
+
+**Use in production only temporarily** — debug logs are verbose and may include file paths.
+
+---
+
+## Common errors
+
+### "Repo not indexed" / "Index not found"
+
+The repository has not been indexed yet, or the `repoId` belongs to a different installation.
+
+```
+Fix: Call index_folder with the correct path.
+     Use resolve_repo to check if the path is indexed.
+```
+
+---
+
+### "Grammar file not found"
+
+A `.wasm` grammar file is missing from the `grammars/` directory.
+
+```bash
+# Check which grammars are present:
+purecontext-mcp config --check
+
+# Fix: reinstall the package
+npm install -g purecontext-mcp
+# or
+npm rebuild purecontext-mcp
+```
+
+---
+
+### "better-sqlite3 bindings failed to load"
+
+The native `better-sqlite3` binary doesn't match your Node.js version or platform.
+
+```bash
+# Fix: rebuild the native module
+cd $(npm root -g)/purecontext-mcp
+npm rebuild better-sqlite3
+
+# If that fails, install build tools first:
+# macOS: xcode-select --install
+# Linux: apt install python3 make g++
+# Windows: npm install -g windows-build-tools
+```
+
+---
+
+### Missing symbols after indexing
+
+Check this list in order:
+
+1. **Is the file excluded?** Check `excludePatterns` and built-in exclusions (node_modules, dist, .env, etc.)
+2. **Is the file too large?** Files > `maxFileSizeBytes` (default 1 MB) are skipped
+3. **Is it a secret file?** `.pem`, `.key`, `credentials.json` etc. are always excluded
+4. **Is it a private symbol?** Go unexported names, C `static` functions, Java/C# `private` members are excluded by design
+5. **Is the language supported?** Check [Language Support](07-language-support.md)
+6. **Did the adapter detect?** Run `purecontext-mcp config --check` — check "Detected adapters" for your project
+
+Force a specific adapter if auto-detection fails:
+```json
+{ "adapters": ["vue", "nuxt"] }
+```
+
+---
+
+### Adapter not activating
+
+Run `purecontext-mcp config --check` and look at the detected adapters. Common detection requirements:
+
+| Framework | Detection file |
+|-----------|---------------|
+| Vue | `vue` in `package.json` dependencies |
+| Nuxt | `nuxt.config.ts` at project root |
+| Django | `manage.py` at project root |
+| Rails | `gem 'rails'` in `Gemfile` |
+| Gin | `github.com/gin-gonic/gin` in `go.mod` |
+| Spring Boot | `spring-boot-starter` in `pom.xml` |
+
+---
+
+### Incremental re-index not triggering
+
+The file watcher uses a 2-second debounce. For manual force re-index:
+
+```
+Call index_folder again — it skips unchanged files via content hashing.
+```
+
+If the watcher seems stuck on Linux:
+
+```bash
+# Check inotify limit
+cat /proc/sys/fs/inotify/max_user_watches
+
+# Increase if needed
+echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
+sudo sysctl -p
+```
+
+---
+
+### HTTP transport not accessible
+
+1. **Check `http.host`** — default is `127.0.0.1` (loopback only). Set to `0.0.0.0` for network access.
+2. **Check CORS** — `http.corsOrigins` must include the browser's origin.
+3. **Check auth** — if `http.auth.enabled: true`, all requests need `Authorization: Bearer <token>`.
+4. **Check firewall** — port 3000 must be open from the client's perspective.
+
+---
+
+### Semantic search not working
+
+1. Verify `semantic.enabled: true` in config
+2. Verify a provider is configured: `semantic.provider` is not `"none"`
+3. Check if the repo meets the threshold: lower `semantic.threshold` for small repos
+4. Verify the API key: test it with a direct API call
+5. Re-index — the HNSW index is built during indexing, not at query time
+
+---
+
+### Config validation errors
+
+```bash
+purecontext-mcp config --check
+```
+
+Reports all schema violations with field names and expected types.
+
+---
+
+### Rate limit errors (HTTP mode)
+
+`429 Too Many Requests` — the API key has hit its per-second token limit.
+
+Check the `Retry-After` header for when to retry. If limits are too low for your use case, ask the server admin to raise `rateLimit.maxTokens` or `rateLimit.refillRate` for your key.
+
+---
+
+### "git not found" when using `index_repo`
+
+```bash
+# Verify git is installed and on PATH
+git --version
+
+# Install if missing:
+# macOS: xcode-select --install
+# Linux: apt install git
+# Windows: https://git-scm.com/download/win
+```
+
+---
+
+### Indexing is slow
+
+- First index is always slower — subsequent runs are incremental (hash-based)
+- AI summarization (`allowRemoteAI: true`) adds API latency — disable during testing
+- Semantic indexing adds API latency — disable if not needed
+- For large repos (> 10k files): increase `workerThreads` in config
+
+---
+
+## Re-indexing from scratch
+
+```
+Use invalidate_cache tool with your repoId, then call index_folder again.
+```
+
+Or manually:
+
+```bash
+rm ~/.purecontext/indexes/<repoId>.db
+rm -rf ~/.purecontext/indexes/<repoId>/
+```
+
+Then call `index_folder` — a clean full index will be built.
+
+---
+
+## Getting more help
+
+- **GitHub Issues:** Report bugs with version, OS, Node.js version, error log, and approximate repo size
+- **`--log-level debug`** output is the most useful information to include in a bug report

package/docs/27-api-stability.md

@@ -0,0 +1,114 @@
+# API Stability & Changelog
+
+
+---
+
+## Versioning policy
+
+PureContext follows **semantic versioning (semver)**:
+
+| Change type | Version bump |
+|-------------|-------------|
+| Remove or rename a tool | `2.0.0` (major) |
+| Remove or rename a required parameter | `2.0.0` (major) |
+| Remove a top-level response field | `2.0.0` (major) |
+| Change the type or semantics of a stable field | `2.0.0` (major) |
+| Add a new tool | `1.x.0` (minor) |
+| Add an optional parameter | `1.x.0` (minor) |
+| Add a new response field | `1.x.0` (minor) |
+| Bug fix with no API surface change | `1.0.x` (patch) |
+
+Agents and integrations built against `v1.x` will not break until `v2.0`.
+
+---
+
+## Public API contract
+
+### What is stable in 1.x
+
+| Surface | Stable? |
+|---------|---------|
+| MCP tool names | Yes |
+| Required tool parameters | Yes |
+| Top-level response field names and types | Yes |
+| CLI flag names and exit codes | Yes |
+| Symbol ID format (deterministic hash) | Yes |
+| `repoId` format (deterministic hash) | Yes |
+
+### What is NOT covered by the stability guarantee
+
+| Surface | Notes |
+|---------|-------|
+| Internal module paths (`src/core/...`) | For contributors only, may change freely |
+| SQLite schema column names | Subject to migration with version upgrades |
+| HTTP admin API (`/admin/*`) | Stable at `1.0` but less strictly than MCP tools |
+| `ai.*` config group | May change as AI summarization matures |
+| Response field order | JSON objects — do not rely on ordering |
+
+For the full list of stable tools and their exact parameter/response contracts, see [docs/API_STABILITY.md](API_STABILITY.md).
+
+---
+
+## Stable tools (1.x)
+
+All tools listed in [MCP Tools Reference](06-tools-reference.md) are stable in 1.x, with the exception of tools in the experimental list below which are marked `@experimental` until a future stabilization release.
+
+**Stable tool list:**
+`index_folder` · `index_repo` · `resolve_repo` · `list_repos` · `search_symbols` · `search_text` · `get_symbol_source` · `get_file_outline` · `get_repo_outline` · `get_file_tree` · `get_context_bundle` · `get_blast_radius` · `find_importers` · `find_dead_code` · `get_savings_stats` · `get_layer_violations`
+
+**Experimental:**
+`invalidate_cache` · `get_file_content` · `get_symbols` · `find_references` · `search_semantic` · `search_cross_repo` · `find_similar` · `get_symbol_history` · `get_churn_metrics` · `get_quality_metrics` · `detect_antipatterns` · `get_architecture_doc` · `search_columns`
+
+---
+
+## Stable symbol kinds
+
+The following `SymbolKind` values are stable in 1.x:
+
+`function` · `class` · `method` · `const` · `type` · `interface` · `enum` · `component` · `composable` · `hook` · `route` · `decorator` · `middleware`
+
+New kinds may be added in minor releases. Clients should handle unknown kinds gracefully (do not throw on unknown kind values).
+
+---
+
+## Deprecation process
+
+Before removing anything in a major version:
+
+1. The deprecated item is marked `_deprecated: true` in the response for one minor release cycle
+2. A note is added to `CHANGELOG.md` and the release notes
+3. The deprecated item is removed in the next major version
+
+---
+
+## Index file compatibility
+
+SQLite index files are forward-compatible within a major version. Upgrading from `v1.0` to `v1.5` requires no action — the migrator runs automatically at startup.
+
+A `v1.x` → `v2.0` upgrade may require a re-index. The CLI warns at startup if it detects an incompatible index version:
+
+```
+Warning: Index at ~/.purecontext/indexes/abc123.db was created by v1.x
+and is not compatible with v2.0. Run index_folder to re-index.
+```
+
+---
+
+## Changelog
+
+See [CHANGELOG.md](../CHANGELOG.md) for the full version history.
+
+**Key milestones:**
+
+| Version | Highlights |
+|---------|-----------|
+| `1.0.0` | Stable release: prebuilt binaries, 19 languages, 20+ framework adapters, FTS5 search, HNSW semantic search, Web UI, rate limiting, Docker |
+| `1.1.0` | `find_references`, `get_file_content`, `get_symbols`, `invalidate_cache` |
+| `1.2.0` | search debug mode, `context_lines`/`verify`, GitHub API indexing, Gemini Flash |
+| `1.3.0` | context providers, dbt, `search_columns`, OpenAPI, SQL handler |
+| `1.4.0` | 15 new language handlers (34 total) |
+| `1.5.0` | cross-repo search, code similarity, MCP Resources |
+| `1.6.0` | git & history integration |
+| `1.7.0` | AI-powered architecture analysis |
+| `1.8.0` | enhanced Web UI |
+| `1.9.0` | distribution & platform (registry, webhooks, GitHub Actions, VS Code extension) |

package/docs/README.md

@@ -0,0 +1,71 @@
+# PureContext MCP — User Manual
+
+PureContext MCP indexes your codebase and gives AI agents a way to navigate it without reading entire files. Instead of loading hundreds of lines of code to find one function, Claude (or any other MCP-compatible AI) can search by name, retrieve just the symbol it needs, and understand the dependency chain — all in a fraction of the tokens.
+
+This manual covers everything from installation through advanced features. Use the sections below to navigate to what you need, or read in order for a full introduction.
+
+---
+
+## Getting Started
+
+These three sections get you from zero to a working setup.
+
+- [Introduction](01-introduction.md) — What PureContext is, why token efficiency matters, key concepts
+- [Installation](02-installation.md) — Install via npm, verify your setup, upgrade and uninstall
+- [Quick Start](03-quick-start.md) — Index a project and search your first symbol in minutes
+
+---
+
+## Reference
+
+Complete reference material for configuration, the CLI, and every MCP tool.
+
+- [Configuration](04-configuration.md) — Full `config.json` schema, every field explained, environment variable overrides
+- [CLI Reference](05-cli-reference.md) — Every command and flag: `config --init`, `--health`, `--transport`, and more
+- [MCP Tools Reference](06-tools-reference.md) — Every tool with inputs, outputs, and examples — grouped by category
+
+---
+
+## Language & Framework Support
+
+- [Language Support](07-language-support.md) — All 34 supported languages: what gets indexed and known limitations
+- [Framework Adapters](08-framework-adapters.md) — Vue, React, Nuxt, Next.js, Angular, NestJS, Express, Django, Rails, Spring, and 20+ more
+
+---
+
+## Core Features
+
+- [Dependency Graph Tools](09-dependency-graph.md) — Find what a symbol depends on, what depends on it, and what is dead code
+- [Semantic Search](10-semantic-search.md) — Search by meaning rather than name using HNSW vector index
+- [Search Quality & Ranking](11-search-quality.md) — How FTS5, camelCase splitting, and relevance ranking work; search tips
+- [AI Summarization](12-ai-summarization.md) — Auto-generate symbol descriptions with Anthropic, OpenAI, or Gemini
+- [Token Savings Tracker](13-token-savings.md) — See exactly how many tokens (and dollars) PureContext saves per session
+
+---
+
+## Deployment
+
+- [Transport Modes](14-transport-modes.md) — stdio (local) vs HTTP/SSE (team/browser); TLS via reverse proxy
+- [Team Setup & Multi-Tenant](15-team-setup.md) — Shared server, workspaces, API keys, rate limiting
+- [Docker Deployment](16-docker.md) — `docker run`, Docker Compose, volumes, environment variables, health checks
+
+---
+
+## Advanced Features
+
+- [Web UI](17-web-ui.md) — Visual graph viewer, heatmap, symbol timeline, test coverage overlay
+- [Git & History Integration](18-git-history.md) — Symbol-level commit history, churn metrics, PR diff analysis
+- [Cross-Repo Intelligence](19-cross-repo.md) — Search across multiple repos, find similar code, MCP Resources
+- [AI-Powered Architecture Analysis](20-architecture-analysis.md) — Quality metrics, anti-pattern detection, auto-generated architecture docs
+- [Ecosystem & Data Tools](21-ecosystem-tools.md) — dbt integration, OpenAPI/Swagger handler, SQL handler, column search
+- [Distribution & Platform](22-distribution.md) — Index export/import, public registry, webhooks, GitHub Actions, VS Code extension
+
+---
+
+## Operations & Reference
+
+- [Performance & Scalability](23-performance.md) — Worker thread pool, large repo tuning, memory usage
+- [Security](24-security.md) — API key model, workspace isolation, path traversal prevention, hardening checklist
+- [Troubleshooting](26-troubleshooting.md) — Common errors, `--health` output, debug logging, re-indexing from scratch
+- [Architecture Overview](25-architecture-overview.md) — How PureContext works internally: three-layer design, data flow, SQLite schema
+- [API Stability & Changelog](27-api-stability.md) — Semver policy, stable vs experimental tools, version history