purecontext-mcp 1.1.0 → 1.1.2

package/docs/21-ecosystem-tools.md

@@ -0,0 +1,189 @@
+# Ecosystem & Data Tools
+
+
+Ecosystem tools extend PureContext to data-centric codebases: dbt projects, SQL schemas, OpenAPI specifications, and a context provider framework for domain-specific integrations.
+
+---
+
+## Context provider framework
+
+A context provider is a plugin that adds domain-specific enrichment to symbol metadata and search results. Providers are loaded automatically when their target framework is detected.
+
+**Built-in providers:**
+- **dbt provider** — enriches dbt model symbols with column lineage and upstream/downstream dependencies
+- **OpenAPI provider** — enriches endpoint symbols with request/response schema details
+- **SQL provider** — enriches table symbols with column definitions and foreign key relationships
+
+**Writing a custom provider:**
+
+```typescript
+interface ContextProvider {
+  name: string;
+  detect(projectRoot: string): Promise<boolean>;
+  enrich(symbol: SymbolRecord): Promise<EnrichedSymbol>;
+}
+```
+
+Register in `config.json`:
+
+```json
+{
+  "contextProviders": ["my-custom-provider"]
+}
+```
+
+---
+
+## dbt integration
+
+**Auto-detected by:** `dbt_project.yml` in project root.
+
+### What is indexed
+
+| dbt artifact | Symbol kind | Notes |
+|-------------|-------------|-------|
+| Model (`.sql`) | `function` | SQL logic as source, dbt Jinja expanded |
+| Source | `const` | External data source reference |
+| Seed (`.csv`) | `const` | Static data table |
+| Macro | `function` | Jinja macro definition |
+| Exposure | `const` | Dashboard/downstream consumer |
+
+Column definitions from `schema.yml` are stored in `frameworkMeta.columns`.
+
+### dbt Jinja expansion
+
+Before parsing, dbt SQL files are pre-processed to expand Jinja templating:
+- `{{ ref('orders') }}` → resolved model name
+- `{{ source('raw', 'events') }}` → source reference
+- `{{ config(...) }}` → stripped
+
+This allows the SQL handler to parse the underlying SQL accurately.
+
+### Configuration
+
+```json
+{
+  "dbt": {
+    "manifestPath": "target/manifest.json",
+    "profilesPath": "~/.dbt/profiles.yml"
+  }
+}
+```
+
+Run `dbt compile` or `dbt run` before indexing to ensure `target/manifest.json` is current.
+
+---
+
+## `search_columns`
+
+Search column definitions across dbt models and SQL tables.
+
+**Parameters:**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `repoId` | `string` | required | Target repository |
+| `query` | `string` | required | Column name fragment |
+| `modelName` | `string` | — | Restrict to a specific model |
+
+**Response:**
+
+```json
+{
+  "columns": [
+    {
+      "name": "user_id",
+      "model": "fct_orders",
+      "dataType": "bigint",
+      "description": "Foreign key to dim_users",
+      "nullable": false,
+      "lineage": {
+        "upstream": ["stg_orders.user_id"],
+        "downstream": ["rpt_user_activity.user_id", "fct_revenue.user_id"]
+      }
+    }
+  ]
+}
+```
+
+**Use cases:**
+- "Find all columns named `user_id` across my dbt project"
+- "What models produce the `revenue` column?"
+- "What is the lineage of `order_status`?"
+
+---
+
+## OpenAPI / Swagger handler
+
+**Auto-detected by:** `openapi.yaml`, `openapi.json`, `swagger.yaml`, or `swagger.json` in the project root, or files with `openapi: 3.x.x` content.
+
+### What is indexed
+
+| OpenAPI artifact | Symbol kind | Notes |
+|-----------------|-------------|-------|
+| Endpoint (`GET /users`) | `route` | Path + method as name |
+| Schema object | `type` | Request/response schema |
+| Parameter | `const` | Query/path/header parameter |
+
+### Using OpenAPI symbols
+
+```
+search_symbols(query: "users", kind: "route")
+→ "GET /users", "POST /users", "GET /users/{id}"
+
+get_symbol_source(symbolId: "GET /users/{id}")
+→ Full endpoint definition including parameters, request body, response schemas
+```
+
+---
+
+## SQL handler
+
+**Extensions:** `.sql` files.
+
+**Detected separately from dbt** — the SQL handler processes raw SQL files without dbt Jinja.
+
+### What is indexed
+
+| SQL statement | Symbol kind |
+|--------------|-------------|
+| `CREATE TABLE` | `class` |
+| `CREATE VIEW` | `function` |
+| `CREATE FUNCTION` | `function` |
+| `CREATE PROCEDURE` | `function` |
+| `CREATE INDEX` | `const` |
+
+For dbt projects, the SQL handler works alongside the dbt provider — the provider handles Jinja expansion and column lineage, the handler handles AST parsing.
+
+### Example
+
+```
+search_symbols(query: "orders", kind: "class")
+→ "orders" table (CREATE TABLE orders ...)
+
+get_symbol_source(symbolId: "orders-table-id")
+→ Full CREATE TABLE statement with all column definitions
+```
+
+---
+
+## Combining data tools
+
+A typical data platform exploration workflow:
+
+```
+1. search_columns(query: "revenue")
+   → Find all columns named 'revenue' and their models
+
+2. get_symbol_source(symbolId: "fct_revenue-model-id")
+   → See the SQL logic that produces the revenue column
+
+3. get_context_bundle(symbolId: "fct_revenue-model-id")
+   → Traverse upstream to understand the full lineage
+
+4. search_symbols(query: "revenue", kind: "route")
+   → Find the API endpoints that expose revenue data
+
+5. get_blast_radius(symbolId: "fct_revenue-model-id")
+   → See which dashboards and downstream models depend on this
+```

package/docs/22-distribution.md

@@ -0,0 +1,240 @@
+# Distribution & Platform
+
+
+PureContext supports distribution and automation through index export/import, a public registry of pre-built indexes, webhooks for auto-reindex, GitHub Actions integration, and a VS Code extension.
+
+---
+
+## Index export and import
+
+Share pre-built indexes without requiring everyone to re-index from scratch.
+
+### Export
+
+```bash
+npx purecontext-mcp export --repo <repoId> --out index.pctx.tar.gz
+```
+
+Or by path:
+
+```bash
+npx purecontext-mcp export --path /path/to/project --out index.pctx.tar.gz
+```
+
+The archive contains: compressed SQLite database, HNSW index (if present), and a metadata JSON file.
+
+### Import
+
+```bash
+npx purecontext-mcp import --file index.pctx.tar.gz
+```
+
+After import, the repo is immediately searchable — no re-indexing required.
+
+### Use cases
+
+- **Team onboarding**: export the index after CI, share as an artifact — new developers get a pre-built index on day one
+- **CI pipeline**: cache the index between runs (see GitHub Actions below)
+- **Server migration**: move indexes from one server to another without re-indexing
+
+---
+
+## Public registry
+
+Pre-built indexes for popular open-source projects are hosted on a CDN.
+
+### Pulling a registry index
+
+```bash
+npx purecontext-mcp pull react@18
+npx purecontext-mcp pull typescript@5
+npx purecontext-mcp pull django@4.2
+```
+
+The index is downloaded and imported automatically. Use `list_repos` to confirm it's available.
+
+### Available packages
+
+```bash
+npx purecontext-mcp registry list
+# Lists all available packages with versions and index sizes
+```
+
+### Requesting a new package
+
+Open an issue on GitHub with the package name and version. Registry indexes are built automatically from GitHub releases using the GitHub Actions integration.
+
+---
+
+## Webhooks for auto-reindex
+
+Configure a webhook endpoint to trigger re-indexing automatically when code is pushed to your repository.
+
+### Setup
+
+1. In your PureContext server config:
+
+```json
+{
+  "webhooks": {
+    "enabled": true,
+    "secret": "${WEBHOOK_SECRET}",
+    "branches": ["main", "develop"]
+  }
+}
+```
+
+2. In your GitHub repository settings:
+   - Go to Settings → Webhooks → Add webhook
+   - Payload URL: `https://your-server/webhook/github`
+   - Content type: `application/json`
+   - Secret: same value as `WEBHOOK_SECRET`
+   - Events: "Just the push event"
+
+### How it works
+
+When a push is received:
+1. PureContext verifies the webhook signature (HMAC-SHA256)
+2. Checks if the pushed branch is in `webhooks.branches`
+3. Triggers an incremental re-index of the affected repo
+4. New symbols are available within seconds
+
+### GitLab and others
+
+Custom webhook formats are supported by mapping them to PureContext's internal format:
+
+```json
+{
+  "webhooks": {
+    "enabled": true,
+    "format": "gitlab",
+    "secret": "${WEBHOOK_SECRET}"
+  }
+}
+```
+
+---
+
+## GitHub Actions integration
+
+The official `purecontext/index-action` automates index building in CI.
+
+### Basic usage
+
+```yaml
+# .github/workflows/index.yml
+name: Index with PureContext
+on:
+  push:
+    branches: [main]
+
+jobs:
+  index:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Index repository
+        uses: purecontext/index-action@v1
+        with:
+          server-url: ${{ vars.PCTX_SERVER_URL }}
+          api-key: ${{ secrets.PCTX_API_KEY }}
+```
+
+### Caching the index in CI
+
+```yaml
+- name: Cache PureContext index
+  uses: actions/cache@v4
+  with:
+    path: ~/.purecontext/indexes
+    key: purecontext-${{ github.sha }}
+    restore-keys: purecontext-
+
+- name: Index repository
+  uses: purecontext/index-action@v1
+  with:
+    path: ${{ github.workspace }}
+```
+
+With caching, only changed files are re-parsed on each run — CI index time drops to seconds after the first run.
+
+### Publishing to the registry
+
+```yaml
+- name: Publish index to registry
+  uses: purecontext/index-action@v1
+  with:
+    action: publish
+    package-name: my-org/my-library
+    api-key: ${{ secrets.PCTX_REGISTRY_KEY }}
+```
+
+See the full `action.yml` in the project root for all available inputs.
+
+---
+
+## VS Code extension
+
+The PureContext VS Code extension integrates symbol search and navigation directly into the editor.
+
+### Installation
+
+Search "PureContext" in the VS Code Extensions panel, or:
+
+```bash
+code --install-extension purecontext.purecontext-vscode
+```
+
+The source is in `vscode-extension/` in the project repo.
+
+### Features
+
+| Feature | Description |
+|---------|-------------|
+| Symbol search | `Ctrl+Shift+P` → "PureContext: Search Symbols" |
+| Hover summary | Hover over any identifier to see its PureContext summary |
+| Go to definition | Uses PureContext index for faster lookup in large repos |
+| Dependency graph | `Ctrl+Shift+P` → "PureContext: Show Dependencies" — opens graph panel |
+| Blast radius | Right-click a symbol → "Show Blast Radius" |
+| Quick outline | `Ctrl+Shift+O` with PureContext — shows AI-enriched summaries |
+
+### Configuration
+
+Extension settings in VS Code match the `config.json` fields:
+
+```json
+// .vscode/settings.json
+{
+  "purecontext.serverUrl": "http://localhost:3000",
+  "purecontext.apiKey": "pctx_...",
+  "purecontext.enabled": true
+}
+```
+
+---
+
+## Programmatic API
+
+For building custom integrations, the `@purecontext/client` npm package provides a typed TypeScript client:
+
+```typescript
+import { PureContextClient } from '@purecontext/client';
+
+const client = new PureContextClient({
+  serverUrl: 'http://localhost:3000',
+  apiKey: 'pctx_...'
+});
+
+const symbols = await client.searchSymbols({
+  repoId: 'a1b2c3d4',
+  query: 'authenticate',
+  kind: 'function'
+});
+```
+
+All tool inputs and outputs are fully typed. Install:
+
+```bash
+npm install @purecontext/client
+```

package/docs/23-performance.md

@@ -0,0 +1,121 @@
+# Performance & Scalability
+
+
+PureContext is designed to handle enterprise-scale repos (10k–50k files) using a worker thread pool for parallel tree-sitter parsing.
+
+---
+
+## Indexing speed
+
+Typical performance on a 4-core machine:
+
+| Repo size | First index | Incremental re-index |
+|-----------|-------------|----------------------|
+| 500 files | ~2 seconds | < 100ms |
+| 5,000 files | ~15 seconds | < 1 second |
+| 20,000 files | ~60 seconds | 1–3 seconds |
+| 50,000 files | ~3 minutes | 2–10 seconds |
+
+These numbers assume no AI summarization or semantic indexing. Both add API round-trip time.
+
+---
+
+## Worker thread pool
+
+The bottleneck in sequential indexing is tree-sitter WASM parsing — each WASM instance is single-threaded. The worker thread pool parallelizes parsing across CPU cores.
+
+```
+                Main thread
+                     │
+        ┌────────────┼────────────┐
+        ▼            ▼            ▼
+   Worker 1      Worker 2     Worker 3
+   (TypeScript)  (Python)     (Go)
+   parse + extract  parse + extract  parse + extract
+        │            │            │
+        └────────────┴────────────┘
+                     │
+                Main thread
+             (SQLite writes)
+```
+
+Each worker loads its own WASM grammar instances. File batches are distributed across workers by the main thread. SQLite writes are serialized on the main thread (better-sqlite3 is synchronous).
+
+### Configuring worker threads
+
+```json
+{
+  "workerThreads": 4   // default: os.cpus().length - 1, minimum 1
+}
+```
+
+Increase for CPU-bound workloads on machines with many cores. Do not exceed `os.cpus().length - 1` — you want to leave one core for the main thread and OS.
+
+---
+
+## Memory usage
+
+| Component | Memory |
+|-----------|--------|
+| WASM grammars (per worker) | ~20–30 MB per grammar loaded |
+| In-memory symbol cache (during indexing) | ~100 MB for 10k symbols |
+| SQLite WAL mode (at rest) | ~50 MB |
+| HNSW vector index (if enabled) | ~100 bytes per embedding dimension per symbol |
+
+**Typical peak during indexing:** 200–500 MB for a 10k-file repo. Returns to ~50 MB at rest.
+
+Workers are spawned once and reused for the lifetime of the server — no spawn/teardown overhead per index run.
+
+---
+
+## Incremental re-indexing
+
+The content hash cache makes re-indexing very fast:
+
+1. Each file's SHA-256 hash is stored in the `files` table after indexing
+2. On re-index, the hash is recomputed and compared
+3. Only files with a changed hash are re-parsed
+4. Symbols for unchanged files are retained as-is
+
+A typical `git pull` touches 10–50 files — re-index completes in milliseconds.
+
+To force a full re-index (bypass the hash cache):
+
+```
+Use invalidate_cache tool, then index_folder again.
+```
+
+Or call `index_folder` with `force: true`.
+
+---
+
+## Large repo tuning
+
+For repos with > 10,000 files:
+
+| Setting | Recommendation |
+|---------|---------------|
+| `workerThreads` | Set to `os.cpus().length - 1` |
+| `watchDebounceMs` | Increase to `5000` if many files change at once (e.g., code generation) |
+| `excludePatterns` | Add patterns for generated files, test fixtures with large data files |
+| `maxFileSizeBytes` | Keep at 1 MB or lower — parsing multi-MB files is slow and rarely useful |
+| `fileLimit` | Set to `0` (unlimited) if you need the full repo indexed |
+
+---
+
+## SQLite performance
+
+SQLite in **WAL (Write-Ahead Logging) mode** provides:
+- Concurrent reads without blocking writes
+- Fast writes (no fsync on every write in WAL mode)
+- Crash safety (WAL journal ensures atomicity)
+
+Query performance:
+- `search_symbols` with FTS5: < 5ms for 100k symbols
+- `get_symbol_source`: < 1ms (single row lookup by primary key)
+- `get_blast_radius` (depth 5): 5–20ms depending on graph density
+- `get_context_bundle` (depth 3): 3–15ms
+
+No tuning is needed for the SQLite layer up to ~500k symbols. At very large scale, consider periodic `VACUUM` to reclaim space from deleted symbols.
+
+

package/docs/24-security.md

@@ -0,0 +1,144 @@
+# Security
+
+
+---
+
+## Threat model
+
+PureContext stores and serves source code metadata. Security measures focus on:
+
+**Protected:**
+- Symbol names, signatures, summaries — stored in SQLite
+- Raw source returned by `get_symbol_source` / `get_file_content`
+- Admin API (workspace/key management)
+
+**Not in scope:**
+- The source repository itself — PureContext only reads it during indexing
+- Network transport — handle TLS at a reverse proxy
+- Host OS security — standard server hardening applies
+
+---
+
+## Path traversal prevention
+
+All file paths are validated before any read:
+
+1. Resolved to an absolute path
+2. Verified to start within the project root (the indexed directory)
+3. Symlinks that resolve outside the root are blocked unless `allowSymlinks: true`
+
+This prevents tools like `get_file_content` from being used to read arbitrary files on the server.
+
+---
+
+## Secret file exclusion
+
+The following files are automatically excluded from indexing (never stored in the index):
+
+- `.env`, `.env.*`, `.env.local`, `.env.production`
+- `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`
+- `id_rsa`, `id_ed25519`, `id_ecdsa`, `id_dsa`
+- `credentials.json`, `credentials.yaml`, `secrets.json`
+- `serviceAccountKey*.json`, `*-service-account.json`
+- `*.token`, `*.secret`
+
+These patterns are built into the file discovery layer and cannot be overridden by `excludePatterns`.
+
+---
+
+## Binary file detection
+
+Files are scanned for null bytes in the first 8 KB. Files with null bytes are treated as binary and skipped — preventing large binary files (which may contain embedded secrets) from entering the index.
+
+---
+
+## API key security
+
+Keys are stored as **bcrypt hashes** in the auth database — plaintext is never persisted after the key is generated.
+
+- Keys are shown once on creation — store in a password manager or CI secrets
+- Key format: `pctx_<workspaceId>_<24-char-random>_<checksum>`
+- The checksum allows fast format validation without a database lookup
+- Rotate keys by revoking the old one and creating a new one
+- Use `read` permission for agents that only query, not `write` or `admin`
+
+---
+
+## Workspace isolation
+
+Every query is scoped to the workspace of the API key used:
+
+- A key from workspace A cannot query repos in workspace B
+- Workspace scoping is enforced in all SQL queries via `workspace_id` column
+- The admin key (`PCTX_ADMIN_KEY`) bypasses workspace isolation — protect it like a root password
+
+---
+
+## Rate limiting
+
+Per-key rate limits (token bucket algorithm):
+
+- `rateLimit.maxTokens` — bucket capacity (default: 100)
+- `rateLimit.refillRate` — tokens/second refill rate (default: 10)
+- Heavy tools (e.g., `index_folder`) cost more tokens per call
+
+When exceeded: `429 Too Many Requests` with `Retry-After` header.
+
+---
+
+## HTTP security
+
+- **Default host: `127.0.0.1`** — loopback only, not exposed on the network
+- A warning is logged at startup if `host` is not loopback and `auth.enabled` is false
+- **Timing-safe comparison** — `crypto.timingSafeEqual()` used for token comparison (prevents timing attacks)
+- **Request body limit** — 1 MB maximum
+- **CORS** — whitelist-controlled via `http.corsOrigins`
+
+---
+
+## Remote repository cloning
+
+When using `index_repo`:
+
+- Only `https://`, `http://`, and `git@` URL schemes are accepted
+- Clone tokens (`token` parameter) are never logged
+- Clones are isolated under `~/.purecontext/clones/`
+
+---
+
+## Self-hosting hardening checklist
+
+- [ ] Run behind a TLS-terminating reverse proxy (nginx, Caddy)
+- [ ] Set `PCTX_ADMIN_KEY` via environment variable, never in `config.json`
+- [ ] Restrict developer API keys to `read` permission where possible
+- [ ] Restrict server bind address to internal network if not public-facing
+- [ ] Use firewall rules to limit access to port 3000
+- [ ] Monitor `/health` endpoint and set up uptime alerts
+- [ ] Rotate API keys regularly
+- [ ] Back up the `/data` volume (contains indexes and auth database)
+
+---
+
+## Data at rest
+
+SQLite files are stored in `indexDir` (`~/.purecontext/indexes/` by default). No encryption at rest is applied by PureContext itself.
+
+For sensitive codebases, use OS-level disk encryption:
+- macOS: FileVault
+- Windows: BitLocker
+- Linux: LUKS
+
+Docker: use encrypted volumes if the host is shared.
+
+---
+
+## Audit logging
+
+The HTTP server logs every MCP tool call with:
+- Timestamp
+- API key label (not the key itself)
+- Tool name
+- `repoId`
+- Response status and duration
+
+At `debug` level, full request/response bodies are included. Pipe logs to your SIEM or log aggregator for audit trails.